Anatomy of the twelve DCGM and five vLLM metrics: analogies, documented anomalies and real cases 2024-2026

This post goes deeper into the list of metrics presented in GPU observability for LLM inference. There each metric got its G/A/R threshold and PromQL query; here each one gets its explanatory analogy and the anomaly documented in production with a referenced public case. It is the post worth having open when an alert fires and you still do not know what to do with it; the next post on runbooks translates each anomaly into concrete action.

TL;DR

The twelve DCGM metrics (compute, memory, thermal-power, health) and the five vLLM engine metrics (concurrency, KV pool, SLO latencies) covered in the previous post paint the cluster’s cockpit, but a list without context does not teach you to diagnose. Each metric has a recurring anomalous pattern documented in the public literature (academic papers, GitHub issues, OEM KBs, operator blogs) that the veteran operator recognises instantly and the junior one does not. This post develops each metric with an analogy of its own that pins down which question it answers, and with the statistically relevant anomaly backed by figures from documented incidents. Three examples of the calibre involved: Meta published that during the training of Llama 3 405B on 16,384 H100s there were 419 unplanned failures in 54 days, one every 3 hours, with GPU + HBM3 accounting for 47 % of the total; the paper Story of Two GPUs (arXiv 2503.11901) quantifies that the H100 has 3.2× worse MTBE from uncorrectable ECC than the A100, attributable to the higher density of HBM3; issue vllm#16300 documents that on a cluster of 8×A100 80 GB TP=8 delivers worse throughput than TP=4 because NVLink saturation kills the partition speedup. The KBs Dell 000220508 and Lenovo HT514380 formalise the recurring HW Power Brake case in H100 racks overcommitted at PDU level. Issue vllm#25677 showed chunked prefill 10× slower than without it on Qwen3-30B-A3B (bad calibration of max_num_batched_tokens). Issue vllm#11912 documents a TPOT regression from 15.7 ms to 25.7 ms crossing version 0.6.4. Each case includes a verifiable URL. The operational rule: when an alert arrives, look first at the anomalous pattern associated with the metric that fired, before opening the request’s trace; 80 % of degradations match one of the documented patterns.

You are here: OBSERVE — the diagnosis layer

Family 1 — Compute

DCGM_FI_PROF_SM_OCCUPANCY — is there parallel work in the engines?

The analogy. An industrial kitchen with 32 hobs and a single chef. The metric answers “how many hobs have a pan on them right now?”. If half are empty, the kitchen is under-used: the orders come out one after another because the chef does not open up parallelism. If all of them are occupied but the chef is standing still watching a stopwatch, the hobs are on but nothing is being cooked (a pathological kernel saturating SMs without doing useful work).

The documented anomaly. The best-known trap: high SM occupancy does not imply real throughput. The article “GPU Utilization Is a Counter, Not a Cause” (Ingero, May 2026) put it in one exact sentence: “a kernel running at 5 % of peak FLOPS for 100 ms still reads 100 % on SM_ACTIVE”. In MoE workloads the effect turns pathological: overloaded experts produce the Straggler Effect (paper arXiv 2503.05066). The SMs look busy while waiting for the saturated expert, and the utilisation dashboard paints green while latency goes through the roof.

Operational implication. Do not trust sizing or autoscaling to SM occupancy alone. Always combine it with PIPE_TENSOR_ACTIVE (is there useful compute?) and DRAM_ACTIVE (is memory the bottleneck?). The normal LLM decode regime is 30–55 %, not 99 %; seeing a sustained 99 % with high TPOT is a symptom of a kernel bug or an MoE straggler.

DCGM_FI_PROF_PIPE_TENSOR_ACTIVE — are the tensor cores producing?

The analogy. A factory with two lines: the manual one (CUDA cores) and the automated one (tensor cores). The metric answers “what percentage of the time is the automated line active?”. If you buy an H100 for its tensor cores and the automated line is at 5 %, you have paid for a Ferrari to do bicycle courier work.

The documented anomaly. Issue vllm#20783 (July 2025) was titled literally “Performance Anomaly: compressed-tensors shows no speedup over BF16 on H100”. The operator expected 1.5–2× with FP8 quantisation and got parity with BF16; the PIPE_TENSOR_ACTIVE metric revealed that the FP8 path was not running on the HMMA units (the tensor unit for FP16/BF16/FP8) and was falling back to CUDA cores. Issue vllm#31475 documented the parallel case on MI300X: FP8 slower than BF16 because of a regression in the ROCm path. DCGM exposes separate counters per unit (HMMA for FP16/BF16/FP8, IMMA for INT8, DMMA for TF32/FP32); if HMMA is low even though the model is BF16, the engine is not using tensor cores.

Operational implication. Check PIPE_TENSOR_ACTIVE after every change of quantisation or engine version; a supposedly neutral change may have disabled the optimised path. For prefill expect 50–80 %; for decode 15–30 % is normal (decode is memory-bound, not compute-bound). A figure below 5 % in prefill means the engine is not using tensor cores.

DCGM_FI_PROF_DRAM_ACTIVE — is the HBM saturated?

The analogy. A motorway with N lanes. The metric answers “what percentage of the time are all the lanes occupied moving cars?”. When the tensor cores ask for data faster than the HBM delivers it, the motorway is at 95 % and the engines wait. In decode, this is the normal regime: you walk the model weights and the KV cache through for every token.

The documented anomaly. The paper “Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inference” (arXiv 2503.08311) quantifies that at contexts ≥ 128k, reading the KV cache dominates the total decode time and saturates the HBM3 (3.35 TB/s on the H100). The distinctive pattern: DRAM_ACTIVE > 80 % with PIPE_TENSOR_ACTIVE around 10–20 %. Raising the batch no longer helps: the bottleneck is not FLOPS, it is bandwidth. The useful lever is compressing the KV, see Quantisation for --kv-cache-dtype=fp8, which cuts the KV footprint by roughly 50 %.

Operational implication. If DRAM_ACTIVE > 95 % is sustained and gpu_cache_usage_perc < 70 %, something other than your engine is asking for HBM (a leak in a library, another process sharing the GPU without MIG). Investigate immediately with nvidia-smi and fuser /dev/nvidia*.

Family 2 — Memory

DCGM_FI_DEV_FB_USED — how much VRAM has been consumed?

The analogy. The fuel level in an aircraft’s tank in flight: the pilot needs to know how much is left and at what rate it is being consumed, not just the instantaneous figure. An H100 at 88 % FB used and stable can operate calmly; the same figure rising 2 %/min announces OOM in 7 minutes.

The documented anomaly. Issue dcgm-exporter#512 documents a surprise relevant to MIG clusters: DCGM_FI_DEV_FB_USED and DCGM_FI_DEV_FB_FREE are absent on H100 GPU instances with MIG enabled, present on A100 and B200, but an exporter bug hides them on H100-MIG. Operators who assume the dashboard covers everything discover the blind spot on the day of the first OOM. Issue dcgm-exporter#271 documents another detail: FB_USED + FB_FREE does not always add up to a constant because there is driver-reserved overhead that shows up in the delta. The original PagedAttention/vLLM paper estimated that pre-PagedAttention serving frameworks wasted 60–80 % of the KV cache to fragmentation; PagedAttention brought it down to under 4 %.

Operational implication. On H100 MIG clusters, check that DCGM_FI_DEV_FB_USED appears per instance before trusting alerts; if it is absent, monitor through nvidia-smi --query-gpu=memory.used directly. Operational rule: alert on the delta (a sustained rise), not just an absolute threshold.

DCGM_FI_DEV_FB_FREE — the absolute complement

The analogy. The “range remaining” indicator in a modern car: it complements the percentage with an absolute figure that is directly actionable.

The documented anomaly. When an aggressive PagedAttention pool leaves FB_FREE at small absolute values (< 2 GiB), any normal allocation of transient buffers (the activations of a large prefill) can push you into OOM. The classic pattern: a “green” percentage (87 %) but a “red” absolute (< 4 GiB free on an 80 GB H100).

Operational implication. A complementary alert with an absolute threshold: DCGM_FI_DEV_FB_FREE < 4096 (MiB). It is the safety net for the cases where the percentage misleads because the engine is configured with a very high gpu_memory_utilization.

DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL — does the internal bus hold up?

The analogy. An interstate motorway between four cities. Every car that crosses to do a tensor parallel all-reduce pays a toll and consumes width. When there are more cars than the motorway supports, the latency to reach the destination shoots up, even if each individual car is fast.

The documented anomaly. Issue vllm#16300 (April 2025) was titled “Performance degradation with tp=8 compared to tp=4 on 8×A100(80G)” and documented TP=8 delivering worse throughput than TP=4 on the same cluster, same model, same quantisation. Root cause: tensor parallelism requires an all-reduce after each attention and MLP block; at TP=8, the communication cost between 8 GPUs (even over NVSwitch) grows faster than the speedup of partitioning the compute. The practical rule that emerges: TP=4 + 2 replicas usually delivers better latency/throughput than TP=8 + 1 replica, except for extremely long contexts (≥128k) where you need the aggregate VRAM. Theoretical NVLink 4.0 capacity on the H100 SXM: around 450 GB/s per GPU; typical sustained TP=4 regime: 50–150 GB/s.

Operational implication. If NVLINK_BANDWIDTH_TOTAL > 90 % of capacity is sustained, this is not a problem solvable by raising parallelism. The opposite: lower TP. The metric is orthogonal to the sizing from capacity planning: the ceiling is not only VRAM/time, it is also the bus.

Family 3 — Thermal and power

DCGM_FI_DEV_GPU_TEMP — is the GPU breathing?

The analogy. The body temperature of an elite athlete under full effort. 36–37 °C is normal; 38 °C is sustainable stress; above 39 °C the body activates protective mechanisms (sweating, slowing down) that degrade performance. The GPU does the same: above a thermal threshold, it reduces its clock automatically. If it did not, it would break.

The documented anomaly. The H100 SXM5 with its 700 W TDP has thermal thresholds that are not entirely public (NVIDIA does not publish them exhaustively in the datasheet), but the behaviour is well known: above roughly 85 °C edge or roughly 95 °C HBM, the bit 0x40 HW_THERMAL appears in the clock throttle reasons. Operators on the NVIDIA developer forum report that with a rack inlet temperature above 27 °C, throttling is routine. The NVIDIA HGX Platform paper states that the minimum recommended air flow is > 1000 CFM/kW; densities above 30 kW/rack at 700 W TDP demand mandatory liquid cooling because forced air does not get there.

Operational implication. If GPU_TEMP > 83 °C is sustained, look first at CLOCK_THROTTLE_REASONS (bit 0x40) and at the rack inlet temperature: this is not an engine problem, it is an air flow one. For legacy air-cooled racks, consider redistributing thermal load or installing a rear-door HX.

DCGM_FI_DEV_POWER_USAGE — how much is it asking of the socket?

The analogy. The instantaneous draw of an industrial appliance plugged into a three-phase outlet with a sized breaker. If the washing machine starts at 9 kW and the breaker is 10 kW, you live on the edge; if the washing machine “gets along” with the breaker it is because somebody sized it consciously.

The documented anomaly. A published real measurement: an H100 SXM5 with vLLM running Llama 3.1 405B at batch=4 draws around 697 W at-wall sustained (NVIDIA TDP 700 W). Now the interesting operational lever: lowering nvidia-smi -pl from 700 W to 500 W delivers around 30 % energy savings with only around 20 % loss of throughput. A cluster of 4 nodes × 8 H100 at 700 W equals around 22 kW of GPU alone; at 500 W, around 16 kW. The difference pays an entire quarter’s electricity bill in clusters operated around the clock. A three-phase 415 VAC PDU branch at 60–80 A supports around 32 kW, roughly 4 DGX H100. Legacy 208 V does not support H100 density; reference: NVIDIA DGX SuperPOD Electrical Specifications.

Operational implication. A useful metric for three things: (1) detecting anomalously low workloads (unexpected idle), (2) computing per-tenant showback of real energy cost (not estimates), (3) alerting if the draw approaches the PDU branch limit. Keep GPU → PDU branch → breaker mapped in the CMDB.

DCGM_FI_DEV_CLOCK_THROTTLE_REASONS — who is stepping on the brake?

The analogy. The “limited mode” warning light on the dashboard of a modern car. When it lights up, the car reduces its performance automatically, but it does not tell you why unless you know how to read the combination of letters. The bits of the bitmap are those letters.

The documented anomaly. A public case formally acknowledged by two different OEMs: Dell KB 000220508 and Lenovo HT514380 address the same phenomenon: HW Power Brake Slowdown active (bit 0x80) on the H100 SXM. The cause is not the GPU: it is the chassis PDU sending an electrical power-brake signal because the rack branch is close to the breaker limit. The operator sees throughput down 30–50 % with no XID and no ECC, and the inference engine “is healthy”; the problem is in the electrics. The NVIDIA developer forum thread on “HW Power Brake Slowdown” corroborates the pattern. Bit 0x40 HW_THERMAL appears in badly ventilated racks; bit 0x04 SW_POWER_CAP appears if somebody left nvidia-smi -pl 500 set and nobody reverts it.

Operational implication. Any bit other than 0 or Idle (bit 0x01) sustained is an immediate alert. The recommended decoding: record the complete bitmap value in the log plus the attribute throttle.reasons.decoded=["HW_THERMAL", "HW_POWER_BRAKE"] on the OTel span. Without that, incident response does not know what to do.

Family 4 — Health (the catastrophic reports)

DCGM_FI_DEV_XID_ERRORS — the driver’s red codes

The analogy. The critical alarm lights in a nuclear control room. They do not climb gradually; they either appear or they do not. Each XID is a predefined code (XID 13 = graphics engine exception; XID 31 = MMU fault; XID 43 = stopped channel; XID 79 = GPU fallen off the bus; XID 95 = uncontained ECC), and each one has its documented procedure.

The documented anomaly. The most studied public case: Meta published that during the training of Llama 3 405B on 16,384 H100s over 54 days there were 419 unplanned failures, one every 3 hours at cluster scale. GPUs accounted for 148 (35 %) plus HBM3 for 72 (17 %), almost half of all failures. The paper “Story of Two GPUs: Characterizing the Resilience of Hopper H100 and Ampere A100” (arXiv 2503.11901) quantifies with a different dataset (2.1M GPU-hours) that the H100 has 3.2× worse MTBE for uncorrectable ECC than the A100. The ByteDance MegaScale paper reports that XID 79 (“GPU fallen off the bus”) co-occurs with PCIe errors in 43 % of cases. The NVIDIA developer forum documents persistent cases of XID 31 (MMU fault) that follow the GPU when it is moved to a different PCIe slot, a hardware bug in the module, not in the backplane.

Operational implication. Any increment of the counter is an immediate alert: many XIDs demand a node reset or an RMA of the GPU. The XID-by-XID distinction matters: XID 13/43 is usually a software bug if it coincides with a recent change; XID 31/48/79/94/95 is usually hardware. Keep a canonical xid → procedure table. See the runbooks for the translation into concrete action.

DCGM_FI_DEV_ECC_DBE_VOL_TOTAL — the errors that corrupt data

The analogy. An accounting ledger where sometimes someone erases an entry and rewrites it (corrected single-bit ECC: note a change in the margin and carry on) and sometimes someone burns two pages at once (double-bit: the information is lost, the audit has to stop).

The documented anomaly. The paper “Characterizing GPU Resilience” quantifies for the H100 that when XID 48 (DBE) appears, the job in flight dies with 100 % probability (5 out of 5 in the dataset studied). The documented recovery: drain the node, reset, complete the row remap, equalling around 19 hours of node downtime. HBM3 density explains the worse MTBE against HBM2e: there are more cells per unit area, and therefore a higher statistical probability of degradation. In Llama 3, HBM3 caused 72 of the 419 interruptions (17 %).

Operational implication. Any value > 0 is a critical alert. The GPU must be drained immediately, removed from the scheduler, fully reset, validated for row remap with nvidia-smi -q -d ROW_REMAPPER (Pending: No), and given an extensive smoke test before returning to the pool. If the row remap uses more than 4–8 spare pages on one GPU, plan a replacement in the next window: the degradation is progressive.

DCGM_FI_DEV_RETIRED_DBE — the pages marked for retirement

The analogy. The tiles a museum restorer marks with yellow tape because they are damaged. They pose no immediate danger (the room stays open), but the accumulation says the floor is degrading structurally and the full replacement has to be planned.

The documented anomaly. NVIDIA documents up to 512 spare pages per HBM bank on the H100; the RETIRED_DBE counter indicates how many have been used. Operators on NVIDIA forums report that above 4–8 retired pages on a specific GPU, the frequency of XID 48 rises. The pattern: a GPU with 6 retired pages today becomes 12 in a month, a first XID 48 two months later, and a forced drain.

Operational implication. A trend metric, not an immediate alert. Record the value per GPU and review it monthly; GPUs with rising values go into the proactive replacement plan before the catastrophic failure.

The five vLLM engine metrics

vllm:num_requests_running — how many requests fit in the batch?

The analogy. The number of cars a toll booth lets through at once. If the barrier opens for N at a time, car N+1 waits in the queue. Saturation shows up because the line does not get shorter.

The documented anomaly. Reaching the configured --max-num-seqs and staying there is the typical symptom of a cluster below its sizing; the engine admits up to the ceiling and no further. The query vllm:num_requests_running == max_num_seqs for more than 5 minutes indicates firm saturation.

Operational implication. Combine it with num_requests_waiting: if running is at the ceiling AND waiting > 0, you have to scale. If running is at the ceiling and waiting is 0, you are in the optimal regime (cluster used to the maximum with no queue).

vllm:num_requests_waiting — the primary saturation indicator

The analogy. The visible queue in front of the toll booth. As long as it is empty, the system flows; the moment a sustained queue forms, drivers start arriving late at their destination, and TTFT goes through the roof.

The documented anomaly. A public case in “11-Second Time to First Token on a Healthy vLLM Server” (Medium, Ingero, 2026): a server with no XIDs, no preemption, DCGM metrics all green, but num_requests_waiting sustained above 0 and a TTFT of 11 seconds. Issue vllm#16985 documents progressive degradation in long sessions: the queue grows slowly over hours without any other indicator moving. The root cause is not hardware, it is admission control: the arrival rate exceeds the completion rate and the system does not enqueue any more, it leaves requests in WAITING. Red Hat designates it as the primary saturation metric in its tutorial “5 steps to triage vLLM performance”.

Operational implication. The primary HPA metric in KEDA, see Autoscaling LLM on Kubernetes. Typical threshold: alert if avg_over_time(vllm:num_requests_waiting[5m]) > 5. For canaries: if the queue only forms in the v2 pool, it is a regression of the new model, not cluster load.

vllm:gpu_cache_usage_perc — the KV cache pool

The analogy. The capacity of an events hall where each guest takes up a variable amount of space. The maître d’ admits guests up to capacity; when a new guest arrives and there is no room, he throws out the guest who has been there longest to make space for the newcomer. That is vLLM’s preempt-on-OOM.

The documented anomaly. Issue vllm#5051 “Add num_requests_preempted metric” was born exactly out of operators observing degradation with no direct metric telling them how many requests were being thrown out. The official vLLM documentation confirms: “sustained gpu_cache_usage_perc above 90 % indicates the server is approaching its KV cache limit and will begin preempting sequences” (oldest-first). The distinctive visual pattern: a sawtooth near 100 % with preemption spikes. In swap mode, the latency of the preempted request explodes because there is a host↔device PCIe copy; in recompute mode (the default in V1), the preempted request redoes its prefill from scratch, which pushes its TTFT to double or triple.

Operational implication. If gpu_cache_usage_perc > 92 % is sustained, there are two levers: lower max_num_seqs (admit less concurrency, but none of it gets preempted) or raise gpu_memory_utilization (more pool, less VRAM for transient activations, a different risk). The choice depends on the workload. The metric that is directly missing, a preempted counter, is exported from vLLM v1.0 onwards as vllm:num_preemptions_total (see issue #5051).

vllm:time_to_first_token_seconds — the latency the client sees

The analogy. The time from a customer walking into a restaurant to receiving the first piece of bread at the table. Too long and the customer thinks they have been forgotten, even if the main course is going to arrive perfect.

The documented anomaly. Three documented patterns of recurring TTFT spikes:

1. Badly calibrated chunked prefill. Issue vllm#25677 (Qwen3-30B-A3B) reported prefill 10–11× slower with chunked prefill enabled than without it. Cause: a very low max_num_batched_tokens forces small chunks that do not fill the kernels. Issue vllm#7604 documents an equivalent regression on Llama-3-70B v0.5.4. The lever: raise max_num_batched_tokens to 4096–8192 for typical prompts above 2k.
2. Regression between engine versions. Issue vllm#8819 documents a regression of vllm:time_to_first_token_seconds_sum between minor versions. Issue vllm#11912 reports that with a prompt of around 8000 tokens, TPOT rose from 15.7 ms → 25.7 ms from v0.6.4.post1 onwards with no config change, a regression confirmed and trackable only with the metric.
3. Long-context prefill blocking decodes. The “11s TTFT on healthy server” case cited above: a 30k-token prefill monopolises the GPU for several seconds and the active decodes freeze. Fix: well-calibrated chunked prefill, or disaggregated serving (see Disaggregated serving).

Operational implication. Do not alert only on the absolute P95; alert also on the v2/v1 ratio when there is a canary (histogram_quantile(0.95, ..., version="v2") / histogram_quantile(0.95, ..., version="v1") > 1.10). If TTFT grows and the queue is stable, the bottleneck is prefill: not solvable by adding replicas, but there are levers in quantisation or chunked prefill.

vllm:time_per_output_token_seconds — streaming smoothness

The analogy. The speed at which the waiter brings the courses one after another after the first. If the next one takes a while, the diner notices something is wrong even though the first course arrived on time.

The documented anomaly. The distinctive pattern is the abrupt step when gpu_cache_usage_perc crosses roughly 85 %: TPOT jumps from 35 ms to 80 ms in a few seconds because the engine starts competing for the HBM with its own evictions. Issue vllm#35387 documents another anomalous case: MTP (speculative decoding) causing a 76 % latency regression on Qwen3-Next-80B-A3B-Instruct-FP8. The TPOT metric caught it before any client complaints were filed.

Operational implication. The difference from TTFT: if TTFT grows and Queue Time is stable, you are prefill bound; if TPOT grows at a steady rate, there is pressure on the HBM (KV cache pool or swap enabled). A secondary alert on the TPOT SLO, but also watch the derivative: TPOT rising 1 ms every 10 minutes is a latent regression that does not break the SLO yet but will.

The operational rule: read metrics by family, not in isolation

Three anti-patterns of the novice operator

Anti-pattern 1 — alerting only on absolute thresholds. An H100 at 87 % FB is not necessarily an alarm; an H100 at 87 % rising 2 %/min is. Alerts that fire on a fixed threshold without looking at the derivative produce twice the noise and half the value. Rule: for metrics with known dynamics (KV cache, FB, queue), alert on a sustained delta, not just on the level.

Anti-pattern 2 — confusing SBE with DBE. The DCGM_FI_DEV_ECC_SBE_VOL_TOTAL counter (single-bit, correctable) grows continuously on any HBM under load; it is not an alarm, it is physics. The one that matters is DCGM_FI_DEV_ECC_DBE_VOL_TOTAL (double-bit, uncorrectable). Confusing them yields false negatives (not alerting on a real DBE) or false positives (alerting on a harmless SBE).

Anti-pattern 3 — treating SM_OCCUPANCY at 99 % as “saturated”. The LLM decode regime is memory-bound, not compute-bound; high SM occupancy with low TENSOR_ACTIVE and high DRAM_ACTIVE is normal. Sizing for “GPU at 60 %” and asking for more hardware when the cluster is saturated in HBM (not in SMs) means buying twice the GPUs without gaining throughput. Rule: always read SM_OCCUPANCY together with TENSOR_ACTIVE and DRAM_ACTIVE; on its own it means nothing.

Applied to typical on-premise hardware

For a generic cluster of 4 nodes × 4×H100 SXM 80 GB with intra-node NVLink:

• DCGM Exporter per node (a GPU Operator DaemonSet) emitting every 15 s; cardinality per GPU is around 80 series. A 16-GPU cluster gives roughly 1.3k base series, around 85k samples/min with a 15 s scrape.
• vLLM /metrics per inference pod; each replica emits around 50 base series. For 16 replicas, around 800 additional series, around 3k samples/min.
• Prometheus retention: 30 days at high resolution plus 1 year downsampled through a Thanos sidecar or Mimir. Estimated volume: 25–35 GB/day.
• Alertmanager: the 6 critical alerts from the previous post plus derived alerts (delta, v2/v1 ratio, decoded throttle bitmap).

Each metric is worth exposing also as an OTel attribute on the spans of the GenAI tracing: gpu.fb_used_pct, gpu.dram_active, gpu.throttle_reasons.decoded. That lets you correlate a slow request with the state of the GPU at that instant, without jumping between dashboards.

What we have not covered (upcoming posts)

• Per-alert runbooks — the translation of each anomalous metric into concrete action (drain, reset, RMA, scaling, rollback) in the next post: Incident response runbooks.
• Tail sampling for metric ↔ trace correlation — what is preserved when an alert fires, for post-mortem investigation.
• Per-tenant showback combining vllm:request_success_total × gen_ai.usage.* × DCGM_FI_DEV_POWER_USAGE to bill real energy cost.
• Multi-tenant fairness metrics — when a tenant hogs the KV cache pool and how to detect it.

See also

• GPU observability for LLM inference — the compact list that this post goes deeper into.
• Incident response runbooks for LLM with Keep + Kafka — the translation of each anomaly into action.
• LLM tracing with OpenTelemetry GenAI — the other half of observability.
• Autoscaling LLM on Kubernetes — num_requests_waiting and gpu_cache_usage_perc as primary HPA metrics.
• Capacity planning for on-premise LLM inference — how the thresholds relate to the sizing.
• Canary, blue-green and shadow — the v2/v1 TTFT ratio as a gate.
• Continuous batching — explains preempt-on-OOM and the KV pool sawtooth.
• KV cache — underpins the calculation of gpu_cache_usage_perc.

References

• Meta — Faulty Nvidia H100 GPUs and HBM3 memory caused half of failures during Llama 3 training (Tom’s Hardware, 2024). tomshardware.com
• Story of Two GPUs: Characterizing the Resilience of Hopper H100 and Ampere A100. arXiv 2503.11901. https://arxiv.org/html/2503.11901v3
• ByteDance — Robust LLM Training Infrastructure at ByteDance. arXiv 2509.16293. https://arxiv.org/pdf/2509.16293
• Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inference. arXiv 2503.08311.
• Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of Experts. arXiv 2503.05066.
• NVIDIA — Analyzing Xid Errors with the Xid Catalog and Memory Error Management (docs.nvidia.com/deploy).
• Dell — PowerEdge XE8640 with H100 - GPU Performance Issue HW Power Brake Slowdown - Active (KB 000220508).
• Lenovo — Power brake reporting on H100 GPU (HT514380).
• vLLM project — issues #5051 (preempted metric), #7604 and #25677 (chunked prefill regression), #11912 (long-prompt regression), #16300 (TP=8 worse than TP=4), #16985 (long-running degradation), #20783 (compressed-tensors no speedup), #35387 (MTP regression).
• Red Hat — 5 steps to triage vLLM performance. https://developers.redhat.com/articles/2026/03/09/5-steps-triage-vllm-performance
• AI21 — Go big or go OOM: the art of scaling vLLM. https://www.ai21.com/blog/scaling-vllm-without-oom/
• 11-Second Time to First Token on a Healthy vLLM Server (Medium, Ingero, 2026).
• NVIDIA — DGX SuperPOD Electrical Specifications (docs.nvidia.com/dgx-superpod).

Sources: the full URLs are linked inline on each reference.