Incident response runbooks for LLM inference: every alert to a concrete action with Kafka and Keep

Contents

This post closes the observability trilogy opened by GPU observability for LLM inference (which metrics) and Anatomy of the twelve DCGM and five vLLM metrics (which documented anomaly per metric). Here each anomaly gets its concrete action and is fitted into the incident management machinery that compliance demands.

TL;DR

The GPU observability alerts are useless without a codified procedure for each one; the operator who interprets them by hand every time is operating on intuition. The right combination has three indispensable pieces. (1) Runbook catalogue: for each of the six critical alerts (GpuHbmNearOom, GpuThermalOrPowerThrottle, GpuXidErrorDetected, GpuEccDoubleBit, VllmKvCachePoolNearFull, VllmTtftP95OutOfSlo), a severity, an immediate mitigation, the evidence to capture before remediating, a resolution action, a closure criterion and a postmortem trigger. (2) Reproducible pipeline: Prometheus + DCGM → Alertmanager → Kafka as event bus (topics gpu.alerts.enriched, incidents.lifecycle, audit.actions with WORM retention) → Keep as workflow engine (declarative YAML workflows versioned in git) → Kubernetes job / script / ChatOps executors. (3) Formal fit into incident management under the regulatory corpus: ISO/IEC 27035 phases identify → report → assess → respond → learn; ENS controls op.exp.7 (incident management), op.exp.8 (activity logging), op.exp.10 (notification to users); NIS2 art. 23 with an early warning at 24 h, a formal notification at 72 h and a final report at 1 month; EU AI Act art. 73 for a serious incident on a high-risk system, deadlines of 2 to 15 days depending on severity; ISO/IEC 42001 clause 10 (continual improvement of the AIMS). The action taxonomy is immediate mitigation (drain, throttle, scale-down: contains the damage in seconds) → diagnosis (evidence capture with nvidia-smi -q, dmesg, a vLLM /metrics snapshot, the related OTel trace; without this the postmortem is not defensible) → resolution (restart, reset, RMA, rollback) → postmortem (5-whys RCA, prevention plan, runbook update). Kafka contributes the immutable audit trail that ENS and the EU AI Act demand: every action executed by Keep or by a human is published as an event on audit.actions with timestamp, actor, decision and evidence, retained WORM for at least 6 months. Keep contributes workflows as code: this post includes three complete workflows (XID with drain plus Jira ticket, ECC DBE with immediate paging and node blocking in the scheduler, automatic canary rollback on TTFT P95 out of SLO). Four anti-patterns close the material: alerts without a runbook (most of them), a runbook without prior evidence capture (it perpetuates the incident because the root cause is lost), escalation by seniority rota instead of severity (a junior operator handling an ECC DBE), and the absence of a human gate for destructive actions (Keep running nvidia-smi --gpu-reset without confirmation). Applicable to a generic 4×H100 SXM cluster with Kafka and Keep already deployed.

You are here: OBSERVE → DEPLOY (incident response closes the loop)

The analogy: a nuclear reactor control room

In a nuclear plant control room, the operator on shift never decides what to do on seeing an alarm. The decision is pre-made and codified in a written procedure (SOP) covering every alarm on the panel: if X sounds, open book X, read steps 1-N, execute exactly, call the supervisor at step M, escalate to the plant director at step N+3. The reason is strict: critical alarms are rare but catastrophic if handled badly; an operator improvising in an emergency makes worse decisions than one applying a procedure reviewed by experts and validated by simulation.

The reactor does not expect the operator to be a genius. It expects them to know the procedures to the letter and the operations management system to hand them the right procedure at the right moment. If the procedures are not written, not versioned, or not integrated with the alarms that fire them, the control room is operating on intuition. The difference between the two operations, procedural versus intuitive, is the difference between a plant that runs 30 years without incidents and one that ends up on a blacklist.

Incident response on an LLM inference cluster works identically. The DCGM and vLLM alerts listed in the previous posts are the panel alarms. Each one needs its written SOP, versioned, integrated with the alert that fires it and reviewed after every incident. Without that codification, the operator on shift improvises in the middle of an ECC DBE failure at 4 in the morning; with it, they execute the nine steps of runbook 12 and the incident closes in 20 minutes.

The incident pipeline architecture

Prometheus + DCGM. Collects the metrics described in the two previous posts. PrometheusRules define the six critical alerts with for: <duration> to avoid noise.

Alertmanager. Receives raw alerts; deduplicates, groups by labels ({cluster, node, gpu, model}), routes. Instead of sending straight to PagerDuty or Slack, it sends to Kafka via a webhook receiver. That turns the alert into a bus event that multiple consumers process (Keep for action, the audit topic for compliance, dashboards for visualisation).

Kafka as event bus. Three canonical topics:

gpu.alerts.enriched — alerts with added context (tenant, model, version, namespace owner, effective severity). Retention: 7 days, replication factor 3.
incidents.lifecycle — incident cycle events: incident.opened, incident.acknowledged, action.proposed, action.executed, incident.escalated, incident.resolved, postmortem.attached. Retention: 90 days.
audit.actions — an immutable record of every action executed (by Keep automatically or by a human confirming). Retention: 6 months minimum with compaction off plus tiered storage, WORM storage. This is the topic ENS op.exp.8, EU AI Act art. 12 and NIS2 require you to keep.

Keep as workflow engine. Consumes from gpu.alerts.enriched, fires YAML workflows versioned in git, executes actions (HTTP calls, kubectl jobs, Slack messages, Jira tickets) and publishes the result to incidents.lifecycle plus audit.actions. Choosing Keep over Alertmanager alone (or over PagerDuty alone) is deliberate: Keep separates runbook declaration (readable, reviewable YAML) from notification delivery (PagerDuty). The runbook is versioned code; notifications are operational details.

Executors. What actually moves the cluster:

Kubernetes jobs: kubectl drain, kubectl cordon, kubectl rollout undo.
NVIDIA API: nvidia-smi --gpu-reset, dcgmi diag -r <level>.
ChatOps: human confirmations through Slack interactive messages before a destructive action runs.
External tooling: Jira ticket, PagerDuty notification, CMDB call.

The six critical alerts and their runbooks

For each alert: severity, immediate mitigation (seconds), evidence to capture before remediating, resolution action, closure criteria, postmortem trigger.

RB-01 · `GpuHbmNearOom` — HBM > 92 % sustained

Severity: WARNING. Risk of OOM on the next PagedAttention allocation.

Immediate mitigation. Reduce admission temporarily by lowering max_num_seqs on the affected engine via hot reload (if the engine supports it) or a staggered restart of replicas. Trigger an additional scale-out via KEDA if there are free GPU nodes. There is no need to drain the node.

Evidence to capture.

nvidia-smi --query-gpu=index,memory.used,memory.free,memory.total --format=csv
nvidia-smi -q -d ROW_REMAPPER | grep -i pending
curl http://vllm-pod:8000/metrics | grep -E "gpu_cache_usage|num_requests"
kubectl logs <pod> --tail=200 | grep -i "preempt\|swap"

Save the snapshot to audit.actions with a timestamp and an incident_id.

Resolution. If the cause is a traffic spike: let the autoscaler scale to a stable regime, monitor for 30 min. If the cause is a model regression (canary v2 consumes more KV cache than v1): roll the canary back (see RB-06). If it is a leak (the metric grows without traffic growing): restart the pod with a heap dump capture.

Closure. gpu_cache_usage_perc < 80 % sustained for 15 min AND num_requests_waiting == 0.

Postmortem. Not mandatory unless the incident lasted > 30 min or had SLO impact.

RB-02 · `GpuThermalOrPowerThrottle` — bit ≠ 0 and not Idle in CLOCK_THROTTLE_REASONS

Severity: WARNING (thermal) or CRITICAL (sustained HW Power Brake, PDU risk).

Immediate mitigation. Identify the bit (decode the bitmap). If it is 0x40 HW_THERMAL or 0x20 SW_THERMAL: drain the workload from the node to other replicas if the temperature does not drop within 2 min, and stop new pods landing on that node (kubectl cordon). If it is 0x80 HW_POWER_BRAKE: alert DC infrastructure immediately (likely an over-committed PDU, the Dell KB 000220508 / Lenovo HT514380 case) and lower the TDP of the rack’s GPUs via nvidia-smi -pl to a lower value to relieve load on the breaker.

Evidence.

nvidia-smi --query-gpu=index,temperature.gpu,temperature.memory,power.draw,clocks_throttle_reasons.active --format=csv
ipmitool sdr | grep -i "fan\|temp\|inlet"
# PDU data if they are instrumented (modbus / SNMP)

Resolution.

Thermal: review rack airflow, check the rear-door HX, T_inlet, DGX fans. An infra issue, not an engine one.
Power Brake: review the sizing of the PDU branch, the breaker, the 415 VAC distribution. Likely a redistribution of load to another branch or a temporary TDP cap.

Closure. CLOCK_THROTTLE_REASONS == 0x1 (Idle only) or 0x0 for 30 min under normal load.

Postmortem. Mandatory if it was a HW Power Brake: that implicates the DC electrical infrastructure.

RB-03 · `GpuXidErrorDetected` — `increase(DCGM_FI_DEV_XID_ERRORS[5m]) > 0`

Severity: CRITICAL.

Immediate mitigation. kubectl cordon the node (no more new pods). If the XID is 31/48/79/94/95 (hardware or cascade): drain the node’s existing pods. If the XID is 13/43 (possibly software): keep the pods but block new ones, capture the trace and the active workload.

Evidence.

# The specific XID from dmesg
dmesg | grep -i xid | tail -30
nvidia-smi -q -d ERROR
nvidia-smi -q -d PCIE
# State of the retired pages
nvidia-smi -q -d ROW_REMAPPER
# Workload that was running
kubectl get pods -o wide | grep <node>
kubectl logs <pod> --previous --tail=500

Resolution.

XID 13/43 (software exception / channel verif): if it recurs only with one specific model, it is a workload bug; raise an issue with the model team. If it is transient, restarting the pod is enough.
XID 31 (MMU fault): usually a cascade from a previous XID 48. Reset the GPU (nvidia-smi --gpu-reset -i <index>) or reboot the node if the reset does not resolve it.
XID 48 / 95 (DBE / uncontained ECC): see RB-04. The node goes into quarantine.
XID 79 (fallen off the bus): reboot the node. If it recurs after the reboot, open an RMA for the GPU. ByteDance reports 43 % co-occurrence with PCIe errors, so check the slot and the cable too.
XID 94 / 145 / 149: catalogued in NVIDIA’s Xid Catalog with a specific procedure.

Closure. Node smoke test passed (dcgmi diag -r 3), 24 h with no new XIDs, back into the pool.

Postmortem. Mandatory. Include the specific XID, the distribution of XIDs across the cluster, an updated MTBE.

RB-04 · `GpuEccDoubleBit` — `DCGM_FI_DEV_ECC_DBE_VOL_TOTAL > 0`

Severity: CRITICAL, data corruption in progress.

Immediate mitigation. Drain the node immediately without waiting for further evidence. Page (PagerDuty / OpsGenie) the primary ON-CALL. Mark the node unschedulable and failed. XID 48 has a 100 % probability of killing the running job according to the Story of Two GPUs dataset; any inference already in flight is compromised.

Evidence (in parallel with the mitigation).

nvidia-smi -q -d ECC
nvidia-smi -q -d ROW_REMAPPER     # Pending: Yes expected
dmesg | grep -E "Xid.*48|DBE|double-bit" | tail -50
# Full capture of the GPU state
dcgmi diag -r 4 -i <gpu_index>

Resolution. Full GPU reset (nvidia-smi --gpu-reset) or a node reboot if the reset does not complete. The reset activates the row remap. After the reboot:

nvidia-smi -q -d ROW_REMAPPER     # Pending: No expected
nvidia-smi -q -d ECC               # volatile counters at 0

If RETIRED_DBE > 8 pages after the remap: schedule a GPU replacement in the next window, because silicon degradation is progressive. The published real case documents ~19 hours of downtime as typical.

Closure. Node back in the pool after 48 h with no new DBEs.

Postmortem. Mandatory. If the incident affected a request carrying personal or classified data, evaluate notification to the DPO under GDPR art. 33 (it is not necessarily a breach, but it has to be assessed).

RB-05 · `VllmKvCachePoolNearFull` — `gpu_cache_usage_perc > 95 %` sustained 3 min

Severity: WARNING (risk of preempt-on-OOM, not of a real OOM).

Immediate mitigation. Activate autoscaler scale-out by lowering the KEDA threshold temporarily (from 0.85 to 0.75) for 30 min. In recompute mode, preempts raise TTFT but do not break requests; acceptable in the short term. In swap mode latency goes through the roof, so it is better to cut new traffic (return 503 from the router) for 5 min.

Evidence.

curl http://vllm-pod:8000/metrics | grep -E "gpu_cache|num_requests|num_preemptions"
kubectl get hpa vllm-llama70b
kubectl logs <pod> --tail=200 | grep -i preempt

Resolution. If it recurs regularly: revisit capacity planning, possibly lowering max_num_seqs or raising the stable replica count. See Capacity planning.

Closure. Pool < 85 % sustained 30 min, no preempts in the last 15 min.

Postmortem. Not mandatory unless it recurs > 3 times / week.

RB-06 · `VllmTtftP95OutOfSlo` — TTFT P95 > 1.5 s for 5 min

Severity: CRITICAL (contractual SLO violation).

Immediate mitigation. Quick diagnosis of the regime (in order of likelihood):

If a v2 canary is active and the ratio ttft_p95(v2)/ttft_p95(v1) > 1.30: automatic rollback of the canary via Argo Rollouts (argo rollouts abort vllm-llama70b).
If num_requests_waiting > 5: scale out via KEDA.
If DRAM_ACTIVE > 90 % plus gpu_cache_usage_perc > 90 %: HBM bottleneck, reach for the quantisation lever or a context reduction.
If CLOCK_THROTTLE_REASONS != 0: see RB-02.

Evidence.

# Histogram snapshot
curl http://vllm-pod:8000/metrics | grep time_to_first_token
# Breakdown by version if a canary is running
# DCGM state at that moment
curl http://dcgm-exporter:9400/metrics | grep -E "PIPE_TENSOR|DRAM_ACTIVE|THROTTLE"
# Active traffic
kubectl top pods -n inference

Resolution. Depends on the diagnosis. Typical cases:

Canary regression → full rollback (see Canary).
Capacity saturation → scale replicas or accept a temporary 503 with Retry-After.
Prefill bound → enable or calibrate chunked prefill or disaggregated serving (see Disaggregated serving).

Closure. TTFT P95 inside SLO sustained 30 min.

Postmortem. Mandatory. Document the root cause and the lever applied; update the runbook.

Keep YAML workflows — three complete examples

Runbooks are only useful if they are codified in the workflow engine. Keep lets you declare them in YAML versioned in git.

Workflow 1 — `xid-detected.yaml`

workflow:
  id: xid-detected-drain
  name: "XID error detected — cordon node and capture evidence"
  description: "RB-03 implementation"
  triggers:
    - type: alert
      filters:
        - key: alertname
          value: GpuXidErrorDetected
  steps:
    - name: capture-evidence
      provider:
        type: bash
        with:
          command: |
            set -e
            NODE="{{ alert.labels.node }}"
            GPU="{{ alert.labels.gpu }}"
            INC_ID="{{ alert.fingerprint }}"
            mkdir -p /var/evidence/$INC_ID
            kubectl debug node/$NODE -it --image=nvcr.io/nvidia/cuda:12.4.0-base-ubuntu22.04 -- \
              bash -c "nvidia-smi -q -d ERROR,PCIE,ROW_REMAPPER > /host/var/evidence/$INC_ID/smi.txt"
            kubectl describe node $NODE > /var/evidence/$INC_ID/node.txt            
    - name: cordon-node
      provider:
        type: kubernetes
        with:
          action: cordon
          name: "{{ alert.labels.node }}"
      if: "{{ alert.labels.severity == 'critical' }}"
  actions:
    - name: open-jira-ticket
      provider:
        type: jira
        config: "{{ providers.jira-prod }}"
        with:
          project: GPUOPS
          issuetype: Incident
          summary: "RB-03: XID {{ alert.annotations.xid_code }} on {{ alert.labels.node }}/{{ alert.labels.gpu }}"
          description: |
            Severity: {{ alert.labels.severity }}
            XID: {{ alert.annotations.xid_code }}
            Evidence: /var/evidence/{{ alert.fingerprint }}
            Runbook: https://runbooks.example.local/RB-03            
    - name: notify-slack
      provider:
        type: slack
        config: "{{ providers.slack-gpu-incidents }}"
        with:
          message: |
            :warning: *RB-03 triggered*
            Node: `{{ alert.labels.node }}` GPU: `{{ alert.labels.gpu }}`
            XID: `{{ alert.annotations.xid_code }}`
            <{{ jira.url }}|Jira ticket>            
    - name: emit-audit
      provider:
        type: kafka
        config: "{{ providers.kafka-audit }}"
        with:
          topic: audit.actions
          message:
            incident_id: "{{ alert.fingerprint }}"
            action: "cordon_node"
            actor: "keep-workflow"
            workflow_id: "xid-detected-drain"
            target: "{{ alert.labels.node }}"
            timestamp: "{{ now }}"

Workflow 2 — `ecc-dbe.yaml` — immediate paging

workflow:
  id: ecc-dbe-critical
  name: "ECC double-bit — page on-call and quarantine node"
  triggers:
    - type: alert
      filters:
        - key: alertname
          value: GpuEccDoubleBit
  steps:
    - name: cordon-immediately
      provider:
        type: kubernetes
        with:
          action: cordon
          name: "{{ alert.labels.node }}"
    - name: drain-workload
      provider:
        type: kubernetes
        with:
          action: drain
          name: "{{ alert.labels.node }}"
          options:
            ignore-daemonsets: true
            delete-emptydir-data: true
            grace-period: 120
    - name: page-oncall
      provider:
        type: pagerduty
        config: "{{ providers.pagerduty-critical }}"
        with:
          service_key: "{{ env.PD_SERVICE_KEY }}"
          severity: critical
          summary: "RB-04 ECC DBE on {{ alert.labels.node }}/{{ alert.labels.gpu }} — node drained"
    - name: emit-lifecycle
      provider:
        type: kafka
        config: "{{ providers.kafka-incidents }}"
        with:
          topic: incidents.lifecycle
          message:
            incident_id: "{{ alert.fingerprint }}"
            event: incident.opened
            severity: critical
            runbook: RB-04
            requires_postmortem: true
    - name: notify-dpo
      provider:
        type: email
        with:
          to: dpo@example.local
          subject: "ECC DBE on a production GPU — assessment required"
          body: |
            RB-04 ECC DBE incident detected on {{ alert.labels.node }}.
            Affected model: {{ alert.labels.model }}.
            Please assess whether personal/classified data was processed
            during the error window and whether GDPR art. 33 notification is needed.

Workflow 3 — `canary-rollback.yaml` — TTFT P95 out of SLO

workflow:
  id: canary-rollback-ttft
  name: "Rollback canary when TTFT P95 ratio v2/v1 > 1.30"
  triggers:
    - type: alert
      filters:
        - key: alertname
          value: VllmTtftP95OutOfSlo
        - key: canary_active
          value: "true"
  steps:
    - name: check-ratio
      provider:
        type: prometheus
        config: "{{ providers.prom-prod }}"
        with:
          query: |
            histogram_quantile(0.95, sum by(le)(rate(vllm:time_to_first_token_seconds_bucket{version="v2"}[5m])))
            /
            histogram_quantile(0.95, sum by(le)(rate(vllm:time_to_first_token_seconds_bucket{version="v1"}[5m])))            
      condition: result > 1.30
  actions:
    - name: argo-rollback
      provider:
        type: kubernetes
        with:
          action: exec
          command:
            - kubectl
            - argo
            - rollouts
            - abort
            - "{{ alert.labels.rollout }}"
            - -n
            - "{{ alert.labels.namespace }}"
    - name: notify-and-audit
      provider:
        type: kafka
        config: "{{ providers.kafka-audit }}"
        with:
          topic: audit.actions
          message:
            incident_id: "{{ alert.fingerprint }}"
            action: canary_rollback
            ratio: "{{ steps.check-ratio.result }}"
            actor: keep-workflow
            timestamp: "{{ now }}"

Each workflow lives in repos/keep-workflows/ versioned in git, reviewed by pull request, validated by CI (keep workflow validate). The written runbook lives as docs/runbooks/RB-XX.md linked from the workflow YAML; the two always evolve together.

The canonical Kafka event schema

For the topics to be consumable by compliance, postmortem tooling and dashboards without each consumer having to guess the shape, the schema is fixed with Avro / Protobuf.

{
  "name": "IncidentLifecycleEvent",
  "type": "record",
  "fields": [
    { "name": "incident_id",    "type": "string"  },
    { "name": "event",          "type": { "type": "enum", "symbols": [
      "incident.opened", "incident.acknowledged", "action.proposed",
      "action.executed", "action.failed", "incident.escalated",
      "incident.resolved", "postmortem.attached"
    ]}},
    { "name": "timestamp",      "type": "string", "logicalType": "timestamp-millis" },
    { "name": "actor",          "type": "string" },
    { "name": "severity",       "type": { "type": "enum", "symbols": ["low","warning","critical"] } },
    { "name": "runbook",        "type": ["null","string"], "default": null },
    { "name": "alert_name",     "type": "string" },
    { "name": "labels",         "type": { "type": "map", "values": "string" } },
    { "name": "annotations",    "type": { "type": "map", "values": "string" } },
    { "name": "evidence_uri",   "type": ["null","string"], "default": null },
    { "name": "requires_postmortem", "type": "boolean", "default": false }
  ]
}

For audit.actions (WORM), a separate and stricter schema with non-modifiable fields:

{
  "name": "AuditAction",
  "type": "record",
  "fields": [
    { "name": "incident_id",    "type": "string" },
    { "name": "action",         "type": "string" },
    { "name": "actor",          "type": "string" },
    { "name": "actor_type",     "type": { "type": "enum", "symbols": ["human","workflow","scheduler"] } },
    { "name": "workflow_id",    "type": ["null","string"], "default": null },
    { "name": "target",         "type": "string" },
    { "name": "command",        "type": ["null","string"], "default": null },
    { "name": "result",         "type": { "type": "enum", "symbols": ["success","failure","partial"] } },
    { "name": "timestamp",      "type": "string", "logicalType": "timestamp-millis" },
    { "name": "evidence_uri",   "type": ["null","string"], "default": null },
    { "name": "approver",       "type": ["null","string"], "default": null }
  ]
}

The topic is configured with cleanup.policy=delete, retention.ms=15552000000 (6 months) and min.insync.replicas=2 with acks=all to guarantee durability. For longer retention without the Kafka cost, use tiered storage to Ceph RGW or an S3-compatible backend: new log in the hot tier, old log in the cold tier, transparently to the consumer.

Formal fit into incident management

Runbooks are not an isolated SRE practice. They fit into four regulatory frameworks that production LLM platforms touch daily.

ISO/IEC 27035 — information security incident management

It defines the formal cycle in five phases: plan & prepare → detect & report → assess & decide → respond → lessons learned. Each phase has outputs that must be documented. Translated to the stack:

Plan & prepare: runbooks RB-01 to RB-06 plus the Keep workflows are part of the Information Security Incident Management Plan. Versioned in git, reviewed annually.
Detect & report: the Prometheus alerts entering Kafka are the materialisation.
Assess & decide: the severity in gpu.alerts.enriched plus the Keep workflow logic.
Respond: execution of the workflow’s steps plus actions.
Lessons learned: a mandatory postmortem for the runbooks that flag it; the output documented in the postmortem repo plus the runbook update.

ENS (Esquema Nacional de Seguridad) — op.exp controls

op.exp.7 Incident management: the runbook catalogue plus the Keep / Kafka pipeline materialise the “organised and procedural response”.
op.exp.8 Activity logging: the audit.actions topic with WORM retention of 6 months (minimum for level ALTO).
op.exp.9 Incident management logging: the incidents.lifecycle topic with the full cycle of each incident.
op.exp.10 Protection of activity logs: WORM plus encryption at rest plus access control (compliance consumers read-only).

NIS2 — notification to the competent authority

For essential and important entities, art. 23 sets three deadlines from the detection of significant impact:

24 hours: an early warning to the national CSIRT (INCIBE-CERT in Spain).
72 hours: a formal notification with an initial assessment.
1 month: a final report with root cause, impact and corrective measures.

The data for those reports comes straight out of incidents.lifecycle plus audit.actions with a consumer that generates the dossier in the required format. Without the auditable pipeline, the NIS2 deadlines are unreachable.

EU AI Act — art. 73 (serious incident reporting)

Applicable to high-risk systems. Deadlines:

2 days: for incidents causing death or irreversible harm to people or critical infrastructure.
10 days: for incidents producing serious disruption of critical infrastructure.
15 days: for all other “serious incidents”.

The definition of “serious incident” includes systematic model failures, a fundamental rights breach, material or environmental harm. Runbooks must flag which alerts can lead to a serious incident (typically anything affecting the model’s output in a high-risk context) and fire a specific legal assessment sub-workflow.

ISO/IEC 42001 — AIMS clause 10 continual improvement

The mandatory post-incident postmortem feeds clause 10. Updating the runbook after every incident that reveals a new pattern is the “corrective action with verification of effectiveness” the standard demands. See ISO 42001 AIMS.

Four anti-patterns

Anti-pattern 1 — alerts without a runbook. The alert fires, the junior operator on call looks at the dashboard, searches Confluence, finds nothing up to date, calls the senior on Slack, waits 20 minutes. In that time the incident has grown. Rule: no alert goes to production without a published runbook and an approved Keep workflow. CI validates that every PrometheusRule with severity ≥ warning has its corresponding keep workflow.

Anti-pattern 2 — a runbook without prior evidence capture. The workflow runs nvidia-smi --gpu-reset as soon as the XID arrives, losing the state that would have diagnosed the root cause. The next identical XID forces the diagnosis to be redone from scratch. Rule: steps before actions; all evidence is captured first, destructive actions come after.

Anti-pattern 3 — escalation by seniority rota instead of severity. The junior operator on call handles an ECC DBE because “it is their turn”. They lack the context to understand row remap, retired pages or the data corruption risk. Rule: page by severity, not by rota: RB-04 and RB-03 page the senior primary ON-CALL with automatic escalation to infra/hardware if there is no acknowledgement within 10 min.

Anti-pattern 4 — no human gate for destructive actions. The workflow runs kubectl drain automatically on any alert marked CRITICAL. On the first false alarm (a transient that resolved itself in 30 s), Keep drained a production node during peak hour. Rule: destructive actions (drain, reset, RMA, full rollback) require human confirmation via a Slack interactive message, with a configurable timeout. A justified exception: ECC DBE confirmed by > 1 measurement, where the corruption risk outweighs the false alarm risk.

Applied to typical on-premise hardware

For a generic cluster of 4 nodes × 4×H100 SXM 80 GB with Kafka and Keep already deployed:

Kafka: a 3-broker cluster on non-GPU nodes of the K8s cluster; topics gpu.alerts.enriched, incidents.lifecycle, audit.actions configured with replication factor 3, min.insync.replicas 2. Audit with tiered storage to Ceph RGW for retention > 6 months without a brutal cost.
Keep: 2 operator replicas plus 1 worker replica in a keep namespace; connected to Prometheus (read provider), Kafka (read plus write provider), Slack, PagerDuty, Jira, Kubernetes (provider with a dedicated SA holding get/list/patch nodes and create jobs permissions).
Workflows: ~25-40 YAML files in the infra/keep-workflows/ repo, synchronised with the cluster via Flux or Argo CD. Validated by CI (keep workflow validate) on every PR.
Event volume: for 16 GPUs in normal operation with debounced alerts, ~50-200 events/day on gpu.alerts.enriched. In a typical incident, peaks of 500-2,000 events/day.
Compliance consumers: a python consumer in the compliance namespace that generates NIS2 / ENS / EU AI Act reports weekly, reading audit.actions and incidents.lifecycle.

What we have not covered (upcoming posts)

Postmortem playbooks — the mechanics of RCA with 5-whys, Ishikawa adapted to LLMs, integration with MLflow tracking for re-training if the postmortem produces an enriched dataset.
Chaos engineering for LLMs — controlled injection of XID errors, simulated ECC, artificial HBM latency to validate runbooks before the real incident.
Multi-cluster incident coordination — how to coordinate Keep across geographic clusters when an incident affects multiple regions.
CMDB and procurement integration — the RMA → ticket → ServiceNow → hardware replacement cycle automated via workflow.
LLM forensics — extracting the full OTel trace of a request affected by an incident, with redacted PII, kept in an evidence vault.

References

ISO/IEC 27035-1:2023 — Information security incident management — Principles and process.
ISO/IEC 27035-2:2023 — Information security incident management — Guidelines to plan and prepare for incident response.
ENS — Real Decreto 311/2022, Anexo II controles op.exp.7 a op.exp.10.
Directiva NIS2 (UE 2022/2555) — art. 23 (notificación de incidentes significativos).
Reglamento EU AI Act (UE 2024/1689) — art. 73 (reporting of serious incidents).
ISO/IEC 42001:2023 — AI management system — cláusula 10 (mejora continua).
Keep project — keephq.dev y github.com/keephq/keep (documentación de workflows YAML, providers).
Apache Kafka — Tiered Storage y cleanup.policy (docs.confluent.io / kafka.apache.org).
Confluent — Schema Registry y best practices para eventos lifecycle.
NVIDIA — Xid Errors Documentation y procedimientos de remediación.
Google SRE Book — Effective Troubleshooting y Postmortem Culture.
Atlassian — Incident Management Handbook (referencia para severity matrices).

TL;DR

You are here: OBSERVE → DEPLOY (incident response closes the loop)

The analogy: a nuclear reactor control room

The incident pipeline architecture

The six critical alerts and their runbooks

RB-01 · GpuHbmNearOom — HBM > 92 % sustained

RB-02 · GpuThermalOrPowerThrottle — bit ≠ 0 and not Idle in CLOCK_THROTTLE_REASONS

RB-03 · GpuXidErrorDetected — increase(DCGM_FI_DEV_XID_ERRORS[5m]) > 0

RB-04 · GpuEccDoubleBit — DCGM_FI_DEV_ECC_DBE_VOL_TOTAL > 0

RB-05 · VllmKvCachePoolNearFull — gpu_cache_usage_perc > 95 % sustained 3 min

RB-06 · VllmTtftP95OutOfSlo — TTFT P95 > 1.5 s for 5 min

Keep YAML workflows — three complete examples

Workflow 1 — xid-detected.yaml

Workflow 2 — ecc-dbe.yaml — immediate paging

Workflow 3 — canary-rollback.yaml — TTFT P95 out of SLO

The canonical Kafka event schema

Formal fit into incident management

ISO/IEC 27035 — information security incident management

ENS (Esquema Nacional de Seguridad) — op.exp controls

NIS2 — notification to the competent authority

EU AI Act — art. 73 (serious incident reporting)

ISO/IEC 42001 — AIMS clause 10 continual improvement

Four anti-patterns

Applied to typical on-premise hardware

What we have not covered (upcoming posts)

See also

References

RB-01 · `GpuHbmNearOom` — HBM > 92 % sustained

RB-02 · `GpuThermalOrPowerThrottle` — bit ≠ 0 and not Idle in CLOCK_THROTTLE_REASONS

RB-03 · `GpuXidErrorDetected` — `increase(DCGM_FI_DEV_XID_ERRORS[5m]) > 0`

RB-04 · `GpuEccDoubleBit` — `DCGM_FI_DEV_ECC_DBE_VOL_TOTAL > 0`

RB-05 · `VllmKvCachePoolNearFull` — `gpu_cache_usage_perc > 95 %` sustained 3 min

RB-06 · `VllmTtftP95OutOfSlo` — TTFT P95 > 1.5 s for 5 min

Workflow 1 — `xid-detected.yaml`

Workflow 2 — `ecc-dbe.yaml` — immediate paging

Workflow 3 — `canary-rollback.yaml` — TTFT P95 out of SLO