Kafka on lo0 — Tech Blog

Debezium and CDC: the notary who hears changes before anyone asks

Thu, 04 Jun 2026 11:00:00 +0200

TL;DR

Change Data Capture (CDC) with Debezium listens to the PostgreSQL Write-Ahead Log and turns every INSERT, UPDATE and DELETE into a structured Kafka event. Unlike traditional polling (SELECT ... WHERE updated_at > ?), it detects deletes, has latency in the tens of milliseconds and adds no extra load to the database. In RAG pipelines, this means that when a document is deleted from Postgres, the Qdrant chunks disappear too, automatically and in real time. The supporting infrastructure is modest: the connector uses 2-4 cores and 4-8 GB RAM to process thousands of events per second.

The master analogy: the land registry notary

Picture the land registry. Every time a flat is sold, mortgaged, or a mortgage is cancelled, the registrar records the operation in the registry ledger, a chronological and immutable journal. If you want to know what has changed in the registry, you have two options.

Option A (polling): you send someone every 5 minutes with a list of properties to ask “has anything changed?”. Problems: if a title was cancelled (DELETE), the property no longer exists by the time your envoy arrives, and there is no trace of it. If 20 different departments are doing the same thing, that is 20 people pestering the registrar every 5 minutes. And the minimum latency is the interval: 5 minutes.

Option B (Debezium): you hire a notary who sits right at the registrar’s desk. Every time the registrar signs an operation into the ledger, the notary notes it down immediately and notifies whoever needs to know. Title cancellations included, because the notary sees them as clearly as any other operation: he was there when it was signed.

In this analogy:

The registry ledger is the PostgreSQL WAL (Write-Ahead Log).
The notary is the Debezium connector.
The notary’s bookmark, which guarantees he loses no page even if he steps out for a moment, is the logical replication slot.
The messenger who carries the notifications to interested parties is Kafka (or Redpanda, or NATS JetStream).

We will pick this thread up in every section. When a technical detail is unclear, go back to the image of the notary.

1. The problem CDC solves

The most common synchronisation pattern between services sharing PostgreSQL is periodic polling:

SELECT id, content, updated_at
FROM documents
WHERE updated_at > $1
ORDER BY updated_at
LIMIT 1000;

This pattern has three structural problems.

DELETEs are invisible. When you delete a row, updated_at is not updated: the row is gone. The next time the poller queries, the row does not exist and there is no way to know it ever did. In a RAG pipeline, this means orphan chunks in Qdrant: the document no longer exists in Postgres, but its vectors keep polluting search results.

The minimum latency is the interval. If the poller runs every 5 seconds, average latency is 2.5 seconds. For near-real-time synchronisation (dashboards, alerts, RAG with frequently changing documents), that is too much.

Load scales with the number of consumers. If 10 services poll the same table every 5 seconds, that is 10 × 12 = 120 queries/minute producing no useful work; they only check whether there is anything new. On large tables with complex indexes, that is real load on the database.

CDC inverts the model: the database notifies, the consumers listen. Zero polling, zero extra load, DELETEs included, latency in the tens of milliseconds.

2. What the PostgreSQL WAL is

The operations journal

The Write-Ahead Log (WAL) is the chronological, immutable record of every operation Postgres performs. Before modifying any data page on disk, Postgres writes the operation to the WAL. That sequence, log first then data, is what guarantees durability (the D in ACID) and enables crash recovery: if Postgres dies mid-transaction, on restart it replays the WAL to return the database to a consistent state.

The WAL is the registry ledger of our analogy: chronological, immutable, complete.

Physical vs logical replication

PostgreSQL supports two WAL-based replication modes:

Physical replication: it replicates disk blocks verbatim. The standby receives the same bytes as the primary. It serves high availability and failover, but the target must be an exact copy of Postgres; you cannot send the changes to an external application.
Logical replication: instead of disk blocks, it replicates semantic operations: “the row with id=42 was inserted into table documents with these values”. The target can be anything that understands the protocol: another Postgres, Debezium, or any custom consumer.

CDC uses logical replication. It is what lets Debezium understand “what changed and in which table” rather than “which disk block changed at which offset”.

The replication slot: the notary’s bookmark

A logical replication slot is a persistent cursor in the WAL. Postgres keeps a record of how far into the WAL each slot has consumed. While a slot exists, Postgres guarantees it will not discard the WAL segments the slot has not yet read.

This is exactly the notary’s bookmark: even if the notary goes out for lunch, the ledger stays open at the last page he read. When he comes back, he carries on from exactly where he left off, having lost nothing.

The risk is the inverse: if the notary never comes back, the bookmark stops the registrar from archiving the old pages. If the Debezium connector goes down and does not recover for hours, the WAL grows indefinitely on disk until the slot is deleted manually or the consumer starts consuming again. This is called WAL disk blowup and is the most important operational risk of Debezium.

Mandatory monitoring:

SELECT slot_name,
 confirmed_flush_lsn,
 pg_current_wal_lsn(),
 pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn) AS lag_bytes
FROM pg_replication_slots
WHERE slot_type = 'logical';

The pgoutput plugin

The WAL stores operations in an internal binary format. For Debezium to understand them, Postgres needs to decode them into a readable format. The pgoutput decoding plugin, included in the Postgres core since version 10, does exactly that: it translates the binary WAL events into messages with the before/after structure of each row.

Debezium uses pgoutput by default. It requires no external extensions (unlike the wal2json plugin that was popular before Postgres 10).

3. Debezium architecture

The connector as a Kafka Connect plugin

Debezium is not a standalone service; it is a plugin for the Kafka Connect framework. Kafka Connect manages the connector’s lifecycle (start, stop, reconnection, offset tracking) and provides the parallelism and fault tolerance infrastructure.

The connector talks to Postgres through the logical replication protocol (not over JDBC), using the credentials of a user with the REPLICATION role.

PostgreSQL (WAL + pgoutput)
│
│ logical replication protocol
▼
Debezium Connector (Kafka Connect worker)
│
│ Kafka Producer API
▼
Kafka topic: rag.public.documents
│
▼
Consumer (Qdrant sync, audit log, fine-tuning pipeline...)

Structure of a Debezium event

Each change in the table becomes a JSON message with this structure:

INSERT ("op": "c" — create):

{
 "before": null,
 "after": {
 "id": 42,
 "content": "Contrato de arrendamiento...",
 "tenant_id": "acme",
 "updated_at": 1748934000000
 },
 "op": "c",
 "source": {
 "version": "2.7.0.Final",
 "connector": "postgresql",
 "db": "rag_db",
 "schema": "public",
 "table": "documents",
 "lsn": 29823948,
 "txId": 1047,
 "ts_ms": 1748934000123
 }
}

DELETE ("op": "d"):

{
 "before": {
 "id": 42,
 "content": "Contrato de arrendamiento...",
 "tenant_id": "acme",
 "updated_at": 1748934000000
 },
 "after": null,
 "op": "d",
 "source": { "lsn": 29824102, "txId": 1051, "ts_ms": 1748934060200 }
}

The before field holds the row’s previous state, available because Postgres can be configured with REPLICA IDENTITY FULL to include the complete row in the WAL on delete or update. Without that setting, before contains only the primary key.

This is the key for the RAG pipeline: the DELETE event carries the document id. The consumer uses it to delete every associated chunk in Qdrant with a doc_id = 42 filter. Without CDC, those chunks would never have been deleted.

Initial snapshot

When the connector starts for the first time (or after a reset), it cannot begin consuming the WAL from “the beginning of time”, only from the moment the slot was created. How does it guarantee the consistency of the initial state?

Through a transactional snapshot: the connector opens a transaction in REPEATABLE READ mode, exports the snapshot ID (pg_export_snapshot()), and does a full SELECT of the configured tables inside that transaction. It then starts consuming the WAL from the snapshot’s LSN. That way there is no gap: the snapshot covers the state up to an instant, and the WAL covers from that instant onwards.

SMT transformations (Single Message Transforms)

Before emitting the event to the Kafka topic, the connector can apply inline transformations called SMT. Common use cases:

Filtering sensitive columns (ReplaceField with blacklist): strip password_hash, phone_number before they reach the topic.
Adding metadata (InsertField): enrich the event with a tenant_id extracted from the original HTTP header (if it is in the row).
Conditional routing (Filter): drop events from rows with status = 'draft' before emitting them.

SMTs are pure configuration, requiring no code, and they are applied inside the connector process with no perceptible extra latency.

4. Debezium vs the Outbox pattern

The Outbox pattern is the most common alternative to pure CDC. Instead of emitting events directly to Kafka, the application writes to an outbox table in Postgres within the same transaction that modifies the data. A separate worker reads that table and publishes the events.

Criterion	Debezium (pure CDC)	Outbox pattern
Event latency	~50-200 ms from commit	Depends on the worker interval (typically 1-5 s)
Consistency	At-least-once	At-least-once
DELETE detection	Native (the DELETE event includes `before`)	Only if the app writes to the outbox on delete
Setup complexity	High (Kafka Connect, replication slot, permissions)	Low (an extra table plus a simple worker)
Infrastructure dependency	Requires Kafka/Redpanda/NATS JetStream	Postgres plus a worker; Kafka optional
WAL disk blowup risk	Yes, if the slot stops consuming	No
Schema visibility	Reads the table’s real schema	The event schema is defined by the app
Schema migration	Needs care (events reflect DDL changes)	More flexible (the event is whatever the app puts there)
When to use it	When you need DELETEs, low latency, or you cannot modify the app	When the app controls the event domain and infrastructure is limited

Rule of thumb: if you control the application code and do not need native DELETEs, the Outbox is simpler. If you do not control the code (legacy database, third-party application) or DELETEs are critical (a RAG pipeline with document deletion), Debezium is the right choice.

5. The maths

Throughput

Debezium on a connector with 4 workers can process between 10,000 and 50,000 events/second on modest hardware (4 cores, 8 GB RAM). The real bottleneck is not the connector but the Kafka broker: with 3 brokers and suitable partitions, Kafka can comfortably sustain 500,000 messages/second with 1 KB messages (source: public Confluent benchmarks, 2023).

For a typical RAG pipeline with 100 documents modified per minute:

$$\text{events/s} = \frac{100}{60} \approx 1{.}7 \text{ events/s}$$

That is 0.0034% of the connector’s capacity. Debezium will not be the bottleneck in any realistic RAG scenario.

End-to-end latency

The path from a commit in Postgres to an upsert in Qdrant has these stages:

Stage	Typical latency
Commit in Postgres → WAL written	< 1 ms (synchronous with the commit)
WAL written → Debezium reads it (WAL lag)	10-50 ms
Debezium → Kafka produce (ack)	5-20 ms
Kafka → Consumer (poll interval)	0-100 ms (configurable)
Consumer → Qdrant upsert/delete	5-15 ms
Typical total	30-200 ms

With fetch.min.bytes=1 and fetch.max.wait.ms=10 on the consumer, the Kafka poll latency drops to ~10 ms. The realistic range for an optimised pipeline is 30-100 ms.

WAL disk blowup risk

If the connector stops consuming, Postgres retains the WAL from the slot’s confirmed_flush_lsn onwards. The retained volume grows linearly with time and the write rate:

$$\text{retained WAL} = \text{write rate} \times \text{average WAL event size} \times \text{time without consuming}$$

Example with a moderate load (50,000 writes/hour, 500 bytes average per WAL event):

$$50{,}000 \times 500 \text{ B} \times 1 \text{ h} = 25 \text{ MB/h}$$

With a high load (1,000,000 writes/hour):

$$1{,}000{,}000 \times 500 \text{ B} \times 1 \text{ h} = 500 \text{ MB/h}$$

If the connector is down for 48 hours under high load: 24 GB of retained WAL. That can fill the disk and block Postgres completely.

Recommended alert: set an alert when lag_bytes > 1 GB or when confirmed_flush_lsn has not advanced for more than 15 minutes. See the monitoring query in section 2.

6. Use cases in LLMOps / RAG

RAG synchronisation with real deletion

This is the use case that most clearly justifies Debezium over polling. The flow:

A user deletes document id=42 from the document management interface.
Postgres runs DELETE FROM documents WHERE id = 42.
Debezium detects the DELETE in the WAL and emits the event with "op": "d" and "before": {"id": 42, ...}.

The consumer receives the event and runs:

qdrant_client.delete(
 collection_name="documents",
 points_selector=Filter(must=[FieldCondition(key="doc_id", match=MatchValue(value=42))])
)

Every chunk with doc_id=42 disappears from Qdrant in ~100 ms.

Without Debezium, those chunks would stay indefinitely, polluting retrieval results with fragments of documents that no longer exist in the source of truth.

Event sourcing for fine-tuning datasets

Every time a human annotator updates a row in the annotations table (correcting an LLM output), Debezium emits the UPDATE with before and after. The consumer writes the pair (original_output, correction) into the dataset curation pipeline, with no need for the annotator to do anything beyond saving in the interface. The fine-tuning pipeline knows exactly what changed and when, with no polling and no risk of duplicates from overlapping time windows.

Immutable audit log

WAL events are, by definition, the most faithful record of what happened in the database: they are the same data Postgres uses for crash recovery. Kafka with long retention (90 days, or size-based retention) serves as an immutable audit log without modifying the application schema or adding triggers. This is particularly useful in regulated environments where traceability of data modifications is required.

7. Architecture diagram

8. Minimum configuration

PostgreSQL: enable logical replication

-- Requires restarting Postgres
ALTER SYSTEM SET wal_level = logical;
ALTER SYSTEM SET max_replication_slots = 10;
ALTER SYSTEM SET max_wal_senders = 10;

-- Reload configuration (wal_level requires a full restart)
SELECT pg_reload_conf();

-- Dedicated user for Debezium
CREATE USER debezium WITH REPLICATION LOGIN PASSWORD 'cambiar_esto';
GRANT SELECT ON TABLE public.documents TO debezium;

-- REPLICA IDENTITY FULL to get a complete 'before' on DELETEs and UPDATEs
ALTER TABLE public.documents REPLICA IDENTITY FULL;

Debezium connector (Kafka Connect REST API)

{
 "name": "postgres-debezium",
 "config": {
 "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
 "database.hostname": "postgres",
 "database.port": "5432",
 "database.user": "debezium",
 "database.password": "cambiar_esto",
 "database.dbname": "rag_db",
 "topic.prefix": "rag",
 "table.include.list": "public.documents",
 "plugin.name": "pgoutput",
 "slot.name": "debezium_rag",
 "publication.autocreate.mode": "filtered",
 "snapshot.mode": "initial",
 "tombstones.on.delete": "true",
 "transforms": "unwrap",
 "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
 "transforms.unwrap.drop.tombstones": "false",
 "transforms.unwrap.delete.handling.mode": "rewrite"
 }
}

curl -X POST http://kafka-connect:8083/connectors \
 -H 'Content-Type: application/json' \
 -d @connector-config.json

Check its status:

curl http://kafka-connect:8083/connectors/postgres-debezium/status

Minimal consumer in Python

from confluent_kafka import Consumer
from qdrant_client import QdrantClient
from qdrant_client.models import Filter, FieldCondition, MatchValue
import json

consumer = Consumer({
 "bootstrap.servers": "kafka:9092",
 "group.id": "qdrant-sync",
 "auto.offset.reset": "earliest",
 "enable.auto.commit": False,
})
consumer.subscribe(["rag.public.documents"])
qdrant = QdrantClient("qdrant", port=6333)

while True:
 msg = consumer.poll(timeout=0.1)
 if msg is None:
 continue
 event = json.loads(msg.value())
 op = event.get("op")
 if op in ("c", "u"): # INSERT or UPDATE
 doc = event["after"]
 # ... vectorise and upsert into Qdrant
 elif op == "d": # DELETE
 doc_id = event["before"]["id"]
 qdrant.delete(
 collection_name="documents",
 points_selector=Filter(
 must=[FieldCondition(key="doc_id", match=MatchValue(value=doc_id))]
 )
 )
 consumer.commit()

9. On-premise deployment

The Debezium stack does not compete for GPU. On a node with 4×H100 SXM (320 GB, NVLink) serving the inference LLM, the CDC pipeline runs entirely on general-purpose (CPU-only) nodes:

Component	Recommended resources	Role
Kafka Connect + Debezium	2-4 cores, 4-8 GB RAM	Read the WAL, emit events
Kafka brokers (×3)	4 cores, 32 GB RAM each	High availability, retention
Qdrant-sync consumer	2 cores, 4 GB RAM	Vectorise plus upsert/delete
Qdrant	8 cores, 64 GB RAM	Vector store

The Debezium connector is notably light: in production at 10,000 events/second, the connector typically uses less than 1 core and 2 GB of RAM. JVM memory (Kafka Connect runs on the JVM) must be capped explicitly with -Xmx4g to stop the GC from causing pauses.

For high availability, Kafka Connect supports distributed mode with multiple workers. If a worker dies, the connector is reassigned automatically to another worker within seconds, and the replication slot guarantees no events are lost during the switchover.

What we have not covered

Debezium with MySQL, MongoDB and Oracle: each connector uses the native log mechanism (binlog on MySQL, oplog on MongoDB, LogMiner on Oracle). The resulting event API is similar, but the configuration details and limitations differ.
Debezium Server: standalone mode without Kafka Connect, with direct sinks to HTTP, S3, Redis Streams or NATS. Useful when the Kafka infrastructure is too complex for the use case.
Schema Registry: how Avro with Confluent Schema Registry or Apicurio manages event schema evolution (adding columns, changing types) without breaking existing consumers.
Exactly-once semantics: why at-least-once is enough for most RAG cases (an idempotent upsert into Qdrant with the same vector does no harm) and when exactly-once is needed (financial counters, inventory deductions).
Outbox pattern plus Debezium combined: Debezium reading the outbox table instead of the business table’s WAL directly, the Transactional Outbox + CDC pattern that combines the best of both worlds.

References

Debezium Documentation — PostgreSQL Connector. debezium.io/documentation/reference/stable/connectors/postgresql.html
PostgreSQL Documentation — Logical Replication. postgresql.org/docs/current/logical-replication.html
PostgreSQL Documentation — Write-Ahead Logging. postgresql.org/docs/current/wal-intro.html
PostgreSQL Documentation — Replication Slots. postgresql.org/docs/current/logicaldecoding-explanation.html
Confluent — Kafka Performance Benchmarks (2023). confluent.io/blog/kafka-fastest-messaging-system
Gunnar Morling — Outbox Pattern. morling.dev/blog/sending-messages-as-part-of-database-transactions
Debezium — SMT documentation. debezium.io/documentation/reference/stable/transformations
Qdrant Documentation — Filtering. qdrant.tech/documentation/concepts/filtering

Incident response runbooks for LLM inference: every alert to a concrete action with Kafka and Keep

Tue, 02 Jun 2026 04:30:00 +0200

This post closes the observability trilogy opened by GPU observability for LLM inference (which metrics) and Anatomy of the twelve DCGM and five vLLM metrics (which documented anomaly per metric). Here each anomaly gets its concrete action and is fitted into the incident management machinery that compliance demands.

TL;DR

The GPU observability alerts are useless without a codified procedure for each one; the operator who interprets them by hand every time is operating on intuition. The right combination has three indispensable pieces. (1) Runbook catalogue: for each of the six critical alerts (GpuHbmNearOom, GpuThermalOrPowerThrottle, GpuXidErrorDetected, GpuEccDoubleBit, VllmKvCachePoolNearFull, VllmTtftP95OutOfSlo), a severity, an immediate mitigation, the evidence to capture before remediating, a resolution action, a closure criterion and a postmortem trigger. (2) Reproducible pipeline: Prometheus + DCGM → Alertmanager → Kafka as event bus (topics gpu.alerts.enriched, incidents.lifecycle, audit.actions with WORM retention) → Keep as workflow engine (declarative YAML workflows versioned in git) → Kubernetes job / script / ChatOps executors. (3) Formal fit into incident management under the regulatory corpus: ISO/IEC 27035 phases identify → report → assess → respond → learn; ENS controls op.exp.7 (incident management), op.exp.8 (activity logging), op.exp.10 (notification to users); NIS2 art. 23 with an early warning at 24 h, a formal notification at 72 h and a final report at 1 month; EU AI Act art. 73 for a serious incident on a high-risk system, deadlines of 2 to 15 days depending on severity; ISO/IEC 42001 clause 10 (continual improvement of the AIMS). The action taxonomy is immediate mitigation (drain, throttle, scale-down: contains the damage in seconds) → diagnosis (evidence capture with nvidia-smi -q, dmesg, a vLLM /metrics snapshot, the related OTel trace; without this the postmortem is not defensible) → resolution (restart, reset, RMA, rollback) → postmortem (5-whys RCA, prevention plan, runbook update). Kafka contributes the immutable audit trail that ENS and the EU AI Act demand: every action executed by Keep or by a human is published as an event on audit.actions with timestamp, actor, decision and evidence, retained WORM for at least 6 months. Keep contributes workflows as code: this post includes three complete workflows (XID with drain plus Jira ticket, ECC DBE with immediate paging and node blocking in the scheduler, automatic canary rollback on TTFT P95 out of SLO). Four anti-patterns close the material: alerts without a runbook (most of them), a runbook without prior evidence capture (it perpetuates the incident because the root cause is lost), escalation by seniority rota instead of severity (a junior operator handling an ECC DBE), and the absence of a human gate for destructive actions (Keep running nvidia-smi --gpu-reset without confirmation). Applicable to a generic 4×H100 SXM cluster with Kafka and Keep already deployed.

You are here: OBSERVE → DEPLOY (incident response closes the loop)

The analogy: a nuclear reactor control room

In a nuclear plant control room, the operator on shift never decides what to do on seeing an alarm. The decision is pre-made and codified in a written procedure (SOP) covering every alarm on the panel: if X sounds, open book X, read steps 1-N, execute exactly, call the supervisor at step M, escalate to the plant director at step N+3. The reason is strict: critical alarms are rare but catastrophic if handled badly; an operator improvising in an emergency makes worse decisions than one applying a procedure reviewed by experts and validated by simulation.

The reactor does not expect the operator to be a genius. It expects them to know the procedures to the letter and the operations management system to hand them the right procedure at the right moment. If the procedures are not written, not versioned, or not integrated with the alarms that fire them, the control room is operating on intuition. The difference between the two operations, procedural versus intuitive, is the difference between a plant that runs 30 years without incidents and one that ends up on a blacklist.

Incident response on an LLM inference cluster works identically. The DCGM and vLLM alerts listed in the previous posts are the panel alarms. Each one needs its written SOP, versioned, integrated with the alert that fires it and reviewed after every incident. Without that codification, the operator on shift improvises in the middle of an ECC DBE failure at 4 in the morning; with it, they execute the nine steps of runbook 12 and the incident closes in 20 minutes.

The incident pipeline architecture

Prometheus + DCGM. Collects the metrics described in the two previous posts. PrometheusRules define the six critical alerts with for: <duration> to avoid noise.

Alertmanager. Receives raw alerts; deduplicates, groups by labels ({cluster, node, gpu, model}), routes. Instead of sending straight to PagerDuty or Slack, it sends to Kafka via a webhook receiver. That turns the alert into a bus event that multiple consumers process (Keep for action, the audit topic for compliance, dashboards for visualisation).

Kafka as event bus. Three canonical topics:

gpu.alerts.enriched — alerts with added context (tenant, model, version, namespace owner, effective severity). Retention: 7 days, replication factor 3.
incidents.lifecycle — incident cycle events: incident.opened, incident.acknowledged, action.proposed, action.executed, incident.escalated, incident.resolved, postmortem.attached. Retention: 90 days.
audit.actions — an immutable record of every action executed (by Keep automatically or by a human confirming). Retention: 6 months minimum with compaction off plus tiered storage, WORM storage. This is the topic ENS op.exp.8, EU AI Act art. 12 and NIS2 require you to keep.

Keep as workflow engine. Consumes from gpu.alerts.enriched, fires YAML workflows versioned in git, executes actions (HTTP calls, kubectl jobs, Slack messages, Jira tickets) and publishes the result to incidents.lifecycle plus audit.actions. Choosing Keep over Alertmanager alone (or over PagerDuty alone) is deliberate: Keep separates runbook declaration (readable, reviewable YAML) from notification delivery (PagerDuty). The runbook is versioned code; notifications are operational details.

Executors. What actually moves the cluster:

Kubernetes jobs: kubectl drain, kubectl cordon, kubectl rollout undo.
NVIDIA API: nvidia-smi --gpu-reset, dcgmi diag -r <level>.
ChatOps: human confirmations through Slack interactive messages before a destructive action runs.
External tooling: Jira ticket, PagerDuty notification, CMDB call.

The six critical alerts and their runbooks

For each alert: severity, immediate mitigation (seconds), evidence to capture before remediating, resolution action, closure criteria, postmortem trigger.

RB-01 · `GpuHbmNearOom` — HBM > 92 % sustained

Severity: WARNING. Risk of OOM on the next PagedAttention allocation.

Immediate mitigation. Reduce admission temporarily by lowering max_num_seqs on the affected engine via hot reload (if the engine supports it) or a staggered restart of replicas. Trigger an additional scale-out via KEDA if there are free GPU nodes. There is no need to drain the node.

Evidence to capture.

nvidia-smi --query-gpu=index,memory.used,memory.free,memory.total --format=csv
nvidia-smi -q -d ROW_REMAPPER | grep -i pending
curl http://vllm-pod:8000/metrics | grep -E "gpu_cache_usage|num_requests"
kubectl logs <pod> --tail=200 | grep -i "preempt\|swap"

Save the snapshot to audit.actions with a timestamp and an incident_id.

Resolution. If the cause is a traffic spike: let the autoscaler scale to a stable regime, monitor for 30 min. If the cause is a model regression (canary v2 consumes more KV cache than v1): roll the canary back (see RB-06). If it is a leak (the metric grows without traffic growing): restart the pod with a heap dump capture.

Closure. gpu_cache_usage_perc < 80 % sustained for 15 min AND num_requests_waiting == 0.

Postmortem. Not mandatory unless the incident lasted > 30 min or had SLO impact.

RB-02 · `GpuThermalOrPowerThrottle` — bit ≠ 0 and not Idle in CLOCK_THROTTLE_REASONS

Severity: WARNING (thermal) or CRITICAL (sustained HW Power Brake, PDU risk).

Immediate mitigation. Identify the bit (decode the bitmap). If it is 0x40 HW_THERMAL or 0x20 SW_THERMAL: drain the workload from the node to other replicas if the temperature does not drop within 2 min, and stop new pods landing on that node (kubectl cordon). If it is 0x80 HW_POWER_BRAKE: alert DC infrastructure immediately (likely an over-committed PDU, the Dell KB 000220508 / Lenovo HT514380 case) and lower the TDP of the rack’s GPUs via nvidia-smi -pl to a lower value to relieve load on the breaker.

Evidence.

nvidia-smi --query-gpu=index,temperature.gpu,temperature.memory,power.draw,clocks_throttle_reasons.active --format=csv
ipmitool sdr | grep -i "fan\|temp\|inlet"
# PDU data if they are instrumented (modbus / SNMP)

Resolution.

Thermal: review rack airflow, check the rear-door HX, T_inlet, DGX fans. An infra issue, not an engine one.
Power Brake: review the sizing of the PDU branch, the breaker, the 415 VAC distribution. Likely a redistribution of load to another branch or a temporary TDP cap.

Closure. CLOCK_THROTTLE_REASONS == 0x1 (Idle only) or 0x0 for 30 min under normal load.

Postmortem. Mandatory if it was a HW Power Brake: that implicates the DC electrical infrastructure.

RB-03 · `GpuXidErrorDetected` — `increase(DCGM_FI_DEV_XID_ERRORS[5m]) > 0`

Severity: CRITICAL.

Immediate mitigation. kubectl cordon the node (no more new pods). If the XID is 31/48/79/94/95 (hardware or cascade): drain the node’s existing pods. If the XID is 13/43 (possibly software): keep the pods but block new ones, capture the trace and the active workload.

Evidence.

# The specific XID from dmesg
dmesg | grep -i xid | tail -30
nvidia-smi -q -d ERROR
nvidia-smi -q -d PCIE
# State of the retired pages
nvidia-smi -q -d ROW_REMAPPER
# Workload that was running
kubectl get pods -o wide | grep <node>
kubectl logs <pod> --previous --tail=500

Resolution.

XID 13/43 (software exception / channel verif): if it recurs only with one specific model, it is a workload bug; raise an issue with the model team. If it is transient, restarting the pod is enough.
XID 31 (MMU fault): usually a cascade from a previous XID 48. Reset the GPU (nvidia-smi --gpu-reset -i <index>) or reboot the node if the reset does not resolve it.
XID 48 / 95 (DBE / uncontained ECC): see RB-04. The node goes into quarantine.
XID 79 (fallen off the bus): reboot the node. If it recurs after the reboot, open an RMA for the GPU. ByteDance reports 43 % co-occurrence with PCIe errors, so check the slot and the cable too.
XID 94 / 145 / 149: catalogued in NVIDIA’s Xid Catalog with a specific procedure.

Closure. Node smoke test passed (dcgmi diag -r 3), 24 h with no new XIDs, back into the pool.

Postmortem. Mandatory. Include the specific XID, the distribution of XIDs across the cluster, an updated MTBE.

RB-04 · `GpuEccDoubleBit` — `DCGM_FI_DEV_ECC_DBE_VOL_TOTAL > 0`

Severity: CRITICAL, data corruption in progress.

Immediate mitigation. Drain the node immediately without waiting for further evidence. Page (PagerDuty / OpsGenie) the primary ON-CALL. Mark the node unschedulable and failed. XID 48 has a 100 % probability of killing the running job according to the Story of Two GPUs dataset; any inference already in flight is compromised.

Evidence (in parallel with the mitigation).

nvidia-smi -q -d ECC
nvidia-smi -q -d ROW_REMAPPER # Pending: Yes expected
dmesg | grep -E "Xid.*48|DBE|double-bit" | tail -50
# Full capture of the GPU state
dcgmi diag -r 4 -i <gpu_index>

Resolution. Full GPU reset (nvidia-smi --gpu-reset) or a node reboot if the reset does not complete. The reset activates the row remap. After the reboot:

nvidia-smi -q -d ROW_REMAPPER # Pending: No expected
nvidia-smi -q -d ECC # volatile counters at 0

If RETIRED_DBE > 8 pages after the remap: schedule a GPU replacement in the next window, because silicon degradation is progressive. The published real case documents ~19 hours of downtime as typical.

Closure. Node back in the pool after 48 h with no new DBEs.

Postmortem. Mandatory. If the incident affected a request carrying personal or classified data, evaluate notification to the DPO under GDPR art. 33 (it is not necessarily a breach, but it has to be assessed).

RB-05 · `VllmKvCachePoolNearFull` — `gpu_cache_usage_perc > 95 %` sustained 3 min

Severity: WARNING (risk of preempt-on-OOM, not of a real OOM).

Immediate mitigation. Activate autoscaler scale-out by lowering the KEDA threshold temporarily (from 0.85 to 0.75) for 30 min. In recompute mode, preempts raise TTFT but do not break requests; acceptable in the short term. In swap mode latency goes through the roof, so it is better to cut new traffic (return 503 from the router) for 5 min.

Evidence.

curl http://vllm-pod:8000/metrics | grep -E "gpu_cache|num_requests|num_preemptions"
kubectl get hpa vllm-llama70b
kubectl logs <pod> --tail=200 | grep -i preempt

Resolution. If it recurs regularly: revisit capacity planning, possibly lowering max_num_seqs or raising the stable replica count. See Capacity planning.

Closure. Pool < 85 % sustained 30 min, no preempts in the last 15 min.

Postmortem. Not mandatory unless it recurs > 3 times / week.

RB-06 · `VllmTtftP95OutOfSlo` — TTFT P95 > 1.5 s for 5 min

Severity: CRITICAL (contractual SLO violation).

Immediate mitigation. Quick diagnosis of the regime (in order of likelihood):

If a v2 canary is active and the ratio ttft_p95(v2)/ttft_p95(v1) > 1.30: automatic rollback of the canary via Argo Rollouts (argo rollouts abort vllm-llama70b).
If num_requests_waiting > 5: scale out via KEDA.
If DRAM_ACTIVE > 90 % plus gpu_cache_usage_perc > 90 %: HBM bottleneck, reach for the quantisation lever or a context reduction.
If CLOCK_THROTTLE_REASONS != 0: see RB-02.

Evidence.

# Histogram snapshot
curl http://vllm-pod:8000/metrics | grep time_to_first_token
# Breakdown by version if a canary is running
# DCGM state at that moment
curl http://dcgm-exporter:9400/metrics | grep -E "PIPE_TENSOR|DRAM_ACTIVE|THROTTLE"
# Active traffic
kubectl top pods -n inference

Resolution. Depends on the diagnosis. Typical cases:

Canary regression → full rollback (see Canary).
Capacity saturation → scale replicas or accept a temporary 503 with Retry-After.
Prefill bound → enable or calibrate chunked prefill or disaggregated serving (see Disaggregated serving).

Closure. TTFT P95 inside SLO sustained 30 min.

Postmortem. Mandatory. Document the root cause and the lever applied; update the runbook.

Keep YAML workflows — three complete examples

Runbooks are only useful if they are codified in the workflow engine. Keep lets you declare them in YAML versioned in git.

Workflow 1 — `xid-detected.yaml`

workflow:
 id: xid-detected-drain
 name: "XID error detected — cordon node and capture evidence"
 description: "RB-03 implementation"
 triggers:
 - type: alert
 filters:
 - key: alertname
 value: GpuXidErrorDetected
 steps:
 - name: capture-evidence
 provider:
 type: bash
 with:
 command: |
 set -e
 NODE="{{ alert.labels.node }}"
 GPU="{{ alert.labels.gpu }}"
 INC_ID="{{ alert.fingerprint }}"
 mkdir -p /var/evidence/$INC_ID
 kubectl debug node/$NODE -it --image=nvcr.io/nvidia/cuda:12.4.0-base-ubuntu22.04 -- \
 bash -c "nvidia-smi -q -d ERROR,PCIE,ROW_REMAPPER > /host/var/evidence/$INC_ID/smi.txt"
 kubectl describe node $NODE > /var/evidence/$INC_ID/node.txt
 - name: cordon-node
 provider:
 type: kubernetes
 with:
 action: cordon
 name: "{{ alert.labels.node }}"
 if: "{{ alert.labels.severity == 'critical' }}"
 actions:
 - name: open-jira-ticket
 provider:
 type: jira
 config: "{{ providers.jira-prod }}"
 with:
 project: GPUOPS
 issuetype: Incident
 summary: "RB-03: XID {{ alert.annotations.xid_code }} on {{ alert.labels.node }}/{{ alert.labels.gpu }}"
 description: |
 Severity: {{ alert.labels.severity }}
 XID: {{ alert.annotations.xid_code }}
 Evidence: /var/evidence/{{ alert.fingerprint }}
 Runbook: https://runbooks.example.local/RB-03
 - name: notify-slack
 provider:
 type: slack
 config: "{{ providers.slack-gpu-incidents }}"
 with:
 message: |
 :warning: *RB-03 triggered*
 Node: `{{ alert.labels.node }}` GPU: `{{ alert.labels.gpu }}`
 XID: `{{ alert.annotations.xid_code }}`
 <{{ jira.url }}|Jira ticket>
 - name: emit-audit
 provider:
 type: kafka
 config: "{{ providers.kafka-audit }}"
 with:
 topic: audit.actions
 message:
 incident_id: "{{ alert.fingerprint }}"
 action: "cordon_node"
 actor: "keep-workflow"
 workflow_id: "xid-detected-drain"
 target: "{{ alert.labels.node }}"
 timestamp: "{{ now }}"

Workflow 2 — `ecc-dbe.yaml` — immediate paging

workflow:
 id: ecc-dbe-critical
 name: "ECC double-bit — page on-call and quarantine node"
 triggers:
 - type: alert
 filters:
 - key: alertname
 value: GpuEccDoubleBit
 steps:
 - name: cordon-immediately
 provider:
 type: kubernetes
 with:
 action: cordon
 name: "{{ alert.labels.node }}"
 - name: drain-workload
 provider:
 type: kubernetes
 with:
 action: drain
 name: "{{ alert.labels.node }}"
 options:
 ignore-daemonsets: true
 delete-emptydir-data: true
 grace-period: 120
 - name: page-oncall
 provider:
 type: pagerduty
 config: "{{ providers.pagerduty-critical }}"
 with:
 service_key: "{{ env.PD_SERVICE_KEY }}"
 severity: critical
 summary: "RB-04 ECC DBE on {{ alert.labels.node }}/{{ alert.labels.gpu }} — node drained"
 - name: emit-lifecycle
 provider:
 type: kafka
 config: "{{ providers.kafka-incidents }}"
 with:
 topic: incidents.lifecycle
 message:
 incident_id: "{{ alert.fingerprint }}"
 event: incident.opened
 severity: critical
 runbook: RB-04
 requires_postmortem: true
 - name: notify-dpo
 provider:
 type: email
 with:
 to: dpo@example.local
 subject: "ECC DBE on a production GPU — assessment required"
 body: |
 RB-04 ECC DBE incident detected on {{ alert.labels.node }}.
 Affected model: {{ alert.labels.model }}.
 Please assess whether personal/classified data was processed
 during the error window and whether GDPR art. 33 notification is needed.

Workflow 3 — `canary-rollback.yaml` — TTFT P95 out of SLO

workflow:
 id: canary-rollback-ttft
 name: "Rollback canary when TTFT P95 ratio v2/v1 > 1.30"
 triggers:
 - type: alert
 filters:
 - key: alertname
 value: VllmTtftP95OutOfSlo
 - key: canary_active
 value: "true"
 steps:
 - name: check-ratio
 provider:
 type: prometheus
 config: "{{ providers.prom-prod }}"
 with:
 query: |
 histogram_quantile(0.95, sum by(le)(rate(vllm:time_to_first_token_seconds_bucket{version="v2"}[5m])))
 /
 histogram_quantile(0.95, sum by(le)(rate(vllm:time_to_first_token_seconds_bucket{version="v1"}[5m])))
 condition: result > 1.30
 actions:
 - name: argo-rollback
 provider:
 type: kubernetes
 with:
 action: exec
 command:
 - kubectl
 - argo
 - rollouts
 - abort
 - "{{ alert.labels.rollout }}"
 - -n
 - "{{ alert.labels.namespace }}"
 - name: notify-and-audit
 provider:
 type: kafka
 config: "{{ providers.kafka-audit }}"
 with:
 topic: audit.actions
 message:
 incident_id: "{{ alert.fingerprint }}"
 action: canary_rollback
 ratio: "{{ steps.check-ratio.result }}"
 actor: keep-workflow
 timestamp: "{{ now }}"

Each workflow lives in repos/keep-workflows/ versioned in git, reviewed by pull request, validated by CI (keep workflow validate). The written runbook lives as docs/runbooks/RB-XX.md linked from the workflow YAML; the two always evolve together.

The canonical Kafka event schema

For the topics to be consumable by compliance, postmortem tooling and dashboards without each consumer having to guess the shape, the schema is fixed with Avro / Protobuf.

{
 "name": "IncidentLifecycleEvent",
 "type": "record",
 "fields": [
 { "name": "incident_id", "type": "string" },
 { "name": "event", "type": { "type": "enum", "symbols": [
 "incident.opened", "incident.acknowledged", "action.proposed",
 "action.executed", "action.failed", "incident.escalated",
 "incident.resolved", "postmortem.attached"
 ]}},
 { "name": "timestamp", "type": "string", "logicalType": "timestamp-millis" },
 { "name": "actor", "type": "string" },
 { "name": "severity", "type": { "type": "enum", "symbols": ["low","warning","critical"] } },
 { "name": "runbook", "type": ["null","string"], "default": null },
 { "name": "alert_name", "type": "string" },
 { "name": "labels", "type": { "type": "map", "values": "string" } },
 { "name": "annotations", "type": { "type": "map", "values": "string" } },
 { "name": "evidence_uri", "type": ["null","string"], "default": null },
 { "name": "requires_postmortem", "type": "boolean", "default": false }
 ]
}

For audit.actions (WORM), a separate and stricter schema with non-modifiable fields:

{
 "name": "AuditAction",
 "type": "record",
 "fields": [
 { "name": "incident_id", "type": "string" },
 { "name": "action", "type": "string" },
 { "name": "actor", "type": "string" },
 { "name": "actor_type", "type": { "type": "enum", "symbols": ["human","workflow","scheduler"] } },
 { "name": "workflow_id", "type": ["null","string"], "default": null },
 { "name": "target", "type": "string" },
 { "name": "command", "type": ["null","string"], "default": null },
 { "name": "result", "type": { "type": "enum", "symbols": ["success","failure","partial"] } },
 { "name": "timestamp", "type": "string", "logicalType": "timestamp-millis" },
 { "name": "evidence_uri", "type": ["null","string"], "default": null },
 { "name": "approver", "type": ["null","string"], "default": null }
 ]
}

The topic is configured with cleanup.policy=delete, retention.ms=15552000000 (6 months) and min.insync.replicas=2 with acks=all to guarantee durability. For longer retention without the Kafka cost, use tiered storage to Ceph RGW or an S3-compatible backend: new log in the hot tier, old log in the cold tier, transparently to the consumer.

Formal fit into incident management

Runbooks are not an isolated SRE practice. They fit into four regulatory frameworks that production LLM platforms touch daily.

ISO/IEC 27035 — information security incident management

It defines the formal cycle in five phases: plan & prepare → detect & report → assess & decide → respond → lessons learned. Each phase has outputs that must be documented. Translated to the stack:

Plan & prepare: runbooks RB-01 to RB-06 plus the Keep workflows are part of the Information Security Incident Management Plan. Versioned in git, reviewed annually.
Detect & report: the Prometheus alerts entering Kafka are the materialisation.
Assess & decide: the severity in gpu.alerts.enriched plus the Keep workflow logic.
Respond: execution of the workflow’s steps plus actions.
Lessons learned: a mandatory postmortem for the runbooks that flag it; the output documented in the postmortem repo plus the runbook update.

ENS (Esquema Nacional de Seguridad) — op.exp controls

op.exp.7 Incident management: the runbook catalogue plus the Keep / Kafka pipeline materialise the “organised and procedural response”.
op.exp.8 Activity logging: the audit.actions topic with WORM retention of 6 months (minimum for level ALTO).
op.exp.9 Incident management logging: the incidents.lifecycle topic with the full cycle of each incident.
op.exp.10 Protection of activity logs: WORM plus encryption at rest plus access control (compliance consumers read-only).

NIS2 — notification to the competent authority

For essential and important entities, art. 23 sets three deadlines from the detection of significant impact:

24 hours: an early warning to the national CSIRT (INCIBE-CERT in Spain).
72 hours: a formal notification with an initial assessment.
1 month: a final report with root cause, impact and corrective measures.

The data for those reports comes straight out of incidents.lifecycle plus audit.actions with a consumer that generates the dossier in the required format. Without the auditable pipeline, the NIS2 deadlines are unreachable.

EU AI Act — art. 73 (serious incident reporting)

Applicable to high-risk systems. Deadlines:

2 days: for incidents causing death or irreversible harm to people or critical infrastructure.
10 days: for incidents producing serious disruption of critical infrastructure.
15 days: for all other “serious incidents”.

The definition of “serious incident” includes systematic model failures, a fundamental rights breach, material or environmental harm. Runbooks must flag which alerts can lead to a serious incident (typically anything affecting the model’s output in a high-risk context) and fire a specific legal assessment sub-workflow.

ISO/IEC 42001 — AIMS clause 10 continual improvement

The mandatory post-incident postmortem feeds clause 10. Updating the runbook after every incident that reveals a new pattern is the “corrective action with verification of effectiveness” the standard demands. See ISO 42001 AIMS.

Four anti-patterns

Anti-pattern 1 — alerts without a runbook. The alert fires, the junior operator on call looks at the dashboard, searches Confluence, finds nothing up to date, calls the senior on Slack, waits 20 minutes. In that time the incident has grown. Rule: no alert goes to production without a published runbook and an approved Keep workflow. CI validates that every PrometheusRule with severity ≥ warning has its corresponding keep workflow.

Anti-pattern 2 — a runbook without prior evidence capture. The workflow runs nvidia-smi --gpu-reset as soon as the XID arrives, losing the state that would have diagnosed the root cause. The next identical XID forces the diagnosis to be redone from scratch. Rule: steps before actions; all evidence is captured first, destructive actions come after.

Anti-pattern 3 — escalation by seniority rota instead of severity. The junior operator on call handles an ECC DBE because “it is their turn”. They lack the context to understand row remap, retired pages or the data corruption risk. Rule: page by severity, not by rota: RB-04 and RB-03 page the senior primary ON-CALL with automatic escalation to infra/hardware if there is no acknowledgement within 10 min.

Anti-pattern 4 — no human gate for destructive actions. The workflow runs kubectl drain automatically on any alert marked CRITICAL. On the first false alarm (a transient that resolved itself in 30 s), Keep drained a production node during peak hour. Rule: destructive actions (drain, reset, RMA, full rollback) require human confirmation via a Slack interactive message, with a configurable timeout. A justified exception: ECC DBE confirmed by > 1 measurement, where the corruption risk outweighs the false alarm risk.

Applied to typical on-premise hardware

For a generic cluster of 4 nodes × 4×H100 SXM 80 GB with Kafka and Keep already deployed:

Kafka: a 3-broker cluster on non-GPU nodes of the K8s cluster; topics gpu.alerts.enriched, incidents.lifecycle, audit.actions configured with replication factor 3, min.insync.replicas 2. Audit with tiered storage to Ceph RGW for retention > 6 months without a brutal cost.
Keep: 2 operator replicas plus 1 worker replica in a keep namespace; connected to Prometheus (read provider), Kafka (read plus write provider), Slack, PagerDuty, Jira, Kubernetes (provider with a dedicated SA holding get/list/patch nodes and create jobs permissions).
Workflows: ~25-40 YAML files in the infra/keep-workflows/ repo, synchronised with the cluster via Flux or Argo CD. Validated by CI (keep workflow validate) on every PR.
Event volume: for 16 GPUs in normal operation with debounced alerts, ~50-200 events/day on gpu.alerts.enriched. In a typical incident, peaks of 500-2,000 events/day.
Compliance consumers: a python consumer in the compliance namespace that generates NIS2 / ENS / EU AI Act reports weekly, reading audit.actions and incidents.lifecycle.

What we have not covered (upcoming posts)

Postmortem playbooks — the mechanics of RCA with 5-whys, Ishikawa adapted to LLMs, integration with MLflow tracking for re-training if the postmortem produces an enriched dataset.
Chaos engineering for LLMs — controlled injection of XID errors, simulated ECC, artificial HBM latency to validate runbooks before the real incident.
Multi-cluster incident coordination — how to coordinate Keep across geographic clusters when an incident affects multiple regions.
CMDB and procurement integration — the RMA → ticket → ServiceNow → hardware replacement cycle automated via workflow.
LLM forensics — extracting the full OTel trace of a request affected by an incident, with redacted PII, kept in an evidence vault.

References

ISO/IEC 27035-1:2023 — Information security incident management — Principles and process.
ISO/IEC 27035-2:2023 — Information security incident management — Guidelines to plan and prepare for incident response.
ENS — Real Decreto 311/2022, Anexo II controles op.exp.7 a op.exp.10.
Directiva NIS2 (UE 2022/2555) — art. 23 (notificación de incidentes significativos).
Reglamento EU AI Act (UE 2024/1689) — art. 73 (reporting of serious incidents).
ISO/IEC 42001:2023 — AI management system — cláusula 10 (mejora continua).
Keep project — keephq.dev y github.com/keephq/keep (documentación de workflows YAML, providers).
Apache Kafka — Tiered Storage y cleanup.policy (docs.confluent.io / kafka.apache.org).
Confluent — Schema Registry y best practices para eventos lifecycle.
NVIDIA — Xid Errors Documentation y procedimientos de remediación.
Google SRE Book — Effective Troubleshooting y Postmortem Culture.
Atlassian — Incident Management Handbook (referencia para severity matrices).

RAG on Kafka: a technical reference architecture for streaming datalakes, with fresh embeddings and vector stores that are always up to date

Thu, 21 May 2026 06:00:00 +0200

TL;DR

The piece that blocks enterprise GenAI projects most in 2026 is not the model, nor even the guardrails: it is data ingestion for RAG. Companies hold valuable information in OLTP databases, in operational logs, in SaaS systems, and all of it is quietly changing every second. Batch RAG systems that reindex every night arrive late (the model’s answer is backed by a snapshot from 18 hours ago) and open the door to operational hallucinations even when the retriever is perfect. The dominant answer in production in 2026 is to build the RAG piece on top of Kafka as the source of truth: an immutable log, massive throughput, managed schema evolution, and a mature stream processing ecosystem (Flink, Kafka Streams, RisingWave) that makes it possible to transform and embed events as they happen, delivering them to vector stores (Milvus, Qdrant, Weaviate, pgvector) in milliseconds. The canonical pattern: source → CDC with Debezium → Kafka topics → Flink SQL with an embedding UDF → sink connector to a vector store → serving with vLLM or equivalent. The 2026 developments that change the game: Confluent Tableflow turns Kafka topics into Iceberg/Delta tables automatically (read from Snowflake/Databricks/Trino with no ETL, 30-50 % lower TCO); native Flink SQL brings openai_embedding() and vector search integrated with Cosmos DB and Amazon S3 Vectors; Confluent’s official MCP server lets AI agents query Kafka/Flink/Tableflow in natural language. This post develops the end-to-end architecture with manifests, Flink SQL code and concrete numbers.

This is the second post in the MLOps series specific to LLMs. The first (2026 landscape) set out the framework. Here we go down to the most operational piece of the stack: how a real enterprise system connects to an LLM agent while keeping RAG fresh without falling into explosive complexity.

The analogy: Kafka as the “single source of truth”

Anyone who has spent time in distributed systems has seen the pattern again and again: an immutable, append-only, replicated, time-ordered log has become the canonical primitive for rebuilding complex systems. DBAs know it as the write-ahead log (PostgreSQL WAL, MySQL binlog). Event system developers know it as event sourcing. Data architects know it as the Kappa architecture. Kafka is the massive, distributed, mature implementation of that primitive: a log that lives on disk, partitioned to scale, replicated for durability, retained by time or size, readable from any point in history.

When you think about RAG, this is exactly what you need. A well-designed RAG system has two critical questions: how do you keep the index fresh? And how do you rebuild the index when something breaks? Kafka answers both naturally: fresh because every change at the source is published as an event to the log and the pipeline processes it in milliseconds; rebuildable because the whole log is there: you drop the vector store, you have the Kafka topic from offset 0, and you build the index back exactly as it was.

There is a second layer to the analogy as well. For a modern GenAI architecture, Kafka plays the role of the WAL of the entire system. Just as the Postgres WAL is the gospel of the database’s state (if you lose the DB but keep the WAL, you can rebuild it), the Kafka log is the gospel of the state of the business as a whole: orders, users, transactions, documents. Connecting your AI agent to Kafka means connecting it to the real pulse of the system, not to stale snapshots.

The static RAG problem

Before presenting the architecture, it is worth pinning down what concrete problem we are solving. The antipattern that trips up most GenAI projects:

A team builds RAG on a static dataset: it dumps Confluence documents, product PDFs, database snapshots.
It embeds the lot with a nightly cron that regenerates the index every 24 hours.
It ships the product.
Day 2: a user asks about a change that happened two hours ago. RAG does not have it; the model answers about the old version.
The team adds fragile logic: “if the query mentions a recent date, escalate to a human agent”.
Day 30: the dataset has moved so much that half the RAG is out of date. The team decides to refactor and migrate to streaming.

It is the repeated story of so many projects that the ecosystem has learned the lesson: streaming from day 1, even when the volume is low. The operational complexity of a well-designed streaming pipeline is constant; the complexity of migrating from batch to streaming on a live project is enormous.

From Lambda to Kappa to Streaming RAG

Three architectures in historical order:

Lambda (the classic big-data design from 2014): two parallel pipelines, one batch for accuracy and one streaming for freshness. The query combines both. It works, but it means maintaining two pipelines.

Kappa (Jay Kreps, 2014, mainstream since 2020): a single streaming pipeline. Batch is a special case of streaming (reprocessing from the beginning). It simplifies a great deal.

Streaming RAG (emerging in 2025-2026): a specific variant of Kappa where the pipeline’s output is embeddings indexed in a vector store that the LLM queries at runtime. The Kafka log is the source of truth, the vector store is a queryable projection.

The mental conversion: think of the vector store as the materialised view of the Kafka log. If the view is corrupted, you rebuild it from the log. If you want a new view (a different embedding model, a different chunking strategy), you create another consumer of the log and build a second view in parallel.

The reference architecture

On to the diagram. I am going to present the canonical architecture that stabilised in 2026, showing where each component fits:

[OLTP DB (Postgres)] [Other sources]
│ │
│ WAL via logical decoding │
▼ ▼
┌──────────────────────────────────────────────────────────┐
│ Debezium / Kafka Connect (Sources) │
└──────────────────────────────────────────────────────────┘
│
▼ produces events
┌──────────────────────────────────────────────────────────┐
│ Kafka cluster │
│ ┌───────────────────────────────────────────────────┐ │
│ │ topic: orders.raw (3 partitions, RF=3) │ │
│ │ topic: users.raw (3 partitions, RF=3) │ │
│ │ topic: documents.raw (6 partitions, RF=3) │ │
│ └───────────────────────────────────────────────────┘ │
│ + Schema Registry (Avro/Protobuf) │
└──────────────────────────────────────────────────────────┘
│
▼ consumes and transforms
┌──────────────────────────────────────────────────────────┐
│ Flink SQL streaming jobs │
│ - chunking text │
│ - calls to the embedding model (UDF) │
│ - enrichment with metadata │
│ - sink to a curated topic: documents.embedded │
└──────────────────────────────────────────────────────────┘
│
┌───────────┼────────────────────┐
▼ ▼ ▼
[Vector store] [Tableflow] [Iceberg/Delta]
Milvus/Qdrant auto-convert for analytics
/pgvector/ topics →
Weaviate tables
│
▼ queried at runtime
┌──────────────────────────────────────────────────────────┐
│ LLM serving (vLLM / SGLang) + Retriever │
│ - receives the agent's query │
│ - searches top-K in the vector store │
│ - builds prompt + context │
│ - generates an answer with citations │
└──────────────────────────────────────────────────────────┘

The five layers you can see (source, ingestion (CDC), transport (Kafka), processing (Flink), storage (vectors + tables)) are the ones that structure any serious RAG on a datalake in 2026. Let us go through each one.

Layer 1 — Sources: your OLTP as the starting point

The typical source is an OLTP database (Postgres, MySQL, SQL Server). It is where the live state of the business lives. The technique for extracting changes in real time is Change Data Capture (CDC): reading the database’s transaction log (PostgreSQL WAL, MySQL binlog) and turning each commit into a Kafka event.

The OSS standard is Debezium. It supports Postgres, MySQL, SQL Server, MongoDB, Oracle, Cassandra and others. The typical deployment is a Kafka Connect cluster with Debezium connectors.

An example Debezium configuration for PostgreSQL:

{
 "name": "postgres-orders-connector",
 "config": {
 "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
 "tasks.max": "1",
 "database.hostname": "postgres.prod.internal",
 "database.port": "5432",
 "database.user": "debezium",
 "database.password": "${secret:postgres-creds}",
 "database.dbname": "ecommerce",
 "database.server.name": "ecommerce-prod",
 "table.include.list": "public.orders,public.users,public.products",
 "publication.autocreate.mode": "filtered",
 "slot.name": "debezium_slot",
 "plugin.name": "pgoutput",
 "topic.prefix": "ecommerce",
 "key.converter": "io.confluent.connect.avro.AvroConverter",
 "value.converter": "io.confluent.connect.avro.AvroConverter",
 "key.converter.schema.registry.url": "http://schema-registry:8081",
 "value.converter.schema.registry.url": "http://schema-registry:8081"
 }
}

For each commit in the database, this produces an Avro event on the corresponding topic (ecommerce.public.orders, ecommerce.public.users, and so on) carrying the change: type (INSERT/UPDATE/DELETE), values before and after, commit timestamp, position in the WAL.

A simpler alternative for 2026: RisingWave can read the Postgres WAL directly, with no Debezium or Kafka Connect in between. When the case is CDC only, with no other sources, it is operationally simpler. For architectures with multiple sources (CDC + APIs + scrapers + logs), Debezium remains the standard piece.

Layer 2 — Kafka as transport and persistence

The Kafka cluster is where all events land. Key operational decisions:

Topics: raw vs curated

The convention that settled in 2026:

*.raw: the raw event exactly as it arrived. Untransformed CDC, unparsed application logs.
*.cleaned: after dedup, schema validation, type normalisation.
*.enriched: after adding metadata (geolocation, cross identifiers, and so on).
*.embedded: the event with its vector embedding already computed.

Multi-stage topics allow debugging per layer and partial reprocessing: if you change the embedding model, discarding *.embedded and rebuilding from *.enriched costs hours; rebuilding from *.raw costs days.

Schema Registry

Without a schema registry, topics break silently when someone changes the schema at the source. Confluent Schema Registry or the OSS Apicurio are the dominant options.

Common formats:

Avro: versioned schema, strict evolution rules. The historical default.
Protobuf: compatible with gRPC stacks, good performance.
JSON Schema: textual, debuggable by eye, less efficient.

For RAG on Kafka we recommend Avro by default. Schema evolution matters because source tables change over time, and an unversioned schema breaks downstream consumers.

Partitions, replication and retention

Operational decisions for RAG topics:

Partitions: typically 3-12. More partitions means more parallelism in the Flink consumer, but more overhead. The rule of thumb: partitions = expected peak events/s ÷ 1000.
Replication factor: 3 as a minimum in production. Replication protects against broker failure; with RAG, the cost of losing a topic can be weeks of re-embedding.
Retention: for topics that feed RAG, use long retention or compaction by key. If document doc-42 changes 100 times, compaction keeps only the last state per key, leaving a smaller, rebuildable log. For data that is never updated (historical logs), retention by time (90 days, 1 year).

Cross-cluster replication

For multi-region or multi-cloud deployments, MirrorMaker 2 or Cluster Linking (Confluent) replicate topics between Kafka clusters. RAG can then query the local cluster without crossing regions.

Layer 3 — Flink as the streaming processor

Apache Flink is the dominant stream processing piece in 2026. Apache 2.0, a mature distribution, a broad ecosystem. The main alternative is Kafka Streams (simpler, Java-only); RisingWave is the emerging option for pure SQL cases.

What Flink adds to Kafka:

Stateful streaming: time-based aggregations, joins between streams, sessions.
Exactly-once semantics: with checkpoint coordination.
Watermarks: correct handling of out-of-order events.
UDFs in Python/Java: including calls to LLM models.

Flink SQL: the most operational piece

Flink SQL is the most usable part of Flink for data engineers who are not streaming experts. Here is a realistic RAG pipeline example:

-- 1. Define the source: Kafka topic with CDC events from documents
CREATE TABLE documents_raw (
 doc_id STRING,
 title STRING,
 body STRING,
 category STRING,
 updated_at TIMESTAMP_LTZ(3),
 PRIMARY KEY (doc_id) NOT ENFORCED
) WITH (
 'connector' = 'upsert-kafka',
 'topic' = 'ecommerce.public.documents',
 'properties.bootstrap.servers' = 'kafka:9092',
 'key.format' = 'avro-confluent',
 'value.format' = 'avro-confluent',
 'value.fields-include' = 'EXCEPT_KEY'
);

-- 2. Define the sink: vector store via an intermediate Kafka topic
CREATE TABLE documents_embedded (
 doc_id STRING,
 chunk_id INT,
 title STRING,
 chunk_text STRING,
 category STRING,
 embedding ARRAY<FLOAT>,
 embedded_at TIMESTAMP_LTZ(3),
 PRIMARY KEY (doc_id, chunk_id) NOT ENFORCED
) WITH (
 'connector' = 'upsert-kafka',
 'topic' = 'rag.documents.embedded',
 'properties.bootstrap.servers' = 'kafka:9092',
 'key.format' = 'json',
 'value.format' = 'json'
);

-- 3. UDF for chunking (defined in Python or Java)
-- CREATE TEMPORARY FUNCTION chunk_text AS 'com.example.ChunkingUDF';

-- 4. Pipeline: chunk, embed, write to the sink
INSERT INTO documents_embedded
SELECT
 doc_id,
 chunk_idx AS chunk_id,
 title,
 chunk AS chunk_text,
 category,
 OPENAI_EMBEDDING(chunk,
 'text-embedding-3-small') AS embedding,
 CURRENT_TIMESTAMP AS embedded_at
FROM documents_raw
CROSS JOIN UNNEST(chunk_text(body, 512, 64))
 WITH ORDINALITY AS t(chunk, chunk_idx);

What happens here, line by line:

The documents_raw table reads the CDC topic in upsert-kafka mode (each new event on the same key replaces the previous one). This correctly reflects the semantics of “this is the latest version of doc 42”.
The documents_embedded table will be the intermediate topic where Flink writes the embedded chunks.
The chunk_text UDF (defined in Python or Java) splits each doc into 512-token chunks with an overlap of 64.
The INSERT INTO query runs continuously: every new event in documents_raw is chunked, each chunk is embedded with OPENAI_EMBEDDING (a built-in Flink SQL function in Confluent Cloud 2026), and written to the embedded topic.

OPENAI_EMBEDDING can be swapped for a custom function that calls a self-hosted model (vLLM with an encoder), SentenceTransformers, or a managed service. The syntax is the same; you change the provider.

Watermarks and late events

For cases where an event can arrive late (for example, the Postgres WAL falls behind because of a network blip), Flink lets you define watermarks:

CREATE TABLE documents_raw (
 doc_id STRING,
 title STRING,
 body STRING,
 updated_at TIMESTAMP_LTZ(3),
 WATERMARK FOR updated_at AS updated_at - INTERVAL '5' MINUTE
) WITH (...)

This tells Flink “assume no event arrives more than 5 minutes late relative to the event timestamp”. For temporal joins and aggregations, Flink uses the watermark to decide when to “close” a window.

Layer 4 — Sinks to vector stores

The last step is indexing the embeddings in a vector store. Three patterns in 2026:

Pattern A — direct Kafka Connect sink

Each vector store has its official connector:

Milvus: official Zilliz sink connector. Supports named/unnamed dense/sparse vectors.
Qdrant: official sink connector. Supports dense, sparse, multi-vector.
pgvector: has no dedicated connector, but the JDBC Sink Connector is used with custom SQL.
Weaviate: community connector.
LanceDB: community connector.

An example Milvus sink configuration:

{
 "name": "milvus-rag-embeddings-sink",
 "config": {
 "connector.class": "com.milvus.io.kafka.MilvusSinkConnector",
 "tasks.max": "3",
 "topics": "rag.documents.embedded",
 "milvus.host": "milvus.prod.internal",
 "milvus.port": "19530",
 "milvus.collection.name": "documents",
 "milvus.collection.dim": "1536",
 "milvus.collection.partition": "default",
 "key.converter": "org.apache.kafka.connect.storage.StringConverter",
 "value.converter": "org.apache.kafka.connect.json.JsonConverter",
 "value.converter.schemas.enable": false
 }
}

Three parallel tasks (tasks.max: 3) consume the embedded topic and write to the Milvus collection. Latency from “event in Kafka” to “indexable vector in Milvus” is typically <5 seconds.

Pattern B — pgvector with a direct CDC pipe

For teams that already live in PostgreSQL, pgvector is the lowest-friction option. The pattern: the same source Postgres cluster holds a second DB for embeddings with the pgvector extension enabled; the Flink pipeline writes to it directly over JDBC.

-- On the Postgres cluster with pgvector enabled
CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE document_embeddings (
 doc_id TEXT,
 chunk_id INT,
 chunk_text TEXT,
 category TEXT,
 embedding vector(1536),
 embedded_at TIMESTAMP,
 PRIMARY KEY (doc_id, chunk_id)
);

CREATE INDEX ON document_embeddings
 USING hnsw (embedding vector_cosine_ops)
 WITH (m = 16, ef_construction = 64);

Advantages: the same DBA operates everything, transactionality across tables, trivial joins with relational metadata. Limitation: beyond 10M vectors, pgvector performance starts to give way compared with dedicated systems.

Pattern C — Confluent Tableflow → Iceberg + Flink SQL vector search

This is the 2026 development that changes the mechanics. Confluent Tableflow materialises Kafka topics automatically as Apache Iceberg or Delta Lake tables. Features:

No ETL pipeline: you do not write Flink/Spark jobs to move Kafka into a table. Tableflow does it.
Automatic schema evolution: changes to the topic schema are reflected in the table.
Unified catalogue: the table appears in Glue, Unity Catalog, Snowflake, Databricks. Any analytical engine queries it without copying data.
Native CDC: handles inserts, updates and deletes correctly.
30-50 % lower TCO according to the figures Confluent publishes against traditional pipelines.

And since 2026, Tableflow + Flink SQL offer native vector search integrated with Cosmos DB and Amazon S3 Vectors. The RAG query can be made directly in Flink SQL:

SELECT doc_id, chunk_text, category
FROM documents_embedded
WHERE VECTOR_SEARCH(embedding,
 OPENAI_EMBEDDING('user query', 'text-embedding-3-small'),
 top_k => 10) > 0.7
ORDER BY VECTOR_SEARCH_SCORE DESC;

This unifies layers that used to be separate (vector store + analytics). For many cases, it removes the need to maintain a dedicated vector store.

Confluent’s official MCP server

One piece added in 2026 deserves a mention: Confluent has published an official MCP server that exposes Kafka, Flink and Tableflow as tools accessible to AI agents over MCP. Any MCP client (Claude Desktop, Cursor, your own agents) can:

List topics, read recent messages, publish to topics.
Run Flink SQL queries in natural language (“give me the orders from the last 24 hours worth more than 1000 €”).
Query Tableflow Iceberg tables.
Manage Kafka Connect connectors.

This closes the circle: as well as reading data from the datalake through RAG (with vector search), your AI agent can write data to the log (through MCP) and trigger transformations (through Flink SQL in natural language). It is the deepest point of fusion between LLM ops and data ops this year.

A connection with the earlier series: this MCP server emits traces with the OpenTelemetry GenAI MCP semantic conventions we covered in the MCP observability post. The spans show up in Langfuse, Phoenix or your OTel backend with the right cardinality. Zero instrumentation code.

Vector stores: a 2026 comparison

The five dominant options:

Vector store	Licence	Operation	Where it fits
pgvector	Postgres ext, OSS	Your DBA	<10M vectors, Postgres-heavy team
Qdrant	Apache 2.0	Self-host or managed	Mid-scale, performance focus
Milvus	Apache 2.0	Self-host or Zilliz Cloud	Large-scale, scalability focus
Weaviate	BSD-3	Self-host or managed	Native hybrid search, semantically rich
LanceDB	Apache 2.0	Embedded or serverless	Small-medium, simplicity

The choice depends on:

Scale: pgvector falls short beyond 10M vectors. Milvus and Qdrant scale to billions.
Hybrid search: Weaviate brings lexical + vector natively. Others support it but less integrated.
Operation: pgvector if you already run Postgres. Qdrant if you want simplicity. Milvus if you need maximum scale.
Cloud managed: Zilliz Cloud for Milvus, Qdrant Cloud for Qdrant, Pinecone if you want pure SaaS (with no OSS behind it).

Freshness vs accuracy: the operational trade-off

A critical decision any RAG system on Kafka must answer: when is a new document considered “live” in the index?

Three options:

Synchronous streaming: the event reaches Kafka, Flink embeds it, the sink writes it to the vector store, and only then is it considered live. Typical latency: 1-5 seconds. The best freshness. But if the embedding model fails or the vector store is slow, events pile up in the topic.

Asynchronous streaming with a baseline: the event is considered live immediately; a background process embeds it when it can. In the meantime, queries asking for that document do not find it. Typical latency: 5-60 seconds. Acceptable for most applications.

Micro-batch: processing happens in mini-batches every 1-5 minutes. Less efficient than continuous streaming but more stable under variable load. Latency: 1-5 minutes.

The decision depends on the product’s SLA. For customer support chatbots, 5-60 seconds is acceptable. For systems that react to critical events (financial prices, alarms), synchronous streaming is necessary.

Schema evolution and re-embedding

When the embedding model changes (you move from text-embedding-3-small to text-embedding-3-large, or from OpenAI to Cohere), the vectors already in the index are incompatible: different dimensions, different semantic spaces. The distance between an old vector and a new one means nothing.

The standard pattern for handling this: dual-index during the migration.

T0: the active index is V1 (embedding model A).
T1: a parallel pipeline starts writing to a V2 index (embedding model B), consuming the topic from offset 0 (reprocessing the whole log).
T2: V2 has caught up with the present.
T3: you switch the retriever over to V2.
T4: a week later, you discard V1.

The Kafka log makes this pattern feasible because it is immutable and reproducible. Without the log, this pattern becomes a weeks-long data migration project.

Operational pitfalls

Topics without adequate retention

Configuring topics with 7-day retention on the grounds that “I already have the vector store” means losing the ability to rebuild if the vector store fails. Long retention (90+ days) or compaction by key for topics that feed RAG.

Heavy CDC during peak load

Debezium reading the WAL at peak hours can hit the source database’s performance. A dedicated read replica for Debezium, not the production primary. Or use logical replication restricted to just the tables you need.

Embedding cost run-away

OPENAI_EMBEDDING on every event of a topic with millions of messages a day comes to thousands of USD per month. Strategies: filter before embedding (only embed what adds value); deduplicate by content hash; use self-hosted open-source embedding models (BGE, E5, GTE) when the cloud cost is prohibitive.

Slow re-embedding due to limited throughput

Recomputing 10M embeddings with the OpenAI API at 3000 req/min takes 55 hours. If you wait for an incident to re-embed, that is two days without service. Embedding throughput is an explicit capacity planning item; reserve capacity or keep an offline job ready to start.

Downstream schema breaks

A change to the raw topic’s schema breaks Flink jobs downstream. Schema Registry with BACKWARD compatibility made mandatory; never ALLOW_ALL. And test schema evolution in CI.

A vector store with no backup

Your vector store holds 50M vectors. It is the only copy (the topics expired). A failure wipes it. Vector stores must be backed up like any primary persistence layer. For Milvus/Qdrant: periodic snapshots. For pgvector: plain pg_dump.

Multi-region without cross-cluster replication

Your RAG serves users in the US and the EU. The vector store is in US-east. Latency from the EU = 100 ms+ per query. MirrorMaker or Cluster Linking to replicate topics and vector stores in both regions.

What we have not covered

Hybrid search in production: combining BM25/lexical + vector + reranker. A topic for its own post.
Multimodal RAG: indexing images, audio and video alongside text. Multimodal embeddings (CLIP, Imagebind), a specific architecture.
GraphRAG: using structured knowledge (knowledge graphs) as well as vector retrieval. Microsoft GraphRAG, LlamaIndex KnowledgeGraphQueryEngine.
RAG with multi-tenant ACLs: filtering by permissions at runtime. The pattern uses metadata in the vector store + server-side filters.
Query rewriting with an LLM: using a first LLM to expand the query before retrieval (HyDE, multi-query, step-back prompting).

References

Kafka and stream processing:

Apache Kafka y Debezium.
Confluent Schema Registry y Apicurio Registry.
Apache Flink y Flink SQL docs.
RisingWave — alternativa SQL streaming con embedding built-in.

Vector store connectors:

Tableflow and 2026 architecture:

Streaming RAG:

Cross-references:

Previous post: MLOps specific to LLMs in 2026: the landscape.
Data series: PostgreSQL + Qdrant at the ingestion stage, RAG corpus curation: the active librarian, Reranker and hybrid retrieval — the four pieces that close the Data block: streaming, ingest, curation and retrieval.
Post-tracing series: Evals, Guardrails, MCP observability, eBPF + drift.
eBPF series: eBPF from zero to Cilium, Tetragon, Hubble, AgentSight.

Kafka on lo0 — Tech Blog

Debezium and CDC: the notary who hears changes before anyone asks

TL;DR

The master analogy: the land registry notary

1. The problem CDC solves

2. What the PostgreSQL WAL is

The operations journal

Physical vs logical replication

The replication slot: the notary’s bookmark

The pgoutput plugin

3. Debezium architecture

The connector as a Kafka Connect plugin

Structure of a Debezium event

Initial snapshot

SMT transformations (Single Message Transforms)

4. Debezium vs the Outbox pattern

5. The maths

Throughput

End-to-end latency

WAL disk blowup risk

6. Use cases in LLMOps / RAG

RAG synchronisation with real deletion

Event sourcing for fine-tuning datasets

Immutable audit log

7. Architecture diagram

8. Minimum configuration

PostgreSQL: enable logical replication

Debezium connector (Kafka Connect REST API)

Minimal consumer in Python

9. On-premise deployment

What we have not covered

See also

References

Incident response runbooks for LLM inference: every alert to a concrete action with Kafka and Keep

TL;DR

You are here: OBSERVE → DEPLOY (incident response closes the loop)

The analogy: a nuclear reactor control room

The incident pipeline architecture

The six critical alerts and their runbooks

RB-01 · GpuHbmNearOom — HBM > 92 % sustained

RB-02 · GpuThermalOrPowerThrottle — bit ≠ 0 and not Idle in CLOCK_THROTTLE_REASONS

RB-03 · GpuXidErrorDetected — increase(DCGM_FI_DEV_XID_ERRORS[5m]) > 0

RB-04 · GpuEccDoubleBit — DCGM_FI_DEV_ECC_DBE_VOL_TOTAL > 0

RB-05 · VllmKvCachePoolNearFull — gpu_cache_usage_perc > 95 % sustained 3 min

RB-06 · VllmTtftP95OutOfSlo — TTFT P95 > 1.5 s for 5 min

Keep YAML workflows — three complete examples

Workflow 1 — xid-detected.yaml

Workflow 2 — ecc-dbe.yaml — immediate paging

Workflow 3 — canary-rollback.yaml — TTFT P95 out of SLO

The canonical Kafka event schema

Formal fit into incident management

ISO/IEC 27035 — information security incident management

ENS (Esquema Nacional de Seguridad) — op.exp controls

NIS2 — notification to the competent authority

EU AI Act — art. 73 (serious incident reporting)

ISO/IEC 42001 — AIMS clause 10 continual improvement

Four anti-patterns

Applied to typical on-premise hardware

What we have not covered (upcoming posts)

See also

References

RAG on Kafka: a technical reference architecture for streaming datalakes, with fresh embeddings and vector stores that are always up to date

TL;DR

The analogy: Kafka as the “single source of truth”

The static RAG problem

From Lambda to Kappa to Streaming RAG

The reference architecture

Layer 1 — Sources: your OLTP as the starting point

Layer 2 — Kafka as transport and persistence

Topics: raw vs curated

Schema Registry

Partitions, replication and retention

Cross-cluster replication

Layer 3 — Flink as the streaming processor

Flink SQL: the most operational piece

Watermarks and late events

Layer 4 — Sinks to vector stores

Pattern A — direct Kafka Connect sink

Pattern B — pgvector with a direct CDC pipe

Pattern C — Confluent Tableflow → Iceberg + Flink SQL vector search

RB-01 · `GpuHbmNearOom` — HBM > 92 % sustained

RB-02 · `GpuThermalOrPowerThrottle` — bit ≠ 0 and not Idle in CLOCK_THROTTLE_REASONS

RB-03 · `GpuXidErrorDetected` — `increase(DCGM_FI_DEV_XID_ERRORS[5m]) > 0`

RB-04 · `GpuEccDoubleBit` — `DCGM_FI_DEV_ECC_DBE_VOL_TOTAL > 0`

RB-05 · `VllmKvCachePoolNearFull` — `gpu_cache_usage_perc > 95 %` sustained 3 min

RB-06 · `VllmTtftP95OutOfSlo` — TTFT P95 > 1.5 s for 5 min

Workflow 1 — `xid-detected.yaml`

Workflow 2 — `ecc-dbe.yaml` — immediate paging

Workflow 3 — `canary-rollback.yaml` — TTFT P95 out of SLO