Debezium and CDC: the notary who hears changes before anyone asks

TL;DR

Change Data Capture (CDC) with Debezium listens to the PostgreSQL Write-Ahead Log and turns every INSERT, UPDATE and DELETE into a structured Kafka event. Unlike traditional polling (SELECT ... WHERE updated_at > ?), it detects deletes, has latency in the tens of milliseconds and adds no extra load to the database. In RAG pipelines, this means that when a document is deleted from Postgres, the Qdrant chunks disappear too, automatically and in real time. The supporting infrastructure is modest: the connector uses 2-4 cores and 4-8 GB RAM to process thousands of events per second.

The master analogy: the land registry notary

Picture the land registry. Every time a flat is sold, mortgaged, or a mortgage is cancelled, the registrar records the operation in the registry ledger, a chronological and immutable journal. If you want to know what has changed in the registry, you have two options.

Option A (polling): you send someone every 5 minutes with a list of properties to ask “has anything changed?”. Problems: if a title was cancelled (DELETE), the property no longer exists by the time your envoy arrives, and there is no trace of it. If 20 different departments are doing the same thing, that is 20 people pestering the registrar every 5 minutes. And the minimum latency is the interval: 5 minutes.

Option B (Debezium): you hire a notary who sits right at the registrar’s desk. Every time the registrar signs an operation into the ledger, the notary notes it down immediately and notifies whoever needs to know. Title cancellations included, because the notary sees them as clearly as any other operation: he was there when it was signed.

In this analogy:

• The registry ledger is the PostgreSQL WAL (Write-Ahead Log).
• The notary is the Debezium connector.
• The notary’s bookmark, which guarantees he loses no page even if he steps out for a moment, is the logical replication slot.
• The messenger who carries the notifications to interested parties is Kafka (or Redpanda, or NATS JetStream).

We will pick this thread up in every section. When a technical detail is unclear, go back to the image of the notary.

1. The problem CDC solves

The most common synchronisation pattern between services sharing PostgreSQL is periodic polling:

SELECT id, content, updated_at
FROM documents
WHERE updated_at > $1
ORDER BY updated_at
LIMIT 1000;

This pattern has three structural problems.

DELETEs are invisible. When you delete a row, updated_at is not updated: the row is gone. The next time the poller queries, the row does not exist and there is no way to know it ever did. In a RAG pipeline, this means orphan chunks in Qdrant: the document no longer exists in Postgres, but its vectors keep polluting search results.

The minimum latency is the interval. If the poller runs every 5 seconds, average latency is 2.5 seconds. For near-real-time synchronisation (dashboards, alerts, RAG with frequently changing documents), that is too much.

Load scales with the number of consumers. If 10 services poll the same table every 5 seconds, that is 10 × 12 = 120 queries/minute producing no useful work; they only check whether there is anything new. On large tables with complex indexes, that is real load on the database.

CDC inverts the model: the database notifies, the consumers listen. Zero polling, zero extra load, DELETEs included, latency in the tens of milliseconds.

2. What the PostgreSQL WAL is

The operations journal

The Write-Ahead Log (WAL) is the chronological, immutable record of every operation Postgres performs. Before modifying any data page on disk, Postgres writes the operation to the WAL. That sequence, log first then data, is what guarantees durability (the D in ACID) and enables crash recovery: if Postgres dies mid-transaction, on restart it replays the WAL to return the database to a consistent state.

The WAL is the registry ledger of our analogy: chronological, immutable, complete.

Physical vs logical replication

PostgreSQL supports two WAL-based replication modes:

• Physical replication: it replicates disk blocks verbatim. The standby receives the same bytes as the primary. It serves high availability and failover, but the target must be an exact copy of Postgres; you cannot send the changes to an external application.

• Logical replication: instead of disk blocks, it replicates semantic operations: “the row with id=42 was inserted into table documents with these values”. The target can be anything that understands the protocol: another Postgres, Debezium, or any custom consumer.

CDC uses logical replication. It is what lets Debezium understand “what changed and in which table” rather than “which disk block changed at which offset”.

The replication slot: the notary’s bookmark

A logical replication slot is a persistent cursor in the WAL. Postgres keeps a record of how far into the WAL each slot has consumed. While a slot exists, Postgres guarantees it will not discard the WAL segments the slot has not yet read.

This is exactly the notary’s bookmark: even if the notary goes out for lunch, the ledger stays open at the last page he read. When he comes back, he carries on from exactly where he left off, having lost nothing.

The risk is the inverse: if the notary never comes back, the bookmark stops the registrar from archiving the old pages. If the Debezium connector goes down and does not recover for hours, the WAL grows indefinitely on disk until the slot is deleted manually or the consumer starts consuming again. This is called WAL disk blowup and is the most important operational risk of Debezium.

Mandatory monitoring:

SELECT slot_name,
       confirmed_flush_lsn,
       pg_current_wal_lsn(),
       pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn) AS lag_bytes
FROM pg_replication_slots
WHERE slot_type = 'logical';

The pgoutput plugin

The WAL stores operations in an internal binary format. For Debezium to understand them, Postgres needs to decode them into a readable format. The pgoutput decoding plugin, included in the Postgres core since version 10, does exactly that: it translates the binary WAL events into messages with the before/after structure of each row.

Debezium uses pgoutput by default. It requires no external extensions (unlike the wal2json plugin that was popular before Postgres 10).

3. Debezium architecture

The connector as a Kafka Connect plugin

Debezium is not a standalone service; it is a plugin for the Kafka Connect framework. Kafka Connect manages the connector’s lifecycle (start, stop, reconnection, offset tracking) and provides the parallelism and fault tolerance infrastructure.

The connector talks to Postgres through the logical replication protocol (not over JDBC), using the credentials of a user with the REPLICATION role.

PostgreSQL (WAL + pgoutput)
        │
        │ logical replication protocol
        ▼
Debezium Connector (Kafka Connect worker)
        │
        │ Kafka Producer API
        ▼
Kafka topic: rag.public.documents
        │
        ▼
Consumer (Qdrant sync, audit log, fine-tuning pipeline...)

Structure of a Debezium event

Each change in the table becomes a JSON message with this structure:

INSERT ("op": "c" — create):

{
  "before": null,
  "after": {
    "id": 42,
    "content": "Contrato de arrendamiento...",
    "tenant_id": "acme",
    "updated_at": 1748934000000
  },
  "op": "c",
  "source": {
    "version": "2.7.0.Final",
    "connector": "postgresql",
    "db": "rag_db",
    "schema": "public",
    "table": "documents",
    "lsn": 29823948,
    "txId": 1047,
    "ts_ms": 1748934000123
  }
}

DELETE ("op": "d"):

{
  "before": {
    "id": 42,
    "content": "Contrato de arrendamiento...",
    "tenant_id": "acme",
    "updated_at": 1748934000000
  },
  "after": null,
  "op": "d",
  "source": { "lsn": 29824102, "txId": 1051, "ts_ms": 1748934060200 }
}

The before field holds the row’s previous state, available because Postgres can be configured with REPLICA IDENTITY FULL to include the complete row in the WAL on delete or update. Without that setting, before contains only the primary key.

This is the key for the RAG pipeline: the DELETE event carries the document id. The consumer uses it to delete every associated chunk in Qdrant with a doc_id = 42 filter. Without CDC, those chunks would never have been deleted.

Initial snapshot

When the connector starts for the first time (or after a reset), it cannot begin consuming the WAL from “the beginning of time”, only from the moment the slot was created. How does it guarantee the consistency of the initial state?

Through a transactional snapshot: the connector opens a transaction in REPEATABLE READ mode, exports the snapshot ID (pg_export_snapshot()), and does a full SELECT of the configured tables inside that transaction. It then starts consuming the WAL from the snapshot’s LSN. That way there is no gap: the snapshot covers the state up to an instant, and the WAL covers from that instant onwards.

SMT transformations (Single Message Transforms)

Before emitting the event to the Kafka topic, the connector can apply inline transformations called SMT. Common use cases:

• Filtering sensitive columns (ReplaceField with blacklist): strip password_hash, phone_number before they reach the topic.
• Adding metadata (InsertField): enrich the event with a tenant_id extracted from the original HTTP header (if it is in the row).
• Conditional routing (Filter): drop events from rows with status = 'draft' before emitting them.

SMTs are pure configuration, requiring no code, and they are applied inside the connector process with no perceptible extra latency.

4. Debezium vs the Outbox pattern

The Outbox pattern is the most common alternative to pure CDC. Instead of emitting events directly to Kafka, the application writes to an outbox table in Postgres within the same transaction that modifies the data. A separate worker reads that table and publishes the events.

Rule of thumb: if you control the application code and do not need native DELETEs, the Outbox is simpler. If you do not control the code (legacy database, third-party application) or DELETEs are critical (a RAG pipeline with document deletion), Debezium is the right choice.

5. The maths

Throughput

Debezium on a connector with 4 workers can process between 10,000 and 50,000 events/second on modest hardware (4 cores, 8 GB RAM). The real bottleneck is not the connector but the Kafka broker: with 3 brokers and suitable partitions, Kafka can comfortably sustain 500,000 messages/second with 1 KB messages (source: public Confluent benchmarks, 2023).

For a typical RAG pipeline with 100 documents modified per minute:

That is 0.0034% of the connector’s capacity. Debezium will not be the bottleneck in any realistic RAG scenario.

End-to-end latency

The path from a commit in Postgres to an upsert in Qdrant has these stages:

With fetch.min.bytes=1 and fetch.max.wait.ms=10 on the consumer, the Kafka poll latency drops to ~10 ms. The realistic range for an optimised pipeline is 30-100 ms.

WAL disk blowup risk

If the connector stops consuming, Postgres retains the WAL from the slot’s confirmed_flush_lsn onwards. The retained volume grows linearly with time and the write rate:

Example with a moderate load (50,000 writes/hour, 500 bytes average per WAL event):

With a high load (1,000,000 writes/hour):

If the connector is down for 48 hours under high load: 24 GB of retained WAL. That can fill the disk and block Postgres completely.

Recommended alert: set an alert when lag_bytes > 1 GB or when confirmed_flush_lsn has not advanced for more than 15 minutes. See the monitoring query in section 2.

6. Use cases in LLMOps / RAG

RAG synchronisation with real deletion

This is the use case that most clearly justifies Debezium over polling. The flow:

1. A user deletes document id=42 from the document management interface.
2. Postgres runs DELETE FROM documents WHERE id = 42.
3. Debezium detects the DELETE in the WAL and emits the event with "op": "d" and "before": {"id": 42, ...}.
4. The consumer receives the event and runs:

   qdrant_client.delete(
       collection_name="documents",
       points_selector=Filter(must=[FieldCondition(key="doc_id", match=MatchValue(value=42))])
   )
   

5. Every chunk with doc_id=42 disappears from Qdrant in ~100 ms.

Without Debezium, those chunks would stay indefinitely, polluting retrieval results with fragments of documents that no longer exist in the source of truth.

Event sourcing for fine-tuning datasets

Every time a human annotator updates a row in the annotations table (correcting an LLM output), Debezium emits the UPDATE with before and after. The consumer writes the pair (original_output, correction) into the dataset curation pipeline, with no need for the annotator to do anything beyond saving in the interface. The fine-tuning pipeline knows exactly what changed and when, with no polling and no risk of duplicates from overlapping time windows.

Immutable audit log

WAL events are, by definition, the most faithful record of what happened in the database: they are the same data Postgres uses for crash recovery. Kafka with long retention (90 days, or size-based retention) serves as an immutable audit log without modifying the application schema or adding triggers. This is particularly useful in regulated environments where traceability of data modifications is required.

7. Architecture diagram

8. Minimum configuration

PostgreSQL: enable logical replication

-- Requires restarting Postgres
ALTER SYSTEM SET wal_level = logical;
ALTER SYSTEM SET max_replication_slots = 10;
ALTER SYSTEM SET max_wal_senders = 10;

-- Reload configuration (wal_level requires a full restart)
SELECT pg_reload_conf();

-- Dedicated user for Debezium
CREATE USER debezium WITH REPLICATION LOGIN PASSWORD 'cambiar_esto';
GRANT SELECT ON TABLE public.documents TO debezium;

-- REPLICA IDENTITY FULL to get a complete 'before' on DELETEs and UPDATEs
ALTER TABLE public.documents REPLICA IDENTITY FULL;

Debezium connector (Kafka Connect REST API)

{
  "name": "postgres-debezium",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "postgres",
    "database.port": "5432",
    "database.user": "debezium",
    "database.password": "cambiar_esto",
    "database.dbname": "rag_db",
    "topic.prefix": "rag",
    "table.include.list": "public.documents",
    "plugin.name": "pgoutput",
    "slot.name": "debezium_rag",
    "publication.autocreate.mode": "filtered",
    "snapshot.mode": "initial",
    "tombstones.on.delete": "true",
    "transforms": "unwrap",
    "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
    "transforms.unwrap.drop.tombstones": "false",
    "transforms.unwrap.delete.handling.mode": "rewrite"
  }
}

Register the connector:

curl -X POST http://kafka-connect:8083/connectors \
  -H 'Content-Type: application/json' \
  -d @connector-config.json

Check its status:

curl http://kafka-connect:8083/connectors/postgres-debezium/status

Minimal consumer in Python

from confluent_kafka import Consumer
from qdrant_client import QdrantClient
from qdrant_client.models import Filter, FieldCondition, MatchValue
import json

consumer = Consumer({
    "bootstrap.servers": "kafka:9092",
    "group.id": "qdrant-sync",
    "auto.offset.reset": "earliest",
    "enable.auto.commit": False,
})
consumer.subscribe(["rag.public.documents"])
qdrant = QdrantClient("qdrant", port=6333)

while True:
    msg = consumer.poll(timeout=0.1)
    if msg is None:
        continue
    event = json.loads(msg.value())
    op = event.get("op")
    if op in ("c", "u"):  # INSERT or UPDATE
        doc = event["after"]
        # ... vectorise and upsert into Qdrant
    elif op == "d":  # DELETE
        doc_id = event["before"]["id"]
        qdrant.delete(
            collection_name="documents",
            points_selector=Filter(
                must=[FieldCondition(key="doc_id", match=MatchValue(value=doc_id))]
            )
        )
    consumer.commit()

9. On-premise deployment

The Debezium stack does not compete for GPU. On a node with 4×H100 SXM (320 GB, NVLink) serving the inference LLM, the CDC pipeline runs entirely on general-purpose (CPU-only) nodes:

The Debezium connector is notably light: in production at 10,000 events/second, the connector typically uses less than 1 core and 2 GB of RAM. JVM memory (Kafka Connect runs on the JVM) must be capped explicitly with -Xmx4g to stop the GC from causing pauses.

For high availability, Kafka Connect supports distributed mode with multiple workers. If a worker dies, the connector is reassigned automatically to another worker within seconds, and the replication slot guarantees no events are lost during the switchover.

What we have not covered

• Debezium with MySQL, MongoDB and Oracle: each connector uses the native log mechanism (binlog on MySQL, oplog on MongoDB, LogMiner on Oracle). The resulting event API is similar, but the configuration details and limitations differ.
• Debezium Server: standalone mode without Kafka Connect, with direct sinks to HTTP, S3, Redis Streams or NATS. Useful when the Kafka infrastructure is too complex for the use case.
• Schema Registry: how Avro with Confluent Schema Registry or Apicurio manages event schema evolution (adding columns, changing types) without breaking existing consumers.
• Exactly-once semantics: why at-least-once is enough for most RAG cases (an idempotent upsert into Qdrant with the same vector does no harm) and when exactly-once is needed (financial counters, inventory deductions).
• Outbox pattern plus Debezium combined: Debezium reading the outbox table instead of the business table’s WAL directly, the Transactional Outbox + CDC pattern that combines the best of both worlds.

See also

• PostgreSQL + Qdrant: ingestion through microservices — the post where CDC with Debezium is used as an alternative to the outbox pattern to keep PostgreSQL and Qdrant in sync.
• RAG corpus curation: fundamentals — the corpus curation that Debezium keeps fresh in near real time.
• LLMOps pipeline: the six stages — the Data stage of the master map, where CDC is the continuous ingestion mechanism.
• Data versioning with DVC and lakeFS — versioning the corpus that Debezium feeds incrementally.
• GPU observability with DCGM and LLM — monitoring the cluster where the Debezium consumer runs alongside the inference stack.

References

1. Debezium Documentation — PostgreSQL Connector. debezium.io/documentation/reference/stable/connectors/postgresql.html
2. PostgreSQL Documentation — Logical Replication. postgresql.org/docs/current/logical-replication.html
3. PostgreSQL Documentation — Write-Ahead Logging. postgresql.org/docs/current/wal-intro.html
4. PostgreSQL Documentation — Replication Slots. postgresql.org/docs/current/logicaldecoding-explanation.html
5. Confluent — Kafka Performance Benchmarks (2023). confluent.io/blog/kafka-fastest-messaging-system
6. Gunnar Morling — Outbox Pattern. morling.dev/blog/sending-messages-as-part-of-database-transactions
7. Debezium — SMT documentation. debezium.io/documentation/reference/stable/transformations
8. Qdrant Documentation — Filtering. qdrant.tech/documentation/concepts/filtering