PostgreSQL + Qdrant in RAG ingestion: the postman who keeps two worlds in sync

TL;DR — In a production RAG system, PostgreSQL holds the official truth about documents and Qdrant holds the vectors for search. Keeping them in sync is not trivial: if you delete a document from Postgres and do not invalidate its chunks in Qdrant, the system returns answers from ghost documents. There are two patterns to avoid this: the outbox pattern (atomic transaction plus asynchronous worker, at-least-once) and CDC with Debezium (reading the Postgres WAL directly, low latency, higher complexity). This article explains when to use each one, how to orchestrate them as microservices and what numbers to expect with bge-m3 on on-premise hardware.

The postman and the civil registry

Picture a town with two complementary offices.

The first is the civil registry: it keeps the official census. Every time someone is born, dies or moves house, the registry is the first to know. It is slow, structured, transactional. If the registry says someone exists, they exist. If it says they died, they are dead. PostgreSQL is the civil registry of your documents.

The second is the postman’s notebook: a copy optimised for finding any resident in seconds, organised by area, phonetic names and habitual routes. The postman cannot update the registry, but can search at speeds the registry would never reach. Qdrant is the postman’s notebook.

The problem is synchronisation. If the registry records a death but nobody tells the postman, he will keep trying to deliver letters to an address that no longer exists. In RAG, that translates into indexed chunks of documents that have already been deleted, edited or replaced: ghost documents that pollute the results.

How does the registry tell the postman?

• Outbox pattern: every time the registry updates its ledger, it notes the change on an outbox sheet. A messenger clerk reads that sheet periodically and updates the postman’s notebook. Guaranteed, asynchronous, fault tolerant.
• CDC with Debezium: the postman has a direct phone line to the registry. Every time the clerk writes something new, the phone rings and the postman updates his notebook in near real time.

The consistency gap problem

In a naive RAG architecture, the flow is:

1. The user uploads a document → it is inserted into Postgres with metadata.
2. A worker splits it into chunks, generates embeddings and upserts into Qdrant.
3. Retrieval uses Qdrant to find relevant chunks and Postgres to hydrate metadata.

So far so good. The problem shows up in updates and deletes:

• The user edits a document → Postgres updates the row, but the vectors of the old chunks are still in Qdrant. Retrieval returns stale context.
• The user deletes a document → Postgres removes the row, but the chunks remain in Qdrant. Retrieval returns chunks from a document that should no longer exist.
• The permission system revokes a tenant’s access → Qdrant has no way of knowing unless there is explicit synchronisation.

This is not a theoretical problem. In living corpora (corporate wikis, knowledge bases updated daily), the consistency gap accumulates noise progressively. An internal study on production pipelines shows that without active reconciliation, 3-8% of indexed chunks correspond to documents that no longer exist in the source of truth after 30 days of operation.

The answer is not “reindex everything every night”. With 10M chunks and a non-trivial embedding model, that costs hours of compute and causes unavailability windows. The answer is change propagation with guarantees.

Outbox pattern: the outbox sheet

Mechanism

The outbox pattern solves the problem of “writing to two systems in the same operation” without distributed transactions (which are expensive and fragile).

The idea is simple: PostgreSQL is the single coordinator. When the ingestion microservice processes a document, it performs two writes in the same local transaction:

1. It inserts or updates the document in the documents table.
2. It inserts an event into the outbox_events table.

If the transaction fails, both writes are rolled back. If it succeeds, both are committed atomically. There is no inconsistent intermediate state.

-- Relevant tables
CREATE TABLE documents (
    id          UUID PRIMARY KEY DEFAULT gen_random_uuid(),
    tenant_id   TEXT NOT NULL,
    title       TEXT NOT NULL,
    content     TEXT NOT NULL,
    checksum    TEXT NOT NULL,
    updated_at  TIMESTAMPTZ DEFAULT now()
);

CREATE TABLE outbox_events (
    id           BIGSERIAL PRIMARY KEY,
    aggregate_id UUID NOT NULL,        -- document id
    event_type   TEXT NOT NULL,        -- 'document.created' | 'document.updated' | 'document.deleted'
    payload      JSONB NOT NULL,
    created_at   TIMESTAMPTZ DEFAULT now(),
    processed_at TIMESTAMPTZ           -- NULL = pending
);

-- Example of an atomic insert
BEGIN;
  INSERT INTO documents (tenant_id, title, content, checksum)
       VALUES ($1, $2, $3, $4)
  RETURNING id INTO _doc_id;

  INSERT INTO outbox_events (aggregate_id, event_type, payload)
       VALUES (_doc_id, 'document.created', jsonb_build_object(
                   'tenant_id', $1,
                   'title',     $2,
                   'checksum',  $4
               ));
COMMIT;

The outbox worker

A separate process (the outbox worker) polls outbox_events where processed_at IS NULL, processes each event (chunking, embedding, upsert into Qdrant) and marks the row as processed:

UPDATE outbox_events
   SET processed_at = now()
 WHERE id = $1;

Guarantee: at-least-once. If the worker fails between the Qdrant upsert and the UPDATE, the event will be reprocessed. Qdrant tolerates idempotent upserts (same point id = overwrite), so reprocessing does not create duplicates.

Latency: it depends on the polling interval. With polling every 500ms, p50 latency is ~250ms; p99, ~500ms. Acceptable for most RAG cases, where the user does not expect to see a document indexed in under a second.

CDC with Debezium: the direct phone line

Mechanism

Change Data Capture (CDC) reads PostgreSQL’s Write-Ahead Log (WAL) directly. Postgres writes every change to the WAL before applying it to the tables; it is the mechanism it uses for replication and recovery. Debezium subscribes to a logical replication slot and turns those events into structured messages.

-- Enable logical replication in postgresql.conf
-- wal_level = logical

-- Create a replication slot for Debezium
SELECT pg_create_logical_replication_slot('debezium_slot', 'pgoutput');

The full flow:

Postgres WAL → Debezium connector → Kafka/NATS → Indexer consumer → Qdrant

Debezium emits events with the record’s before/after structure:

{
  "op": "d",
  "before": {
    "id": "550e8400-e29b-41d4-a716-446655440000",
    "tenant_id": "acme",
    "checksum": "sha256:abc123"
  },
  "after": null
}

With "op": "d" (delete), the consumer knows it must delete every point in Qdrant whose payload contains that document_id.

# Consumer: delete by payload filter
qdrant_client.delete(
    collection_name="corpus",
    points_selector=FilterSelector(
        filter=Filter(
            must=[
                FieldCondition(
                    key="document_id",
                    match=MatchValue(value=event["before"]["id"])
                )
            ]
        )
    )
)

Upsides and downsides

CDC removes polling and cuts latency to tens of milliseconds (WAL propagation time plus consumer processing). But it adds operational complexity: you need to manage the replication slot (unconsumed slots retain WAL indefinitely, which can fill the disk), the message broker and the consumer offset state.

Comparison: outbox vs CDC

Rule of thumb: start with outbox. Move to CDC when the change volume goes past ~50k events/hour or when latency measured in seconds becomes unacceptable for the use case (for example, real-time news indexing).

Microservice architecture

The ingestion pipeline is made up of three microservices with cleanly separated responsibilities:

Microservice 1: Ingestor

Responsibilities: receive documents, split them into chunks and request embeddings. It does not write to Qdrant directly.

# ingestor/main.py (simplified)
from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
    chunk_size=512,       # tokens, not characters
    chunk_overlap=64,
    length_function=token_count,
)

def ingest_document(doc: Document, db: Session) -> None:
    chunks = splitter.split_text(doc.content)
    with db.begin():
        db.execute(
            "UPDATE documents SET checksum=$1 WHERE id=$2",
            [doc.checksum, doc.id]
        )
        db.execute(
            """INSERT INTO outbox_events (aggregate_id, event_type, payload)
               VALUES ($1, 'document.updated', $2)""",
            [doc.id, {"chunks": chunks, "tenant_id": doc.tenant_id,
                      "model": "bge-m3", "model_version": "1.0.0"}]
        )

Microservice 2: Indexer

It reads the outbox, generates embeddings by calling the inference server (vLLM or TEI) and upserts into Qdrant.

# indexer/worker.py
import asyncio
from qdrant_client import QdrantClient
from qdrant_client.models import PointStruct, VectorParams, Distance

qdrant = QdrantClient(host="qdrant-service", port=6333)

async def process_event(event: dict) -> None:
    payload = event["payload"]
    chunks   = payload["chunks"]
    doc_id   = str(event["aggregate_id"])

    # Batch embedding on TEI (Text Embeddings Inference)
    embeddings = await embed_batch(chunks, model="bge-m3")

    points = [
        PointStruct(
            id=f"{doc_id}_{i}",
            vector=emb,
            payload={
                "document_id": doc_id,
                "tenant_id":   payload["tenant_id"],
                "chunk_index": i,
                "text":        chunks[i],
                "model_version": payload["model_version"],
            }
        )
        for i, emb in enumerate(embeddings)
    ]

    # If it is an update, delete the previous chunks first
    if event["event_type"] in ("document.updated", "document.deleted"):
        qdrant.delete(
            collection_name="corpus",
            points_selector=filter_by_doc_id(doc_id)
        )

    if event["event_type"] != "document.deleted":
        qdrant.upsert(collection_name="corpus", points=points)

Microservice 3: Reconciler

The reconciler is the safety net. Periodically (every hour, for example) it compares the set of document_id values in Postgres with the set of document_id values in Qdrant. IDs present in Qdrant but absent from Postgres are ghosts: they get deleted.

# reconciler/diff.py
async def reconcile(tenant_id: str) -> int:
    pg_ids    = set(await fetch_all_doc_ids(tenant_id))
    qdrant_ids = set(await scroll_all_doc_ids(tenant_id))  # paginated scroll

    orphans = qdrant_ids - pg_ids
    if orphans:
        logger.warning("Orphan chunks for %d documents", len(orphans))
        for doc_id in orphans:
            qdrant.delete("corpus", filter_by_doc_id(doc_id))

    return len(orphans)

Throughput and ingestion maths

Embedding throughput

The bge-m3 model (1024 dimensions, dense + sparse + colbert support) on a node with 4×H100 SXM (320 GB NVLink) running via vLLM or HuggingFace TEI reaches roughly 2,000 chunks/second with batch size = 256 and 512-token sequences.

The figure of 500 chunks/s per GPU comes from public TEI benchmarks with bge-m3 on H100 SXM5, batch=256, seq_len=512 ¹.

Total re-ingestion time

For a corpus of 10M chunks:

That is pure embedding time. Adding Qdrant write latency (~0.5ms per upsert batch of 100 points):

Estimated total for a full re-ingestion: ~85-90 minutes on a 4×H100 node.

Storage cost in Qdrant

Each bge-m3 vector has 1024 dimensions in float32 (4 bytes):

For 10M chunks (dense vectors only):

Adding the JSON payload (estimated at ~500 bytes/chunk):

HNSW index (roughly 1.2× the vector size for $m=16$):

Estimated total on disk: ~95 GB for 10M chunks with dense bge-m3.

With scalar quantization (int8), the vector size shrinks 4×:

Binary quantization loses recall precision (~2-5% in NDCG@10), but it lets you host much larger corpora in RAM. For production where recall is critical, int8 is the usual balance point.

Recommended on-premise hardware

For a continuous ingestion pipeline in production:

Embedding node: 4×H100 SXM (320 GB, NVLink), 2× 64-core CPU (EPYC 9654), 1 TB DDR5 RAM, 100 GbE. Runs vLLM or TEI serving bge-m3. Sustained throughput: ~2,000 chunks/s with an asynchronous batch pipeline.

Qdrant node: 32-core CPU, 256 GB RAM (to keep the HNSW index in memory with 10M chunks and no quantization), 2 TB NVMe (Qdrant snapshot and WAL writes). Qdrant recommends that the HNSW index fit in RAM for p99 latency < 5ms.

PostgreSQL node: 16-core CPU, 128 GB RAM, 4 TB NVMe for the WAL (especially relevant if you use CDC with a logical replication slot; the slot retains WAL until Debezium consumes it).

Broker (if CDC): 3-broker Kafka with 500 GB NVMe per node, or NATS JetStream with 3 nodes for more modest loads.

Kubernetes manifests

Indexer Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
  name: rag-indexer
  namespace: rag-pipeline
spec:
  replicas: 3
  selector:
    matchLabels:
      app: rag-indexer
  template:
    metadata:
      labels:
        app: rag-indexer
    spec:
      containers:
      - name: indexer
        image: registry.example.com/rag-indexer:1.0.0
        env:
        - name: POSTGRES_DSN
          valueFrom:
            secretKeyRef:
              name: pg-credentials
              key: dsn
        - name: QDRANT_HOST
          value: "qdrant-service.qdrant.svc.cluster.local"
        - name: EMBEDDING_ENDPOINT
          value: "http://tei-service.embeddings.svc.cluster.local:8080"
        - name: POLL_INTERVAL_MS
          value: "500"
        resources:
          requests:
            memory: "512Mi"
            cpu: "500m"
          limits:
            memory: "2Gi"
            cpu: "2"

Reconciler CronJob

apiVersion: batch/v1
kind: CronJob
metadata:
  name: rag-reconciler
  namespace: rag-pipeline
spec:
  schedule: "0 * * * *"   # hourly
  concurrencyPolicy: Forbid
  jobTemplate:
    spec:
      template:
        spec:
          restartPolicy: OnFailure
          containers:
          - name: reconciler
            image: registry.example.com/rag-reconciler:1.0.0
            env:
            - name: POSTGRES_DSN
              valueFrom:
                secretKeyRef:
                  name: pg-credentials
                  key: dsn
            - name: QDRANT_HOST
              value: "qdrant-service.qdrant.svc.cluster.local"
            - name: SCROLL_PAGE_SIZE
              value: "1000"
            resources:
              requests:
                memory: "256Mi"
                cpu: "250m"
              limits:
                memory: "1Gi"
                cpu: "1"

Production gotchas

Reindexing when the embedding model changes

This is the most painful problem. If you move from bge-m3 to nomic-embed-text-v2, the vectors are incompatible: they live in different embedding spaces and the cosine distances between them are meaningless.

The answer is dual-index aliasing:

1. Create a new collection in Qdrant: corpus_v2.
2. Re-embed the whole corpus with the new model and load corpus_v2.
3. When the new collection is complete and validated (recall test), switch the corpus_prod alias from corpus_v1 to corpus_v2.
4. Delete corpus_v1 once traffic has migrated.

During the migration the two indexes coexist. The retriever uses the alias, not the collection’s direct name.

Index versioning

Store model_version in the payload of every point in Qdrant. This lets you:

• Filter by version during retrieval (useful in model A/B testing).
• Have the reconciler detect points with an old version and reprocess them selectively.
• Audit: know which model generated each embedding.

# Filter by model version at retrieval time
results = qdrant.search(
    collection_name="corpus",
    query_vector=query_embedding,
    query_filter=Filter(
        must=[
            FieldCondition(key="model_version", match=MatchValue(value="1.0.0")),
            FieldCondition(key="tenant_id", match=MatchValue(value=tenant_id)),
        ]
    ),
    limit=10
)

Namespace per tenant (multi-tenancy)

There are two strategies in Qdrant:

For fewer than 100 tenants with large corpora (> 1M chunks/tenant), use a collection per tenant. For hundreds or thousands of tenants with small corpora, use a payload filter with tenant_id indexed:

qdrant.create_payload_index(
    collection_name="corpus",
    field_name="tenant_id",
    field_schema=PayloadSchemaType.KEYWORD
)

What we have not covered

• Streaming corpus updates with near-real-time CDC: selective chunk invalidation when only one section of a document changes (incremental chunking based on content diff, not full re-chunking).
• Multi-tenant corpus isolation with per-chunk ACLs: going beyond the tenant_id filter towards group, role or even individual document permissions applied at retrieval time.
• Federated corpus: corpora distributed across cross-border silos where regulation (GDPR, CCPA) prevents centralising the embeddings; federated search patterns without moving data.
• Zero-downtime incremental reindexing using dual-index aliasing: the full model migration protocol with rollback, recall regression testing and progressive traffic splitting.

See also

• End-to-end document ingestion: from PDF to indexed chunk — the full ingestion pipeline (parsing, chunking, dedup, metadata) around this synchronisation.

• Taking RAG to the CPU: separating the data plane from the generation plane — ingestion and index building are CPU work, not GPU work.

• The curated corpus this architecture has to index — curation and filtering strategies before ingestion.

• The retrieval that consumes this vector store — how the reranker and hybrid retrieval use what we build here.

• The embedder that generates the vectors — a comparison of bge-m3, nomic-embed-text-v2 and multivector models.

• The Data stage of the LLMOps master map — where this pipeline sits within the full cycle.

• Versioning the raw corpus before ingestion — how DVC and LakeFS manage lineage before anything reaches PostgreSQL.

• Debezium and CDC: the notary who listens to changes before anyone asks for them — the CDC deep dive this article introduces: the Postgres WAL, replication slots, pgoutput and the full comparison with the outbox pattern.

References