--- title: "PostgreSQL + Qdrant in RAG ingestion: the postman who keeps two worlds in sync" url: https://blog.lo0.es/en/posts/postgresql-qdrant-rag-ingestion-postman-syncing-two-worlds/ markdown: https://blog.lo0.es/en/posts/postgresql-qdrant-rag-ingestion-postman-syncing-two-worlds/index.md lang: en date: 2026-06-04 lastmod: 2026-09-13 author: "José Manuel Román" description: "How to keep PostgreSQL and Qdrant in sync in a production RAG pipeline: outbox pattern, CDC with Debezium, microservice architecture and throughput maths." tags: ["postgresql","qdrant","rag","ingestion","outbox-pattern","cdc","debezium","microservicios","vector-store","embeddings"] translations: es: https://blog.lo0.es/posts/postgresql-qdrant-ingestion-microservicios/index.md --- # PostgreSQL + Qdrant in RAG ingestion: the postman who keeps two worlds in sync > **TL;DR** — In a production RAG system, PostgreSQL holds the official truth about documents and Qdrant holds the vectors for search. Keeping them in sync is not trivial: if you delete a document from Postgres and do not invalidate its chunks in Qdrant, the system returns answers from ghost documents. There are two patterns to avoid this: the **outbox pattern** (atomic transaction plus asynchronous worker, at-least-once) and **CDC with Debezium** (reading the Postgres WAL directly, low latency, higher complexity). This article explains when to use each one, how to orchestrate them as microservices and what numbers to expect with `bge-m3` on on-premise hardware. --- ## The postman and the civil registry Picture a town with two complementary offices. The first is the **civil registry**: it keeps the official census. Every time someone is born, dies or moves house, the registry is the first to know. It is slow, structured, transactional. If the registry says someone exists, they exist. If it says they died, they are dead. **PostgreSQL is the civil registry of your documents.** The second is the **postman's notebook**: a copy optimised for finding any resident in seconds, organised by area, phonetic names and habitual routes. The postman cannot update the registry, but can search at speeds the registry would never reach. **Qdrant is the postman's notebook.** The problem is synchronisation. If the registry records a death but nobody tells the postman, he will keep trying to deliver letters to an address that no longer exists. In RAG, that translates into indexed chunks of documents that have already been deleted, edited or replaced: ghost documents that pollute the results. How does the registry tell the postman? - **Outbox pattern**: every time the registry updates its ledger, it notes the change on an *outbox sheet*. A messenger clerk reads that sheet periodically and updates the postman's notebook. Guaranteed, asynchronous, fault tolerant. - **CDC with Debezium**: the postman has a direct phone line to the registry. Every time the clerk writes something new, the phone rings and the postman updates his notebook in near real time. --- ## The consistency gap problem In a naive RAG architecture, the flow is: 1. The user uploads a document → it is inserted into Postgres with metadata. 2. A worker splits it into chunks, generates embeddings and upserts into Qdrant. 3. Retrieval uses Qdrant to find relevant chunks and Postgres to hydrate metadata. So far so good. The problem shows up in **updates and deletes**: - The user **edits** a document → Postgres updates the row, but the vectors of the old chunks are still in Qdrant. Retrieval returns stale context. - The user **deletes** a document → Postgres removes the row, but the chunks remain in Qdrant. Retrieval returns chunks from a document that should no longer exist. - The permission system **revokes a tenant's access** → Qdrant has no way of knowing unless there is explicit synchronisation. This is not a theoretical problem. In living corpora (corporate wikis, knowledge bases updated daily), the *consistency gap* accumulates noise progressively. An internal study on production pipelines shows that without active reconciliation, 3-8% of indexed chunks correspond to documents that no longer exist in the source of truth after 30 days of operation. The answer is not "reindex everything every night". With 10M chunks and a non-trivial embedding model, that costs hours of compute and causes unavailability windows. The answer is **change propagation with guarantees**. --- ## Outbox pattern: the outbox sheet ### Mechanism The outbox pattern solves the problem of "writing to two systems in the same operation" without distributed transactions (which are expensive and fragile). The idea is simple: **PostgreSQL is the single coordinator**. When the ingestion microservice processes a document, it performs two writes in the *same local transaction*: 1. It inserts or updates the document in the `documents` table. 2. It inserts an event into the `outbox_events` table. If the transaction fails, both writes are rolled back. If it succeeds, both are committed atomically. There is no inconsistent intermediate state. ```sql -- Relevant tables CREATE TABLE documents ( id UUID PRIMARY KEY DEFAULT gen_random_uuid(), tenant_id TEXT NOT NULL, title TEXT NOT NULL, content TEXT NOT NULL, checksum TEXT NOT NULL, updated_at TIMESTAMPTZ DEFAULT now() ); CREATE TABLE outbox_events ( id BIGSERIAL PRIMARY KEY, aggregate_id UUID NOT NULL, -- document id event_type TEXT NOT NULL, -- 'document.created' | 'document.updated' | 'document.deleted' payload JSONB NOT NULL, created_at TIMESTAMPTZ DEFAULT now(), processed_at TIMESTAMPTZ -- NULL = pending ); -- Example of an atomic insert BEGIN; INSERT INTO documents (tenant_id, title, content, checksum) VALUES ($1, $2, $3, $4) RETURNING id INTO _doc_id; INSERT INTO outbox_events (aggregate_id, event_type, payload) VALUES (_doc_id, 'document.created', jsonb_build_object( 'tenant_id', $1, 'title', $2, 'checksum', $4 )); COMMIT; ``` ### The outbox worker A separate process (the *outbox worker*) polls `outbox_events` where `processed_at IS NULL`, processes each event (chunking, embedding, upsert into Qdrant) and marks the row as processed: ```sql UPDATE outbox_events SET processed_at = now() WHERE id = $1; ``` **Guarantee**: at-least-once. If the worker fails between the Qdrant upsert and the `UPDATE`, the event will be reprocessed. Qdrant tolerates idempotent upserts (same point `id` = overwrite), so reprocessing does not create duplicates. **Latency**: it depends on the polling interval. With polling every 500ms, p50 latency is ~250ms; p99, ~500ms. Acceptable for most RAG cases, where the user does not expect to see a document indexed in under a second. --- ## CDC with Debezium: the direct phone line ### Mechanism *Change Data Capture* (CDC) reads PostgreSQL's **Write-Ahead Log (WAL)** directly. Postgres writes every change to the WAL before applying it to the tables; it is the mechanism it uses for replication and recovery. Debezium subscribes to a *logical replication slot* and turns those events into structured messages. ```sql -- Enable logical replication in postgresql.conf -- wal_level = logical -- Create a replication slot for Debezium SELECT pg_create_logical_replication_slot('debezium_slot', 'pgoutput'); ``` The full flow: ``` Postgres WAL → Debezium connector → Kafka/NATS → Indexer consumer → Qdrant ``` Debezium emits events with the record's *before/after* structure: ```json { "op": "d", "before": { "id": "550e8400-e29b-41d4-a716-446655440000", "tenant_id": "acme", "checksum": "sha256:abc123" }, "after": null } ``` With `"op": "d"` (delete), the consumer knows it must delete every point in Qdrant whose payload contains that `document_id`. ```python # Consumer: delete by payload filter qdrant_client.delete( collection_name="corpus", points_selector=FilterSelector( filter=Filter( must=[ FieldCondition( key="document_id", match=MatchValue(value=event["before"]["id"]) ) ] ) ) ) ``` ### Upsides and downsides CDC removes polling and cuts latency to **tens of milliseconds** (WAL propagation time plus consumer processing). But it adds operational complexity: you need to manage the replication slot (unconsumed slots retain WAL indefinitely, which can fill the disk), the message broker and the consumer offset state. --- ## Comparison: outbox vs CDC | Criterion | Outbox pattern | CDC with Debezium | |---|---|---| | **Typical latency** | 250ms – 2s | 20ms – 200ms | | **Delivery guarantee** | At-least-once | At-least-once | | **Operational complexity** | Low (Postgres only) | High (Debezium + broker) | | **WAL retention risk** | None | High if the slot stalls | | **Idempotency required** | Yes (in the indexer) | Yes (in the consumer) | | **Multi-table support** | Manual | Automatic (any table) | | **Backpressure** | Natural (polling) | Needs explicit design | | **When to choose it** | Corpus < 100k docs/day, small team | Corpus > 1M docs/day, low latency critical | **Rule of thumb**: start with outbox. Move to CDC when the change volume goes past ~50k events/hour or when latency measured in seconds becomes unacceptable for the use case (for example, real-time news indexing). --- ## Microservice architecture The ingestion pipeline is made up of three microservices with cleanly separated responsibilities: > **Diagram:** — [see the HTML version](https://blog.lo0.es/en/posts/postgresql-qdrant-rag-ingestion-postman-syncing-two-worlds/) ### Microservice 1: Ingestor Responsibilities: receive documents, split them into chunks and request embeddings. It does not write to Qdrant directly. ```python # ingestor/main.py (simplified) from langchain_text_splitters import RecursiveCharacterTextSplitter splitter = RecursiveCharacterTextSplitter( chunk_size=512, # tokens, not characters chunk_overlap=64, length_function=token_count, ) def ingest_document(doc: Document, db: Session) -> None: chunks = splitter.split_text(doc.content) with db.begin(): db.execute( "UPDATE documents SET checksum=$1 WHERE id=$2", [doc.checksum, doc.id] ) db.execute( """INSERT INTO outbox_events (aggregate_id, event_type, payload) VALUES ($1, 'document.updated', $2)""", [doc.id, {"chunks": chunks, "tenant_id": doc.tenant_id, "model": "bge-m3", "model_version": "1.0.0"}] ) ``` ### Microservice 2: Indexer It reads the outbox, generates embeddings by calling the inference server (vLLM or TEI) and upserts into Qdrant. ```python # indexer/worker.py import asyncio from qdrant_client import QdrantClient from qdrant_client.models import PointStruct, VectorParams, Distance qdrant = QdrantClient(host="qdrant-service", port=6333) async def process_event(event: dict) -> None: payload = event["payload"] chunks = payload["chunks"] doc_id = str(event["aggregate_id"]) # Batch embedding on TEI (Text Embeddings Inference) embeddings = await embed_batch(chunks, model="bge-m3") points = [ PointStruct( id=f"{doc_id}_{i}", vector=emb, payload={ "document_id": doc_id, "tenant_id": payload["tenant_id"], "chunk_index": i, "text": chunks[i], "model_version": payload["model_version"], } ) for i, emb in enumerate(embeddings) ] # If it is an update, delete the previous chunks first if event["event_type"] in ("document.updated", "document.deleted"): qdrant.delete( collection_name="corpus", points_selector=filter_by_doc_id(doc_id) ) if event["event_type"] != "document.deleted": qdrant.upsert(collection_name="corpus", points=points) ``` ### Microservice 3: Reconciler The reconciler is the safety net. Periodically (every hour, for example) it compares the set of `document_id` values in Postgres with the set of `document_id` values in Qdrant. IDs present in Qdrant but absent from Postgres are *ghosts*: they get deleted. ```python # reconciler/diff.py async def reconcile(tenant_id: str) -> int: pg_ids = set(await fetch_all_doc_ids(tenant_id)) qdrant_ids = set(await scroll_all_doc_ids(tenant_id)) # paginated scroll orphans = qdrant_ids - pg_ids if orphans: logger.warning("Orphan chunks for %d documents", len(orphans)) for doc_id in orphans: qdrant.delete("corpus", filter_by_doc_id(doc_id)) return len(orphans) ``` --- ## Throughput and ingestion maths ### Embedding throughput The `bge-m3` model (1024 dimensions, dense + sparse + colbert support) on a node with **4×H100 SXM (320 GB NVLink)** running via vLLM or HuggingFace TEI reaches roughly **2,000 chunks/second** with batch size = 256 and 512-token sequences. $$\text{throughput} = 4 \times 500 \text{ chunks/s/GPU} = 2{,}000 \text{ chunks/s}$$ > The figure of 500 chunks/s per GPU comes from public TEI benchmarks with bge-m3 on H100 SXM5, batch=256, seq\_len=512 [^1]. ### Total re-ingestion time For a corpus of **10M chunks**: $$t = \frac{10{,}000{,}000 \text{ chunks}}{2{,}000 \text{ chunks/s}} = 5{,}000 \text{ s} \approx 83 \text{ minutes}$$ That is pure embedding time. Adding Qdrant write latency (~0.5ms per upsert batch of 100 points): $$t_{\text{qdrant}} = \frac{10{,}000{,}000}{100} \times 0.5\text{ ms} = 50{,}000 \text{ ms} = 50 \text{ s}$$ Estimated total for a full re-ingestion: **~85-90 minutes** on a 4×H100 node. ### Storage cost in Qdrant Each `bge-m3` vector has **1024 dimensions** in `float32` (4 bytes): $$\text{size per vector} = 1{,}024 \times 4 \text{ B} = 4{,}096 \text{ B} = 4 \text{ KB}$$ For 10M chunks (dense vectors only): $$\text{total vectors} = 10^7 \times 4{,}096 \text{ B} = 40.96 \text{ GB}$$ Adding the JSON payload (estimated at ~500 bytes/chunk): $$\text{payload} = 10^7 \times 500 \text{ B} = 5 \text{ GB}$$ HNSW index (roughly 1.2× the vector size for $m=16$): $$\text{HNSW} \approx 40.96 \text{ GB} \times 1.2 = 49.15 \text{ GB}$$ **Estimated total on disk: ~95 GB** for 10M chunks with dense bge-m3. With `scalar` quantization (int8), the vector size shrinks 4×: $$\text{with int8 quantization} \approx \frac{40.96}{4} + 5 + \frac{49.15}{4} \approx 27.5 \text{ GB}$$ | Configuration | Vectors | HNSW | Payload | Total | |---|---|---|---|---| | float32, no quantization | 40.96 GB | 49.15 GB | 5 GB | ~95 GB | | int8 scalar quantization | 10.24 GB | 12.29 GB | 5 GB | ~28 GB | | binary quantization | 1.28 GB | 1.54 GB | 5 GB | ~8 GB | Binary quantization loses recall precision (~2-5% in NDCG@10), but it lets you host much larger corpora in RAM. For production where recall is critical, int8 is the usual balance point. --- ## Recommended on-premise hardware For a continuous ingestion pipeline in production: **Embedding node**: 4×H100 SXM (320 GB, NVLink), 2× 64-core CPU (EPYC 9654), 1 TB DDR5 RAM, 100 GbE. Runs vLLM or TEI serving `bge-m3`. Sustained throughput: ~2,000 chunks/s with an asynchronous batch pipeline. **Qdrant node**: 32-core CPU, 256 GB RAM (to keep the HNSW index in memory with 10M chunks and no quantization), 2 TB NVMe (Qdrant snapshot and WAL writes). Qdrant recommends that the HNSW index fit in RAM for p99 latency < 5ms. **PostgreSQL node**: 16-core CPU, 128 GB RAM, 4 TB NVMe for the WAL (especially relevant if you use CDC with a logical replication slot; the slot retains WAL until Debezium consumes it). **Broker (if CDC)**: 3-broker Kafka with 500 GB NVMe per node, or NATS JetStream with 3 nodes for more modest loads. --- ## Kubernetes manifests ### Indexer Deployment ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: rag-indexer namespace: rag-pipeline spec: replicas: 3 selector: matchLabels: app: rag-indexer template: metadata: labels: app: rag-indexer spec: containers: - name: indexer image: registry.example.com/rag-indexer:1.0.0 env: - name: POSTGRES_DSN valueFrom: secretKeyRef: name: pg-credentials key: dsn - name: QDRANT_HOST value: "qdrant-service.qdrant.svc.cluster.local" - name: EMBEDDING_ENDPOINT value: "http://tei-service.embeddings.svc.cluster.local:8080" - name: POLL_INTERVAL_MS value: "500" resources: requests: memory: "512Mi" cpu: "500m" limits: memory: "2Gi" cpu: "2" ``` ### Reconciler CronJob ```yaml apiVersion: batch/v1 kind: CronJob metadata: name: rag-reconciler namespace: rag-pipeline spec: schedule: "0 * * * *" # hourly concurrencyPolicy: Forbid jobTemplate: spec: template: spec: restartPolicy: OnFailure containers: - name: reconciler image: registry.example.com/rag-reconciler:1.0.0 env: - name: POSTGRES_DSN valueFrom: secretKeyRef: name: pg-credentials key: dsn - name: QDRANT_HOST value: "qdrant-service.qdrant.svc.cluster.local" - name: SCROLL_PAGE_SIZE value: "1000" resources: requests: memory: "256Mi" cpu: "250m" limits: memory: "1Gi" cpu: "1" ``` --- ## Production gotchas ### Reindexing when the embedding model changes This is the most painful problem. If you move from `bge-m3` to `nomic-embed-text-v2`, the vectors are **incompatible**: they live in different embedding spaces and the cosine distances between them are meaningless. The answer is **dual-index aliasing**: 1. Create a new collection in Qdrant: `corpus_v2`. 2. Re-embed the whole corpus with the new model and load `corpus_v2`. 3. When the new collection is complete and validated (recall test), switch the `corpus_prod` alias from `corpus_v1` to `corpus_v2`. 4. Delete `corpus_v1` once traffic has migrated. During the migration the two indexes coexist. The retriever uses the alias, not the collection's direct name. ### Index versioning Store `model_version` in the payload of every point in Qdrant. This lets you: - Filter by version during retrieval (useful in model A/B testing). - Have the reconciler detect points with an old version and reprocess them selectively. - Audit: know which model generated each embedding. ```python # Filter by model version at retrieval time results = qdrant.search( collection_name="corpus", query_vector=query_embedding, query_filter=Filter( must=[ FieldCondition(key="model_version", match=MatchValue(value="1.0.0")), FieldCondition(key="tenant_id", match=MatchValue(value=tenant_id)), ] ), limit=10 ) ``` ### Namespace per tenant (multi-tenancy) There are two strategies in Qdrant: | Strategy | Pros | Cons | |---|---|---| | **Collection per tenant** | Total isolation, no extra filter | N collections = N HNSW indexes in RAM | | **Payload filter per tenant** | A single collection, less RAM | The filter adds ~10-15% search latency | For fewer than 100 tenants with large corpora (> 1M chunks/tenant), use a collection per tenant. For hundreds or thousands of tenants with small corpora, use a payload filter with `tenant_id` indexed: ```python qdrant.create_payload_index( collection_name="corpus", field_name="tenant_id", field_schema=PayloadSchemaType.KEYWORD ) ``` --- ## What we have not covered - **Streaming corpus updates with near-real-time CDC**: selective chunk invalidation when only one section of a document changes (incremental chunking based on content diff, not full re-chunking). - **Multi-tenant corpus isolation with per-chunk ACLs**: going beyond the `tenant_id` filter towards group, role or even individual document permissions applied at retrieval time. - **Federated corpus**: corpora distributed across cross-border silos where regulation (GDPR, CCPA) prevents centralising the embeddings; federated search patterns without moving data. - **Zero-downtime incremental reindexing using dual-index aliasing**: the full model migration protocol with rollback, recall regression testing and progressive traffic splitting. --- ## See also - [End-to-end document ingestion: from PDF to indexed chunk](https://blog.lo0.es/en/posts/end-to-end-document-ingestion-pdf-to-indexed-chunk/index.md) — the full ingestion pipeline (parsing, chunking, dedup, metadata) around this synchronisation. - [Taking RAG to the CPU: separating the data plane from the generation plane](https://blog.lo0.es/en/posts/rag-cpu-data-plane-generation-plane/index.md) — ingestion and index building are CPU work, not GPU work. - [The curated corpus this architecture has to index](https://blog.lo0.es/en/posts/rag-corpus-curation-active-librarian/index.md) — curation and filtering strategies before ingestion. - [The retrieval that consumes this vector store](https://blog.lo0.es/en/posts/reranker-hybrid-retrieval-five-chunks-llm-reads/index.md) — how the reranker and hybrid retrieval use what we build here. - [The embedder that generates the vectors](https://blog.lo0.es/en/posts/embeddings-2026-dense-sparse-multivector-families/index.md) — a comparison of bge-m3, nomic-embed-text-v2 and multivector models. - [The Data stage of the LLMOps master map](https://blog.lo0.es/en/posts/six-stage-llmops-pipeline-architecture-and-deep-dive/index.md) — where this pipeline sits within the full cycle. - [Versioning the raw corpus before ingestion](https://blog.lo0.es/en/posts/data-versioning-llmops-dvc-lakefs-reproducible-golden-dataset/index.md) — how DVC and LakeFS manage lineage before anything reaches PostgreSQL. - [Debezium and CDC: the notary who listens to changes before anyone asks for them](https://blog.lo0.es/en/posts/debezium-cdc-notary-who-hears-changes/index.md) — the CDC deep dive this article introduces: the Postgres WAL, replication slots, pgoutput and the full comparison with the outbox pattern. --- ## References [^1]: HuggingFace Text Embeddings Inference — benchmarks oficiales con modelos de la familia bge en hardware A100/H100. https://github.com/huggingface/text-embeddings-inference [^2]: Qdrant documentation — *Multitenancy*. https://qdrant.tech/documentation/guides/multiple-partitions/ [^3]: Debezium documentation — *PostgreSQL connector*. https://debezium.io/documentation/reference/stable/connectors/postgresql.html [^4]: Qdrant documentation — *Quantization*. https://qdrant.tech/documentation/guides/quantization/ [^5]: Richardson, C. — *Pattern: Transactional outbox*. https://microservices.io/patterns/data/transactional-outbox.html [^6]: PostgreSQL documentation — *Logical Replication*. https://www.postgresql.org/docs/current/logical-replication.html [^7]: BAAI/bge-m3 model card — HuggingFace. https://huggingface.co/BAAI/bge-m3