Ingestion on lo0 — Tech Blog

End-to-end document ingestion: from PDF to indexed chunk

Thu, 11 Jun 2026 09:00:00 +0000

Fourth piece in an operational series about squeezing a generic on-premise LLM cluster of 4×H100 SXM 80 GB. Its siblings in this batch: serving embeddings and rerankers with TEI in production details the inference piece that this ingestion feeds; GitOps for the inference stack with Flux versions and deploys this whole pipeline; and hardening and secrets for the sovereign stack protects the corpus and the credentials this ingestion touches. The final consumer of what we build here, an end-to-end sovereign assistant, is assembled in a later instalment.

TL;DR

A RAG system inherits the quality of its corpus, and the corpus inherits the quality of the ingestion that manufactured it. This is the garbage-in/garbage-out of retrieval: a chunk badly extracted from a table, a page header repeated a thousand times, a scanned PDF from which you only got OCR noise; all of it gets embedded, indexed and reappears as poisoned context in the model’s answer. Document ingestion is not an afternoon’s script, it is a six-stage pipeline with engineering decisions at every one: (1) extract/parse, where almost everything is decided: layout-aware (Docling, which with Granite-Docling-258M preserves tables, formulas and structure, and which according to IBM avoids classic OCR and is up to ~30× faster) against plain text extraction (PyMuPDF), with OCR or a VLM for scans; (2) clean/normalise, removing boilerplate, normalising Unicode, rebuilding broken paragraphs; (3) chunk, and here 2026 has shifted the consensus: a February 2026 benchmark put recursive at 512 tokens in the lead (69% accuracy) ahead of semantic chunking (54%), and late chunking adds global context at no extra storage cost; (4) enrich metadata, with source, page, section, timestamp, ACL/tenant: for filtered retrieval, citation and auditability; (5) embed, through an embeddings server such as TEI; (6) index, in pgvector or Qdrant, with payload, dense + sparse. In between, exact deduplication (hash) and near-duplicate detection (MinHash/LSH or cosine with a threshold), because near-duplicates degrade retrieval recall and diversity. The numbers matter: a corpus of $N_{docs}\times$ pages $\times$ tokens/page gives the total tokens and, at a given CPU throughput, the ingestion time; and $N_{chunks}\times d\times\text{bytes}$ gives the index size, which in int8 falls 4× against fp32. The reading for the 4×H100: ingestion is CPU work, which links to the RAG-on-CPU piece; the GPU only comes in if you parse with a VLM (Granite-Docling) or use a 7B embedder.

The analogy: an archive’s cataloguing chain

Picture the cataloguing department of a large document archive. It is not one person putting papers into boxes; it is a chain of workstations, each with a different craft and its own quality standard. A document goes in raw at one end and comes out the other turned into a record you can find in seconds. If one station does its job badly, the ones downstream inherit the error and amplify it.

The first station is intake and reading. A heterogeneous box arrives: typed reports, crooked photocopies, folded tables, microfiche. An expert archivist really reads each item: distinguishes the body of the text from the margins, rebuilds a table that spans two pages, transcribes by hand what the scanner could not read. A novice archivist, by contrast, photocopies everything flat and hands over a slab of text where the table columns are interleaved into nonsense. That is exactly the difference between layout-aware parsing and plain text extraction, and it is where almost all of the quality is won or lost.

The second station is weeding out. Before filing anything, someone removes the duplicates (three copies of the same memo, two near-identical versions of a report) and cleans off the useless marks: “COPY” stamps, footers repeated on every sheet, coffee stains. If you do not weed, the archive fills up with copies that, when someone searches, return the same document six times and bury everything else. This is deduplication and cleaning.

The third station cuts it into records. A 400-page book is not catalogued as one giant record; it is broken down into manageable entries, by chapter, by section, of a size a reader can take in at a glance. Too big and the record mixes topics; too small and it loses context. This is chunking, and the size of the record is the decision that most shapes retrieval.

The fourth station labels. Each record carries a shelf mark, a date, its provenance, an access level (can anyone see this, or only the legal department?). Without those labels you cannot filter a search or say which document a statement came from. These are the metadata: source, page, section, timestamp, ACL.

The fifth and sixth stations place the record on the indexed shelf: they translate it into the catalogue’s language, a vector, and put it in the right drawer of the filing cabinet, so that a query finds the related records without walking through the whole thing. This is embedding and indexing in the vector store.

The moral runs through the whole post: RAG cannot retrieve better than what the cataloguing chain filed. If the first station chopped up a table badly, no reranker will fix it afterwards. Ingestion is the quality station for the entire system, and almost all of it, like the archive full of patient people working with nobody timing them, is background work that fits on CPU.

The six-stage pipeline

The temptation to treat ingestion as “read the PDF and chop it up” is the source of 80% of mediocre RAG systems. Each stage has a quality standard and a characteristic failure, and failures cascade downstream: if you parse badly, cleaning does not recover what was lost; if you chunk badly, embedding freezes the error into the vector; if you do not label, you will not be able to filter or cite. The rest of the post walks through the six stations and their decisions.

Stage 1 — Parsing: layout-aware vs plain text

Almost everything is won or lost here. A PDF is not text: it is a set of instructions for drawing glyphs on a canvas. “Extracting the text” from a PDF means reconstructing a reading order that the format does not guarantee, and tables, columns and figures break it systematically.

There are two philosophies, and the choice shapes the rest of the pipeline.

Plain text extraction. Tools such as PyMuPDF (fast, robust, no heavy dependencies) read the PDF’s text stream and dump it out. For single-column documents, running prose and no tables, it is perfect: extremely fast, faithful and cheap on CPU. Its limit shows up with structure: a two-column table comes out with the cells interleaved, a double-column document mixes the end of one with the start of the other, and a scanned invoice does not come out at all because there is no text layer. PyMuPDF can also segment by the table of contents (TOC) when the PDF carries one, which helps with chunking by section (Omdena, Document Parsing for RAG: A Complete Guide for 2026).

Layout-aware parsing. Tools such as Docling (an open source project driven by IBM Research) and unstructured.io first understand the layout, identifying titles, paragraphs, tables, figures, lists and formulas, and only then extract the content while respecting that structure. Docling captures table structure (rows, columns, multi-level headers) and, in its 2026 evolution, does so with a VLM: Granite-Docling-258M, released by IBM in January 2026 under Apache 2.0, a compact vision-language model (~258M parameters, Granite 3 backbone + SigLIP2 visual encoder) that converts pages, whether PDF, slides or scans, directly into a structured format called DocTags, preserving tables, code, inline and block mathematics, and the document’s hierarchy (IBM, Granite-Docling: End-to-end document understanding; model card on Hugging Face; Docling repo). IBM states that the VLM route avoids classic OCR and that this “reduces errors and speeds up the solution by up to 30×” against a traditional OCR pipeline. That is an IBM Research figure; I quote it and treat it as indicative, not as an independent benchmark reproduced here.

unstructured.io offers partitioning strategies graded by document complexity: fast (plain text, quick), hi_res (identifies the layout, recommended when classifying tables and elements properly matters), VLM and auto, balancing speed, cost and accuracy (Unstructured, PDF Parsing Strategies for RAG). The practical rule: fast for prose, hi_res or VLM for documents with tables and structure.

OCR for scans

When the document has no text layer (a scan, a photo, a digitised microfiche), you have to recognise the characters. Three routes:

Classic OCR (Tesseract, PaddleOCR, EasyOCR, which is what Docling integrates with when explicit OCR is needed). Mature, CPU-friendly, good with clean text; it struggles with tables, handwriting and complex layouts.
End-to-end VLM (Granite-Docling and similar). The model “looks” at the page and emits structure directly, with no separate OCR stage. Better with complex layouts; this is where the GPU does come in if the VLM is large or the volume is high.
Hybrid: OCR for character transcription, a layout model for the structure.

The honest criterion: for a corpus of native PDFs with text, PyMuPDF or unstructured fast handle it on CPU and cheaply. For a corpus with dense tables, forms or scans, layout-aware Docling/Granite-Docling earns its cost in chunk quality, and it is the only stage of the pipeline where the GPU can be justified.

Document case	Recommended tool	Silicon
Native PDF, single column, prose	PyMuPDF / unstructured `fast`	CPU
PDF with tables, double column, hierarchy	Docling / unstructured `hi_res`	CPU (layout models)
Scan, form, handwriting, complex layout	Granite-Docling (VLM) or OCR+layout	GPU if large VLM / high volume
HTML, DOCX, PPTX	Docling (multi-format) / native parsers	CPU

Stage 2 — Cleaning, normalising and deduplicating

Freshly parsed text comes in dirty. Cleaning means removing what adds nothing and normalising what is represented in a thousand ways:

Boilerplate: headers and footers repeated on every sheet, page numbers, watermarks, HTML navigation menus, cookie banners. If you do not remove them, they get embedded a thousand times over and contaminate both the index and the answers.
Unicode normalisation (NFC/NFKC), whitespace and dashes: the same character represented in several ways breaks exact matching and dirties the embeddings.
Paragraph reconstruction: undoing the hard line breaks the PDF inserted mid-sentence, without merging paragraphs that were meant to stay separate.

Deduplication: why it matters

RAG suffers two ills from duplicates. The exact kind, the same document uploaded three times, inflates the index and makes a search return the same answer repeatedly, wasting the top-k context slots. The near-duplicate kind, two near-identical versions of a report, a document and its draft, is worse: they look different to a hash but say the same thing, and they degrade retrieval recall and diversity. This is not theory: in the MS MARCO V2 collection a substantial overlap of near-duplicates has been documented which, left untreated, degrades retrieval precision and reduces the diversity of retrieved documents (Ragnarök / TREC RAG 2024).

Two levels of dedup, complementary:

Exact (hash). Compute a sha256 of the normalised content of each document (or chunk) and discard the matches. Cost $O(N)$, trivial. It catches byte-for-byte duplicates.
Near-dup (MinHash + LSH, or cosine of embeddings with a threshold). For those that differ slightly but mean the same thing. MinHash compresses each document into a compact signature such that the probability of two signatures matching at a position equals the Jaccard similarity of the original shingle sets; combined with Locality-Sensitive Hashing (LSH) it finds every near-duplicate pair without comparing all against all, reducing a quadratic problem to almost linear (Brenndoerfer, MinHash, Jaccard, LSH). It is the dominant technique in cleaning LLM training corpora (C4 and RefinedWeb use it) and applies just the same to a RAG corpus (Zilliz, Data Deduplication at Trillion Scale). The alternative, cosine of embeddings with a threshold (say > 0.97), catches semantic duplicates that MinHash cannot see (paraphrasing), but it requires you to already have the embeddings and is more expensive.

The practical rule: exact hash always (it is free); MinHash/LSH for large corpora with versions; cosine with a threshold if paraphrasing is a real problem. And deduplicate before embedding: re-embedding a duplicate means spending compute on rubbish you will then have to filter out.

Stage 3 — Chunking

Chunking is the decision that most shapes retrieval, and the one that carries the most myths. The trade-off is threefold: chunk size ↔ retrieval granularity ↔ context cost.

Large chunks: each one holds more context and there is less risk of cutting an idea in half, but the search is less precise (one vector represents too many topics) and, on retrieval, you put more tokens into the LLM’s prompt, which means more cost and more risk of diluting what is relevant.
Small chunks: very granular, precise retrieval, but each chunk loses context (a 43-token fragment may mean nothing outside its section) and you need to retrieve more of them to cover an answer.

The strategies, from least to most sophisticated:

Fixed size + overlap. Cut every $N$ tokens with an overlap of $k$ tokens between consecutive chunks so as not to slice a sentence dead. Simple, predictable, a reasonable baseline. The overlap is the insurance against cutting an idea right at the boundary.
Recursive (LangChain’s RecursiveCharacterTextSplitter). It tries to cut on separators in priority order (paragraph, then sentence, then word) to respect the natural structure as far as possible before falling back to a hard cut. It is the workhorse.
Semantic. Embed sentences and cut where the similarity between consecutive sentences falls below a threshold, grouping by coherence of meaning. It sounds better on paper; in 2026 practice it has disappointed (see below).
Structure/layout-aware (by headings). It exploits the hierarchy the layout-aware parser has already extracted: one chunk per section or subsection. unstructured offers the by_title strategy, which opens a new chunk when a title element appears, avoiding mixing text from different sections (Unstructured docs). If you parsed with Docling/hi_res, this strategy is almost free and is usually the best one for well-structured documents.
Late chunking. The twist of 2024–2026: instead of chunking and then embedding each chunk separately, embed the whole document first (with a long-context encoder) and then apply the chunk boundaries by mean-pooling the token embeddings inside each span. The result: every chunk keeps the document’s global context (a pronoun or a reference that only makes sense given the previous paragraph is encoded in the vector) and all of it with no extra storage cost, because you end up with one vector per chunk just as before (Jina AI, Late Chunking; arXiv:2409.04701).

What the 2026 benchmarks say (and why semantic disappoints)

It is worth being sceptical about fashion. A Vecta benchmark from February 2026 covering 7 strategies over 50 academic papers put recursive at 512 tokens in first place with 69% accuracy, while semantic chunking came in at 54%, partly because it produced tiny fragments, averaging 43 tokens, too small to mean anything (Firecrawl, Best Chunking Strategies for RAG in 2026). A systematic analysis from January 2026 also identified a “context cliff” around 2,500 tokens, where answer quality drops once you feed in contexts that are too long, an extra argument against giant chunks ([ibid.]). The honest reading: recursive at a moderate size with overlap remains the baseline that is hard to beat; late chunking is the improvement with the best cost/benefit ratio when the model supports it; semantic promises more than it delivers.

A worked numerical example of chunking

Take a technical document of 30 pages, with ~500 tokens of useful prose per page after cleaning (tables and figures are chunked separately). That is $30 \times 500 = 15{,}000$ tokens of text. We chunk with recursive at 512 tokens and 20% overlap ($0.20 \times 512 \approx 102$ tokens). The effective “step” between the start of one chunk and the next is:

$$\text{step} = \text{size} - \text{overlap} = 512 - 102 = 410 \text{ tokens}$$

The number of chunks in the document is then, approximately:

$$N_{chunks} \approx \left\lceil \frac{T_{doc} - \text{overlap}}{\text{step}} \right\rceil = \left\lceil \frac{15{,}000 - 102}{410} \right\rceil \approx \lceil 36.3 \rceil = 37 \text{ chunks}$$

Without overlap it would have been $\lceil 15{,}000 / 512 \rceil = 30$ chunks; the 20% overlap costs us 7 extra chunks (~23% more) in exchange for not splitting ideas at the boundaries. That is the concrete price of overlap: more vectors to embed, index and store, in exchange for robustness in retrieval. We will use this factor when sizing the full corpus.

Stage 4 — Enriching with metadata

A chunk with no metadata is a record with no shelf mark: it exists but it is no use. Each chunk gets a payload attached with, at a minimum:

Source and location: document_id, document name/URI, page number, section/heading (which the layout-aware parser already gave you). Essential for citation: being able to say “this comes from document X, page 12, section 3.2” is what separates an auditable RAG from one that hallucinates with no traceability.
Timestamp: when it was ingested and the document’s date. It lets you filter by freshness and detect stale content.
ACL / tenant: who is allowed to see this chunk. It is critical and it is applied as a filter at retrieval time: a user from department A must not retrieve chunks marked for B only. Without this, the RAG system is a data leak waiting to happen.
Embedding model version (model_version): so you know which embedder generated each vector and can migrate without mixing incompatible spaces.

These metadata are not decoration: they enable filtered retrieval (searching only in what the user can see, or only in documents after a given date), citation (reconstructing the origin of every statement) and auditability (knowing what was retrieved, from where and when). All of it lives in the point’s payload in the vector store.

Stages 5 and 6 — Embedding and indexing

The last two stations translate the chunk into a vector and place it on the shelf.

Embedding. The chunks are sent in batches to an embeddings server, typically Hugging Face’s TEI (Text Embeddings Inference), which exposes the OpenAI /v1/embeddings contract and runs on CPU or GPU. It is throughput-bound work with no latency SLA: its natural home is the CPU with a small encoder in int8 (the sibling piece in this batch, serving embeddings and rerankers with TEI, details the how). It is worth emitting dense + sparse at the same time: the dense vector captures semantics, the sparse one (SPLADE/BM25-like) exact lexical overlap, and together they make hybrid retrieval more robust.

Indexing. The vectors, with their payload, are upserted into the vector store. Two reference options, both current in 2026:

pgvector (a PostgreSQL extension). Its great virtue is living inside Postgres: ACID transactions, joins with the relational metadata, a single database to operate. Version 0.8 added halfvec (half-precision vectors, 2× less storage) and 0.9 (early 2026) added support for sparse vectors and speed improvements. Its known limit: it has no native int8 quantisation, so high-dimensional embeddings consume RAM linearly (Encore, pgvector vs Qdrant 2026; Katz, Scalar and binary quantization for pgvector).
Qdrant (a dedicated vector engine). It supports int8 scalar quantisation (float32 → int8, 4× less memory) and product quantisation, native sparse vectors and RRF fusion for hybrid search. It is more efficient in memory and in quantisation; the cost is operating one more system alongside Postgres (Markaicode, pgvector vs Qdrant 2026).

The practical rule: pgvector if you already have Postgres and the corpus fits comfortably in RAM (a single database to operate and back up); Qdrant if memory efficiency and int8 quantisation are critical because of the corpus size. Synchronisation between the relational truth (Postgres) and the index (Qdrant) when both are used is detailed in PostgreSQL + Qdrant in microservices.

A cross-cutting stage — Incremental ingestion and idempotency

A living corpus changes: documents are added, edited, deleted. Re-indexing everything each night is expensive and causes windows of unavailability. The alternative is incremental ingestion with two pillars:

Idempotency by doc-id + hash. Each chunk is identified deterministically ({doc_id}_{chunk_index}) and each document carries a content hash. On reprocessing, if the hash has not changed, nothing is re-embedded: the compute is saved. If it changed, the old chunks for that doc_id are deleted and the new ones are upserted. An upsert with a deterministic id is idempotent: reprocessing an event twice does not create duplicates.
CDC (Change Data Capture). Instead of polling, Debezium reads the PostgreSQL WAL and propagates additions, edits and deletions to the index in near real time. A deletion in Postgres triggers the deletion of that document’s chunks in the vector store, avoiding the “ghost documents” that contaminate retrieval. The deep dive is in PostgreSQL + Qdrant in microservices and in the one on Debezium and CDC.

The arithmetic: sizing the corpus and the index

Two calculations you need to be able to do before provisioning anything.

Corpus sizing and ingestion time

Suppose a corporate corpus of $N_{docs} = 50{,}000$ documents, averaging 20 pages and 500 useful tokens per page after cleaning. The total tokens in the corpus:

$$T_{corpus} = N_{docs} \times \text{pages} \times \text{tokens/page} = 50{,}000 \times 20 \times 500 = 5 \times 10^{8} \text{ tokens}$$

Five hundred million tokens. Applying the overlap factor from the chunking example (~1.23×, the 23% extra chunks from the 20% overlap) and an effective size of 512 tokens/chunk, the number of chunks is:

$$N_{chunks} \approx \frac{T_{corpus}}{\text{step}} = \frac{5 \times 10^{8}}{410} \approx 1.22 \times 10^{6} \text{ chunks}$$

That is, ~1.22 million chunks to embed. At a conservative CPU embedding throughput of 3,000 tok/s per Xeon server in int8, the same figure we use in the RAG-on-CPU piece, the ingestion time for the first full corpus on one box is:

$$t_{ingesta} = \frac{T_{corpus}}{\text{throughput}} = \frac{5 \times 10^{8}}{3{,}000} \approx 1.67 \times 10^{5} \text{ s} \approx 46 \text{ hours}$$

46 hours on a single box sounds bad, but ingestion is embarrassingly parallel: the corpus is shared out. With 8 CPU servers it drops to ~6 hours, comfortably within a weekend window for the initial load; and subsequent incremental ingestions (only what changed) take minutes. Layout-aware parsing adds its own cost, since Docling with a VLM is slower than PyMuPDF, but it is also batch and parallelises the same way.

Vector index size

Each vector has dimension $d = 1024$ (that of bge-m3). In float32 (4 bytes/dimension), each vector takes:

$$\text{bytes}_{fp32} = d \times 4 = 1024 \times 4 = 4096 \text{ B} = 4 \text{ KB}$$

For the 1.22 M chunks, the dense vectors alone in fp32:

$$\text{size}_{fp32} = N_{chunks} \times d \times 4 = 1.22 \times 10^{6} \times 4096 \text{ B} \approx 5.0 \text{ GB}$$

In int8 (1 byte/dimension), each vector takes 1 KB and the total falls 4×:

$$\text{size}_{int8} = N_{chunks} \times d \times 1 = 1.22 \times 10^{6} \times 1024 \text{ B} \approx 1.25 \text{ GB}$$

To this you must add the HNSW index (~1.2× the size of the vectors for $m=16$) and the payload (metadata + chunk text, ~500 B/chunk → ~0.6 GB). In round numbers:

Configuration	Vectors	HNSW (~1.2×)	Payload	Total
fp32	~5.0 GB	~6.0 GB	~0.6 GB	~11.6 GB
int8	~1.25 GB	~1.5 GB	~0.6 GB	~3.4 GB

The reading: a corpus of 50,000 documents fits in the RAM of a single node even in fp32, and in int8 (Qdrant) it fits with room to spare, which keeps search latency in single-digit milliseconds. int8 quantisation is almost always the balance point, saving 4× with a recall loss typically below 1%. (These numbers are order-of-magnitude, with the constants and assumptions stated; they are for sizing, not for pinning down an invoice.)

Applied to the generic 4×H100 cluster

Let us bring this down to the series’ cluster: 4×H100 SXM 80 GB plus a generic CPU fleet (Xeon with AMX, NUCs). The correct split for ingestion is almost all CPU, with the GPU as an occasional exception:

Cleaning, dedup, chunking, metadata, ingestion embedding and indexing → the CPU fleet. All of this is batch work, throughput-bound, with no latency SLA. It is exactly the “data plane” the RAG on CPU piece talks about: no H100 should spend a cycle chunking documents or building an HNSW index (which was always CPU by design). The nightly incremental re-indexing of a corpus that changes on the scale of hours is the textbook case of “CPU work in no hurry”.
The GPU only comes in at two points. First, in VLM parsing: if the corpus has scans, dense tables or forms and you choose Granite-Docling-258M or another vision-language model, that parsing can be accelerated on GPU, although at 258M parameters it is light and, at moderate volumes, runs on CPU without drama. Second, in the large embedder: if retrieval quality demands a 7B embedder (gte-Qwen2, NV-Embed) instead of bge-m3 (568M), that embedder is once again an LLM-class model and lives where the 7B models live, on the GPU.
The 4×H100 are reserved for generation. As throughout the series, the expensive and scarce silicon is kept for what is latency-bound, the LLM producing the answer, and at most for the peaks of VLM parsing or 7B embedding the CPU cannot absorb. To give a sense of the ceiling: a 4×H100 node serving bge-m3 through TEI runs at around ~2,000 chunks/s, against the thousands of tok/s of a Xeon in int8; but using the H100s for daily ingestion means spending the resource the whole organisation fights over on a job the CPU fleet does overnight without anyone missing it.

The sentence that sums up the split: ingestion is the archive’s cataloguing chain, and almost all of it is done by patient, cheap staff (CPU); the star writer (GPU) is only disturbed when there is a page no classic OCR can decipher.

Closing: quality is decided upstream

The recurring error in mediocre RAG is not in the reranker or the prompt: it is in an ingestion that parsed a table badly, did not weed out the duplicates, or chunked at a size that destroys context. Garbage-in, garbage-out: no downstream component fixes what ingestion spoiled upstream. Investing in the cataloguing chain, with layout-aware parsing where it is needed, real dedup, measured chunking, complete metadata, is what moves the needle on system quality the most, and almost all of it fits on the CPU fleet. The GPU, like the star writer, should only touch the corpus when its brain is genuinely required.

References

IBM — Granite-Docling: End-to-end document understanding (Granite-Docling-258M, Apache 2.0, DocTags, enero 2026). https://www.ibm.com/new/announcements/granite-docling-end-to-end-document-conversion
ibm-granite — granite-docling-258M model card (VLM ~258M, Granite 3 + SigLIP2). https://huggingface.co/ibm-granite/granite-docling-258M
Docling project — Docling: Get your documents ready for gen AI (layout, tablas, multi-formato). https://github.com/docling-project/docling
Unstructured — PDF Parsing Strategies for RAG (fast / hi_res / VLM / auto; chunking by_title). https://unstructured.io/blog/mastering-pdf-transformation-strategies-with-unstructured-part-2 · https://docs.unstructured.io/
Omdena — Document Parsing for RAG: A Complete Guide for 2026 (PyMuPDF, TOC, pipeline). https://www.omdena.com/blog/document-parsing-for-rag
Firecrawl — Best Chunking Strategies for RAG (and LLMs) in 2026 (benchmark Vecta feb-2026: recursive 512 → 69%, semántico 54%; context cliff ~2.500 tok). https://www.firecrawl.dev/blog/best-chunking-strategies-rag
Günther, M., et al. — Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models. arXiv 2409.04701. https://arxiv.org/abs/2409.04701 · Jina AI: https://jina.ai/news/late-chunking-in-long-context-embedding-models/
Brenndoerfer, M. — MinHash: Jaccard Similarity, LSH, and Near-Duplicate Detection. https://mbrenndoerfer.com/writing/minhash-algorithm-jaccard-similarity-lsh-deduplication
Zilliz — Data Deduplication at Trillion Scale (MinHash LSH dominante en limpieza de corpus). https://zilliz.com/blog/data-deduplication-at-trillion-scale-solve-the-biggest-bottleneck-of-llm-training
Ragnarök / TREC RAG 2024 — near-duplicados en MS MARCO V2 degradan recuperación y diversidad. https://arxiv.org/pdf/2406.16828
Encore — pgvector vs Qdrant in 2026 (halfvec, sparse, límites de cuantización de pgvector). https://encore.dev/articles/pgvector-vs-qdrant
Markaicode — pgvector vs Qdrant (2026 Benchmarks) (Qdrant int8 scalar quantization 4×). https://markaicode.com/vs/pgvector-vs-qdrant/
Katz, J. — Scalar and binary quantization for pgvector. https://jkatz05.com/post/postgres/pgvector-scalar-binary-quantization/
Hugging Face — Text Embeddings Inference (TEI), backends CPU/GPU, endpoints OpenAI-compatibles. https://github.com/huggingface/text-embeddings-inference

Taking RAG to the CPU: separating the data plane from the generation plane

Thu, 11 Jun 2026 03:20:00 +0000

Third piece in an operational series about squeezing a generic on-premise LLM cluster of 4×H100 SXM 80 GB. Its siblings: sharing one GPU between workloads (time-slicing, MPS, MIG) and serving several models on one GPU (swap + sleep) attack the split inside the GPU. This one attacks the split outside: which parts of RAG never need to touch the GPU at all. The closing piece of the series, the end-to-end sovereign assistant (fourth instalment, in preparation), assembles the complete system where these pieces fit together.

TL;DR

A RAG system is not one thing, it is three phases with opposing compute profiles, and putting them all on the GPU “because it is AI” is an allocation mistake. (1) Build/ingest, embedding the corpus and building the index, is batch, throughput-bound work with no latency SLA: its natural home is the CPU. (2) Query-time retrieval, embedding the query, HNSW search, RRF fusion, lightweight rerank, is mostly CPU, with caveats only for heavy reranking; vector search always ran on CPU, even in stacks sold as “GPU”. (3) Generation, the LLM producing the answer, is latency-bound, and there the GPU is irreplaceable: a 7B on CPU gives a time to first token of seconds, unacceptable for chat. The technical key to why (1) and (2) fit on CPU: the embedder is not an LLM. bge-m3 is around 568M parameters (an XLM-RoBERTa encoder), not 7B+; in int8 it occupies about 580 MB and activates fast integer compute paths (Intel AVX-512 + VNNI + AMX on 4th-gen Xeon and later; NEON SDOT/UDOT on ARM). The runtimes are ready: TEI with a CPU backend (same OpenAI API /v1/embeddings and /rerank), Qdrant’s fastembed (ONNX-CPU), bge-m3 in ONNX int8 with its three heads (dense/sparse/ColBERT). The Intel + Hugging Face blog post with Optimum Intel and fastRAG reports up to around 10× on indexing for BGE-large int8 on a 4th-gen Xeon (their benchmark figure, encoding-only; I quote it and qualify it below). The operational conclusion: separate the data plane (CPU) from the generation plane (GPU). In the 4×H100 cluster, no H100 should be spent re-indexing a corpus that changes once a day. That goes to the generic CPU fleet (Xeon AMX, NUCs), and the H100s are reserved for generation and, at most, for rerank peaks or large 7B embedders. What does not go down to CPU: interactive generation, massive reranking at high QPS, re-indexing with a strict real-time SLA, and 7B embedders (gte-Qwen2, NV-Embed).

The analogy: the library and the librarian

Picture a serious research library. There are three distinct jobs, done by different people, on different clocks.

The first is cataloguing. Boxes of new books arrive; someone opens them, classifies them, assigns a shelf mark, indexes them in the catalogue and puts them on the right shelf. It is patient background work, done at night or between other tasks. Nobody is standing over you with a stopwatch waiting for today’s batch to be catalogued: what matters is that tomorrow it is done and done well. It is pure throughput: how many books you catalogue per hour, not how long you take on one specific book. This is ingestion.

The second is handling an enquiry at the desk. A reader turns up and asks about a topic. The librarian goes to the catalogue, which already exists, finds half a dozen relevant shelf marks, fetches them and puts the books on the desk. It is fast, light, and consists of searching an index that already exists, not building it. This is retrieval.

The third is writing a reasoned report from those books. The reader, or an expert you commission, reads the six books, compares them, synthesises, and writes an argued answer with citations. This is slow, demands a very well-trained mind, and the reader is waiting: here there is a human stopwatch. This is generation, the LLM.

The moral is about staff allocation. You do not put your star writer, expensive, scarce, with a queue of people waiting for reports, to catalogue boxes of books at dawn. Cataloguing is done by a large, cheap team that works through the night without rushing. The star writer touches only what genuinely needs their mind: writing. In our system, the star writer is the H100, and cataloguing at dawn is corpus ingestion. Spending the H100 on re-indexing is exactly the error of putting the writer to work labelling boxes.

The rest of the post is, essentially, which parts of the library work the cheap CPU team can do (almost all of them) and which is irreducibly the GPU writer’s job (only the last one).

The three phases and their compute profiles

The confusion that GPU over-provisioning feeds on is treating “RAG” as a monolithic block that “uses AI, therefore goes on the GPU”. No. RAG is a data pipeline with a generative model plugged in at the end. The correct architectural boundary does not separate “what uses models” from “what does not”, because both sides use models, but throughput-bound from latency-bound, which is the same as separating the data plane from the generation plane.

Why ingestion fits on CPU: the embedder is not an LLM

The whole argument rests on a size asymmetry that gets overlooked. People hear “embeddings” and “generation” and put them in the same bag of “big models that need a GPU”. But the embedding encoder and the generative LLM are two orders of magnitude apart in parameters.

bge-m3, the reference multilingual embedder, is an XLM-RoBERTa of around 568M parameters (model card, paper arXiv:2402.03216). Its sibling reranker, bge-reranker-v2-m3, is built on the same base and lands at roughly the same 568M parameters (model card). Compare that with an entry-level generative LLM: a Llama 3.1 8B has around 14× more parameters, and the big production models sit at 70B+. A 568M encoder is, in compute budget terms, a different animal.

Two structural differences make that encoder comfortable on CPU:

It is an encoder, not an autoregressive decoder. It processes the whole sequence in a single forward pass and emits the vector. There is no token-by-token decode, no growing KV cache, none of the memory-bound generation phase that kills the CPU. It is a dense matrix pass and that is it.
It quantises to int8 with almost no loss. In int8, bge-m3 occupies on the order of 580 MB and, above all, activates the integer compute paths that a modern CPU executes quickly: matrix instructions such as Intel AMX (Advanced Matrix Extensions, 4th-generation Xeon and later), AVX-512 with VNNI (Vector Neural Network Instructions) on earlier Xeons, and NEON SDOT/UDOT on ARM. The quality loss from going FP32 to int8 on these models usually stays below 1% of retrieval recall, practically invisible (Intel + Hugging Face, CPU Optimized Embeddings).

Let us quantify the int8 size. For $P = 568 \times 10^6$ parameters at 1 byte each:

$$\text{size}_{\text{int8}} \approx 568 \times 10^6 \text{ params} \times 1 \text{ byte/param} \approx 568 \text{ MB}$$

In other words, the model fits in the cache and RAM of any server or NUC without blinking, and the bottleneck is integer compute, exactly what AMX/VNNI accelerate. There is nothing in this profile that asks for a GPU.

Runtimes that already do this effortlessly

Nothing needs inventing. The CPU ecosystem for the data plane is mature:

Text Embeddings Inference (TEI) from Hugging Face: a Rust server with CPU backends via ONNX Runtime (recommended) or Intel MKL, and OpenAI-compatible endpoints (/v1/embeddings) plus /rerank (TEI repo). That is, the data plane on CPU exposes exactly the same HTTP contract as a GPU server; the rest of the system never learns which silicon is behind it.
fastembed from Qdrant: a lightweight library that loads embedders on ONNX-CPU and generates dense, sparse and ColBERT vectors (fastembed repo). Designed from the start to run without a GPU.
bge-m3 in ONNX int8 with its three heads (dense / sparse-lexical / ColBERT multivector) exported and quantised, ready for ONNX Runtime CPU.

The Intel and Hugging Face figure that anchors viability: in their benchmark with Optimum Intel + fastRAG on a 4th-generation Xeon (8480+, 56 cores, 1 socket), the int8 variant of BGE-large reaches up to around 10× indexing throughput against FP32 (HF blog, Haystack/deepset). The small print needs reading, so I read it: that ~10× is encoding-only (tokenisation excluded), at sequence length 256, comparing int8 against FP32 on the same CPU. It is not “CPU 10× faster than GPU”, it is “int8 10× faster than FP32 on CPU”. It is still the relevant figure: it tells you that with quantisation the CPU goes from unworkable to perfectly useful for batch ingestion.

Viability table: CPU for each component?

This is the operational table. The column that matters is the caveat, because a bare “yes” or “no” lies.

Component	CPU viable?	Caveat
Chunking (slicing the corpus)	Yes, always	It is regex, parsing and windows; it never had anything to do with the GPU.
Ingest embedding `bge-m3` dense	Yes, its best case	Overnight batch, int8 + AMX/VNNI. This is exactly what the CPU shines at.
Sparse / SPLADE / BM25 head	Yes, CPU-native	Lexical work is pure inverted index; the GPU adds nothing here.
Building the HNSW index (Qdrant, pgvector)	Yes, always CPU	The HNSW graph build is CPU by design in these engines.
Query embedding (online)	Yes	A single short text; tens of ms on CPU, more than enough for chat.
Dense + sparse search + RRF	Yes	Vector search always ran on CPU, even in “GPU” stacks. RRF is sorting lists.
Cross-encoder reranker `bge-reranker-v2-m3` top-20/50	Yes, with care	A cross-encoder evaluates $k$ query-doc pairs: cost $\propto k$. Over 20-50 candidates it works; over hundreds at high QPS, no.
ColBERT late interaction	Marginal on CPU	The token-by-token matrix product of late interaction is heavy; viable at low volumes, suffers with QPS.
LLM generation	No, in practice	A 7B on CPU gives TTFT in seconds. Interactive latency = GPU.

Two rows deserve underlining because they dismantle myths.

“Vector search needs a GPU.” False from the start. The HNSW index, the navigable small-world graph used by Qdrant, pgvector with vector/halfvec, Milvus in CPU mode and almost everything else, was always built and traversed on CPU. Even the stacks advertised as “GPU-accelerated RAG” do the embedding on GPU but the ANN search stays on CPU in the vast majority of deployments; the GPU variants of the index (CAGRA and similar) are the expensive exception, not the norm, and are justified only with billions of vectors and extreme QPS. For a corporate corpus of millions of chunks, HNSW on CPU resolves in single-digit milliseconds.

“The reranker is a model, therefore GPU.” The bge-reranker-v2-m3 reranker is a cross-encoder of around 568M: it runs on CPU. The caveat is the number of pairs. A cross-encoder does not produce a reusable vector; it evaluates the (query, document) pair jointly, so its cost grows linearly with the number of candidates $k$:

$$\text{cost}_{\text{rerank}} \propto k \times \text{forward}(\text{query} + \text{doc})$$

Reranking the top-20 or top-50 coming out of hybrid retrieval is perfectly affordable on CPU. Reranking hundreds of candidates at high QPS is not: there the linear cost explodes and the GPU wins. The rule of thumb: broad cheap recall in the retriever, precision rerank over few candidates. (The detail of hybrid retrieval and reranking is in the fundamentals piece linked below.)

The numbers, with honest methodology

Here comes the part where a lot of people lie by omission. I am going to give throughput ranges, but they are literature ranges and orders of magnitude, not my own measurements on this hardware. Take them as such: the right decision does not depend on nailing the number, it depends on understanding the allocation.

For bge-m3 dense, sequence ≈256 tokens, embedding throughput moves roughly like this:

Platform	Dense throughput (order of magnitude)	Reading
High-end GPU (5090 fp16, TEI)	~12k tok/s+ (indicative)	The ceiling; expensive and scarce.
Large server CPU (Xeon ~56 cores, int8 ONNX)	low band of thousands tok/s	~1/5–1/10 of the GPU, but horizontally scalable and cheap.
Edge CPU / NUC (4-8 cores, int8)	tens to low hundreds tok/s	Enough for overnight ingestion of a local corpus.

The temptation is to read the second row as “CPU is 5-10× slower, discarded”. That is the wrong reading for ingestion. For batch work with no SLA, what rules is not absolute tok/s but throughput per euro and throughput per watt, and there the arithmetic changes sign.

Take a numerical example of allocation. Suppose a corpus of 2 million chunks of around 256 tokens that has to be re-indexed once a day (the corpus changes, the embeddings have to be redone). That is:

$$2 \times 10^6 \text{ chunks} \times 256 \text{ tok/chunk} \approx 5.1 \times 10^8 \text{ tokens}$$

At a conservative CPU throughput of, say, 3,000 tok/s per int8 Xeon server:

$$t_{\text{ingest}} \approx \frac{5.1 \times 10^8 \text{ tok}}{3000 \text{ tok/s}} \approx 1.7 \times 10^5 \text{ s} \approx 47 \text{ hours on a single server}$$

47 hours on one box sounds bad until you remember two things. First, this is embarrassingly parallel: the corpus is sliced and distributed; with 8 CPU servers it drops to around 6 hours, with 16 to around 3 hours, comfortably inside the overnight window. Second, and more importantly: that same work on the GPU blocks the GPU. If the H100 does 12k tok/s, it takes around 12 hours, but they are 12 hours of the H100, the resource the whole organisation fights over in order to generate. Spending the scarce, expensive resource on re-indexing a corpus that changes once a day is bad allocation, even if it is “faster”: you are optimising the wrong tok/s.

The mental rule: for ingestion, optimise throughput/€ and throughput/W; absolute tok/s belongs to the generation plane, where the human stopwatch really is running.

Decision tree: CPU or GPU for this piece?

Reference architecture (a): CPU-only

The first case is a node with no GPU: a NUC, an office Xeon, a sovereign edge server in a branch or in an air-gapped environment. The whole data plane lives there; generation is delegated to a remote GPU endpoint or done in batch with an SLM when latency is not pressing.

The stack:

TEI-CPU serving bge-m3 int8 with dense + sparse (same OpenAI contract /v1/embeddings, plus /rerank for the reranker).
Qdrant with a dense HNSW index plus sparse vectors, native RRF fusion.
Reranker bge-reranker-v2-m3 over the top-k (via TEI’s /rerank).
Gateway that orchestrates and, for generation, calls an external endpoint.

# docker-compose: full RAG data plane on CPU (no GPU)
services:
 tei-embed:
 image: ghcr.io/huggingface/text-embeddings-inference:cpu-latest
 command: ["--model-id", "BAAI/bge-m3", "--pooling", "cls", "--dtype", "int8"]
 ports: ["8081:80"]
 # ONNX/MKL backend: exploits AVX-512+VNNI / AMX if the Xeon supports it

 tei-rerank:
 image: ghcr.io/huggingface/text-embeddings-inference:cpu-latest
 command: ["--model-id", "BAAI/bge-reranker-v2-m3", "--dtype", "int8"]
 ports: ["8082:80"]
 # exposes /rerank — invoked ONLY over top-20/50, never over hundreds

 qdrant:
 image: qdrant/qdrant:latest
 ports: ["6333:6333"]
 volumes: ["./qdrant_storage:/qdrant/storage"]
 # HNSW dense + sparse vectors + RRF, all CPU

Hybrid search with RRF fusion in Qdrant (dense + sparse in a single query):

from qdrant_client import QdrantClient, models

client = QdrantClient(url="http://qdrant:6333")

# query embedding: dense and sparse from TEI-CPU (HTTP wiring omitted)
hits = client.query_points(
 collection_name="corpus",
 prefetch=[
 models.Prefetch(query=dense_vec, using="dense", limit=50),
 models.Prefetch(query=sparse_vec, using="sparse", limit=50),
 ],
 query=models.FusionQuery(fusion=models.Fusion.RRF), # native RRF
 limit=20,
).points
# -> then: POST /rerank (TEI) over these 20, keep the top-5
# -> then: the gateway sends query + top-5 to the GENERATION endpoint (GPU)

Generation, on this CPU-only node, leaves the node: the gateway builds the augmented prompt and sends it to a vLLM endpoint on the GPU cluster (or, if there is no interactive SLA, to an SLM on CPU in batch mode, accepting a TTFT of seconds). The entire data plane, everything above, runs without a single GPU.

Reference architecture (b): the recommended hybrid

This is the one I recommend for the general case with a GPU cluster available: data plane on CPU, generation plane on GPU, connected by OpenAI-compatible HTTP contracts so that each side is replaceable.

Generation server, minimal, on GPU:

# vLLM on the GPU cluster — generation ONLY
services:
 vllm-gen:
 image: vllm/vllm-openai:latest
 command: >
 --model meta-llama/Llama-3.1-8B-Instruct
 --dtype bfloat16 --max-model-len 8192
 --gpu-memory-utilization 0.85
 # exposes /v1/chat/completions — the gateway sends it query + the top-5 already retrieved
 deploy:
 resources:
 reservations:
 devices: [{driver: nvidia, count: 1, capabilities: [gpu]}]

The virtue of the design: because both sides speak the OpenAI contract over HTTP, the CPU data plane and the GPU generation plane scale separately and are replaceable. If tomorrow you want to move the rerank to GPU because QPS went up, you change one URL. If you want to add more CPU ingest nodes, you add them without touching generation. The whole stack is OSS and licence-clean: bge-m3 and bge-reranker-v2-m3 are MIT (bge-m3, reranker), Qdrant is Apache-2.0, TEI and vLLM are OSS.

Applied to the generic 4×H100 cluster

Let us bring this down to the cluster in the series: 4×H100 SXM 80 GB plus a generic CPU fleet (Xeon with AMX, NUCs). The correct allocation:

Build and indexing → CPU fleet. No H100 should spend a cycle re-embedding the corpus. That goes to the Xeon AMX machines (large servers, throughput in the thousands of tok/s in int8) or, for small local corpora, to the NUCs overnight. Overnight re-indexing of a corpus that changes once a day is the textbook case of “unhurried CPU work”.
The H100s → generation. All four cards are reserved for what only they do well: producing tokens at interactive latency. This is what the sibling pieces in the series, sharing a GPU and several models on one GPU, help to squeeze: once ingestion stops competing for the GPU, all the expensive silicon is free to generate and is shared better between models and tenants.
The H100s, at most, → rerank peaks or large embedders. If at some point you need a 7B embedder (gte-Qwen2, NV-Embed) for a domain where bge-m3 does not reach, or a massive rerank at a QPS the CPU cannot absorb, those peaks can indeed visit the GPU. But they are the occasional exception, not the base load.

The auditability angle: ENS / NIS2

There is a compliance argument that rarely gets mentioned and that the CPU/GPU split gives you almost for free.

A CPU-only node with no proprietary driver is easier to audit. There is no closed NVIDIA kernel stack, no CUDA and firmware versions to reconcile with the supply chain, no proprietary driver surface to document for an ENS or a NIS2 assessment. The whole data plane, chunking, embeddings, index, search, runs on OSS software on generic CPU with standard instructions. For a sovereign or classified environment, being able to say “the plane that touches the corpus depends on no proprietary binary” is a real argument, not marketing.

And there is a second auditability angle intrinsic to RAG done well: source traceability. A RAG system that retrieves identifiable chunks and cites them is auditable, since you can reconstruct which document each claim came from, as opposed to context-stuffing or the model’s opaque parametric knowledge, where there is no way to know where a fact came from. That traceability lives in the data plane (what was retrieved, from which source, with what score), precisely the plane we are putting on auditable CPU. The two arguments reinforce each other: the auditable silicon and the auditable chain of evidence are the same plane.

When NOT to take it to CPU

For honesty, and to avoid falling into the mirror image of the hype, the cases where CPU is not the answer:

Generation at interactive latency. The obvious case. A 7B on CPU gives TTFT in seconds: unacceptable for chat. If the user is waiting, generation goes to GPU. With no practical exceptions today.
Massive reranking at high QPS. A cross-encoder or ColBERT over hundreds of candidates, multiplied by many requests per second, saturates the CPU. The cost $\propto k \times \text{QPS}$ crosses the threshold where the GPU pays off. Keep CPU rerank bounded to top-20/50; if you need more breadth at higher QPS, move up to GPU.
Real-time re-indexing with a strict SLA. If the corpus changes continuously and freshness is measured in seconds (not hours), CPU throughput may not fit the window. There, ingest embedding may need a GPU, but note that this is rare: most corporate corpora change at the pace of hours or days, not seconds.
Large embedders (7B). bge-m3 (568M) is comfortable on CPU; a gte-Qwen2 or NV-Embed at 7B is LLM-class again and drags the same cost profile as generation. If your retrieval quality demands a 7B embedder, that embedder lives where 7Bs live: on the GPU.

The sentence that sums it all up: the CPU is the default home of the data plane; the GPU is the justified exception for what is latency-bound and massively online. Start by putting everything on CPU and move up to GPU only what proves it does not fit, not the other way round.

References

Chen, J., et al. BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings. arXiv 2402.03216. https://arxiv.org/abs/2402.03216
BAAI — BGE-M3 model card (568M params, XLM-RoBERTa, 8192 tokens, MIT). https://huggingface.co/BAAI/bge-m3
BAAI — bge-reranker-v2-m3 model card (cross-encoder on top of bge-m3, ~568M). https://huggingface.co/BAAI/bge-reranker-v2-m3
Intel + Hugging Face — CPU Optimized Embeddings with Optimum Intel and fastRAG (~10× indexing for BGE-large int8, 4th-gen Xeon). https://huggingface.co/blog/intel-fast-embedding
deepset / Haystack — CPU-Optimized Embedding Models with fastRAG and Haystack. https://haystack.deepset.ai/blog/cpu-optimized-models-with-fastrag
Hugging Face — Text Embeddings Inference (TEI), CPU ONNX/MKL backends, OpenAI-compatible endpoints. https://github.com/huggingface/text-embeddings-inference
Qdrant — fastembed (ONNX-CPU, dense/sparse/ColBERT) and Hybrid Search with RRF. https://github.com/qdrant/fastembed · https://qdrant.tech/documentation/beginner-tutorials/hybrid-search-fastembed/

PostgreSQL + Qdrant in RAG ingestion: the postman who keeps two worlds in sync

Thu, 04 Jun 2026 07:00:00 +0200

TL;DR — In a production RAG system, PostgreSQL holds the official truth about documents and Qdrant holds the vectors for search. Keeping them in sync is not trivial: if you delete a document from Postgres and do not invalidate its chunks in Qdrant, the system returns answers from ghost documents. There are two patterns to avoid this: the outbox pattern (atomic transaction plus asynchronous worker, at-least-once) and CDC with Debezium (reading the Postgres WAL directly, low latency, higher complexity). This article explains when to use each one, how to orchestrate them as microservices and what numbers to expect with bge-m3 on on-premise hardware.

The postman and the civil registry

Picture a town with two complementary offices.

The first is the civil registry: it keeps the official census. Every time someone is born, dies or moves house, the registry is the first to know. It is slow, structured, transactional. If the registry says someone exists, they exist. If it says they died, they are dead. PostgreSQL is the civil registry of your documents.

The second is the postman’s notebook: a copy optimised for finding any resident in seconds, organised by area, phonetic names and habitual routes. The postman cannot update the registry, but can search at speeds the registry would never reach. Qdrant is the postman’s notebook.

The problem is synchronisation. If the registry records a death but nobody tells the postman, he will keep trying to deliver letters to an address that no longer exists. In RAG, that translates into indexed chunks of documents that have already been deleted, edited or replaced: ghost documents that pollute the results.

How does the registry tell the postman?

Outbox pattern: every time the registry updates its ledger, it notes the change on an outbox sheet. A messenger clerk reads that sheet periodically and updates the postman’s notebook. Guaranteed, asynchronous, fault tolerant.
CDC with Debezium: the postman has a direct phone line to the registry. Every time the clerk writes something new, the phone rings and the postman updates his notebook in near real time.

The consistency gap problem

In a naive RAG architecture, the flow is:

The user uploads a document → it is inserted into Postgres with metadata.
A worker splits it into chunks, generates embeddings and upserts into Qdrant.
Retrieval uses Qdrant to find relevant chunks and Postgres to hydrate metadata.

So far so good. The problem shows up in updates and deletes:

The user edits a document → Postgres updates the row, but the vectors of the old chunks are still in Qdrant. Retrieval returns stale context.
The user deletes a document → Postgres removes the row, but the chunks remain in Qdrant. Retrieval returns chunks from a document that should no longer exist.
The permission system revokes a tenant’s access → Qdrant has no way of knowing unless there is explicit synchronisation.

This is not a theoretical problem. In living corpora (corporate wikis, knowledge bases updated daily), the consistency gap accumulates noise progressively. An internal study on production pipelines shows that without active reconciliation, 3-8% of indexed chunks correspond to documents that no longer exist in the source of truth after 30 days of operation.

The answer is not “reindex everything every night”. With 10M chunks and a non-trivial embedding model, that costs hours of compute and causes unavailability windows. The answer is change propagation with guarantees.

Outbox pattern: the outbox sheet

Mechanism

The outbox pattern solves the problem of “writing to two systems in the same operation” without distributed transactions (which are expensive and fragile).

The idea is simple: PostgreSQL is the single coordinator. When the ingestion microservice processes a document, it performs two writes in the same local transaction:

It inserts or updates the document in the documents table.
It inserts an event into the outbox_events table.

If the transaction fails, both writes are rolled back. If it succeeds, both are committed atomically. There is no inconsistent intermediate state.

-- Relevant tables
CREATE TABLE documents (
 id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
 tenant_id TEXT NOT NULL,
 title TEXT NOT NULL,
 content TEXT NOT NULL,
 checksum TEXT NOT NULL,
 updated_at TIMESTAMPTZ DEFAULT now()
);

CREATE TABLE outbox_events (
 id BIGSERIAL PRIMARY KEY,
 aggregate_id UUID NOT NULL, -- document id
 event_type TEXT NOT NULL, -- 'document.created' | 'document.updated' | 'document.deleted'
 payload JSONB NOT NULL,
 created_at TIMESTAMPTZ DEFAULT now(),
 processed_at TIMESTAMPTZ -- NULL = pending
);

-- Example of an atomic insert
BEGIN;
 INSERT INTO documents (tenant_id, title, content, checksum)
 VALUES ($1, $2, $3, $4)
 RETURNING id INTO _doc_id;

 INSERT INTO outbox_events (aggregate_id, event_type, payload)
 VALUES (_doc_id, 'document.created', jsonb_build_object(
 'tenant_id', $1,
 'title', $2,
 'checksum', $4
 ));
COMMIT;

The outbox worker

A separate process (the outbox worker) polls outbox_events where processed_at IS NULL, processes each event (chunking, embedding, upsert into Qdrant) and marks the row as processed:

UPDATE outbox_events
 SET processed_at = now()
 WHERE id = $1;

Guarantee: at-least-once. If the worker fails between the Qdrant upsert and the UPDATE, the event will be reprocessed. Qdrant tolerates idempotent upserts (same point id = overwrite), so reprocessing does not create duplicates.

Latency: it depends on the polling interval. With polling every 500ms, p50 latency is ~250ms; p99, ~500ms. Acceptable for most RAG cases, where the user does not expect to see a document indexed in under a second.

CDC with Debezium: the direct phone line

Mechanism

Change Data Capture (CDC) reads PostgreSQL’s Write-Ahead Log (WAL) directly. Postgres writes every change to the WAL before applying it to the tables; it is the mechanism it uses for replication and recovery. Debezium subscribes to a logical replication slot and turns those events into structured messages.

-- Enable logical replication in postgresql.conf
-- wal_level = logical

-- Create a replication slot for Debezium
SELECT pg_create_logical_replication_slot('debezium_slot', 'pgoutput');

The full flow:

Postgres WAL → Debezium connector → Kafka/NATS → Indexer consumer → Qdrant

Debezium emits events with the record’s before/after structure:

{
 "op": "d",
 "before": {
 "id": "550e8400-e29b-41d4-a716-446655440000",
 "tenant_id": "acme",
 "checksum": "sha256:abc123"
 },
 "after": null
}

With "op": "d" (delete), the consumer knows it must delete every point in Qdrant whose payload contains that document_id.

# Consumer: delete by payload filter
qdrant_client.delete(
 collection_name="corpus",
 points_selector=FilterSelector(
 filter=Filter(
 must=[
 FieldCondition(
 key="document_id",
 match=MatchValue(value=event["before"]["id"])
 )
 ]
 )
 )
)

Upsides and downsides

CDC removes polling and cuts latency to tens of milliseconds (WAL propagation time plus consumer processing). But it adds operational complexity: you need to manage the replication slot (unconsumed slots retain WAL indefinitely, which can fill the disk), the message broker and the consumer offset state.

Comparison: outbox vs CDC

Criterion	Outbox pattern	CDC with Debezium
Typical latency	250ms – 2s	20ms – 200ms
Delivery guarantee	At-least-once	At-least-once
Operational complexity	Low (Postgres only)	High (Debezium + broker)
WAL retention risk	None	High if the slot stalls
Idempotency required	Yes (in the indexer)	Yes (in the consumer)
Multi-table support	Manual	Automatic (any table)
Backpressure	Natural (polling)	Needs explicit design
When to choose it	Corpus < 100k docs/day, small team	Corpus > 1M docs/day, low latency critical

Rule of thumb: start with outbox. Move to CDC when the change volume goes past ~50k events/hour or when latency measured in seconds becomes unacceptable for the use case (for example, real-time news indexing).

Microservice architecture

The ingestion pipeline is made up of three microservices with cleanly separated responsibilities:

Microservice 1: Ingestor

Responsibilities: receive documents, split them into chunks and request embeddings. It does not write to Qdrant directly.

# ingestor/main.py (simplified)
from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
 chunk_size=512, # tokens, not characters
 chunk_overlap=64,
 length_function=token_count,
)

def ingest_document(doc: Document, db: Session) -> None:
 chunks = splitter.split_text(doc.content)
 with db.begin():
 db.execute(
 "UPDATE documents SET checksum=$1 WHERE id=$2",
 [doc.checksum, doc.id]
 )
 db.execute(
 """INSERT INTO outbox_events (aggregate_id, event_type, payload)
 VALUES ($1, 'document.updated', $2)""",
 [doc.id, {"chunks": chunks, "tenant_id": doc.tenant_id,
 "model": "bge-m3", "model_version": "1.0.0"}]
 )

Microservice 2: Indexer

It reads the outbox, generates embeddings by calling the inference server (vLLM or TEI) and upserts into Qdrant.

# indexer/worker.py
import asyncio
from qdrant_client import QdrantClient
from qdrant_client.models import PointStruct, VectorParams, Distance

qdrant = QdrantClient(host="qdrant-service", port=6333)

async def process_event(event: dict) -> None:
 payload = event["payload"]
 chunks = payload["chunks"]
 doc_id = str(event["aggregate_id"])

 # Batch embedding on TEI (Text Embeddings Inference)
 embeddings = await embed_batch(chunks, model="bge-m3")

 points = [
 PointStruct(
 id=f"{doc_id}_{i}",
 vector=emb,
 payload={
 "document_id": doc_id,
 "tenant_id": payload["tenant_id"],
 "chunk_index": i,
 "text": chunks[i],
 "model_version": payload["model_version"],
 }
 )
 for i, emb in enumerate(embeddings)
 ]

 # If it is an update, delete the previous chunks first
 if event["event_type"] in ("document.updated", "document.deleted"):
 qdrant.delete(
 collection_name="corpus",
 points_selector=filter_by_doc_id(doc_id)
 )

 if event["event_type"] != "document.deleted":
 qdrant.upsert(collection_name="corpus", points=points)

Microservice 3: Reconciler

The reconciler is the safety net. Periodically (every hour, for example) it compares the set of document_id values in Postgres with the set of document_id values in Qdrant. IDs present in Qdrant but absent from Postgres are ghosts: they get deleted.

# reconciler/diff.py
async def reconcile(tenant_id: str) -> int:
 pg_ids = set(await fetch_all_doc_ids(tenant_id))
 qdrant_ids = set(await scroll_all_doc_ids(tenant_id)) # paginated scroll

 orphans = qdrant_ids - pg_ids
 if orphans:
 logger.warning("Orphan chunks for %d documents", len(orphans))
 for doc_id in orphans:
 qdrant.delete("corpus", filter_by_doc_id(doc_id))

 return len(orphans)

Throughput and ingestion maths

Embedding throughput

The bge-m3 model (1024 dimensions, dense + sparse + colbert support) on a node with 4×H100 SXM (320 GB NVLink) running via vLLM or HuggingFace TEI reaches roughly 2,000 chunks/second with batch size = 256 and 512-token sequences.

$$\text{throughput} = 4 \times 500 \text{ chunks/s/GPU} = 2{,}000 \text{ chunks/s}$$

The figure of 500 chunks/s per GPU comes from public TEI benchmarks with bge-m3 on H100 SXM5, batch=256, seq_len=512 ¹.

Total re-ingestion time

For a corpus of 10M chunks:

$$t = \frac{10{,}000{,}000 \text{ chunks}}{2{,}000 \text{ chunks/s}} = 5{,}000 \text{ s} \approx 83 \text{ minutes}$$

That is pure embedding time. Adding Qdrant write latency (~0.5ms per upsert batch of 100 points):

$$t_{\text{qdrant}} = \frac{10{,}000{,}000}{100} \times 0.5\text{ ms} = 50{,}000 \text{ ms} = 50 \text{ s}$$

Estimated total for a full re-ingestion: ~85-90 minutes on a 4×H100 node.

Storage cost in Qdrant

Each bge-m3 vector has 1024 dimensions in float32 (4 bytes):

$$\text{size per vector} = 1{,}024 \times 4 \text{ B} = 4{,}096 \text{ B} = 4 \text{ KB}$$

For 10M chunks (dense vectors only):

$$\text{total vectors} = 10^7 \times 4{,}096 \text{ B} = 40.96 \text{ GB}$$

Adding the JSON payload (estimated at ~500 bytes/chunk):

$$\text{payload} = 10^7 \times 500 \text{ B} = 5 \text{ GB}$$

HNSW index (roughly 1.2× the vector size for $m=16$):

$$\text{HNSW} \approx 40.96 \text{ GB} \times 1.2 = 49.15 \text{ GB}$$

Estimated total on disk: ~95 GB for 10M chunks with dense bge-m3.

With scalar quantization (int8), the vector size shrinks 4×:

$$\text{with int8 quantization} \approx \frac{40.96}{4} + 5 + \frac{49.15}{4} \approx 27.5 \text{ GB}$$

Configuration	Vectors	HNSW	Payload	Total
float32, no quantization	40.96 GB	49.15 GB	5 GB	~95 GB
int8 scalar quantization	10.24 GB	12.29 GB	5 GB	~28 GB
binary quantization	1.28 GB	1.54 GB	5 GB	~8 GB

Binary quantization loses recall precision (~2-5% in NDCG@10), but it lets you host much larger corpora in RAM. For production where recall is critical, int8 is the usual balance point.

Recommended on-premise hardware

For a continuous ingestion pipeline in production:

Embedding node: 4×H100 SXM (320 GB, NVLink), 2× 64-core CPU (EPYC 9654), 1 TB DDR5 RAM, 100 GbE. Runs vLLM or TEI serving bge-m3. Sustained throughput: ~2,000 chunks/s with an asynchronous batch pipeline.

Qdrant node: 32-core CPU, 256 GB RAM (to keep the HNSW index in memory with 10M chunks and no quantization), 2 TB NVMe (Qdrant snapshot and WAL writes). Qdrant recommends that the HNSW index fit in RAM for p99 latency < 5ms.

PostgreSQL node: 16-core CPU, 128 GB RAM, 4 TB NVMe for the WAL (especially relevant if you use CDC with a logical replication slot; the slot retains WAL until Debezium consumes it).

Broker (if CDC): 3-broker Kafka with 500 GB NVMe per node, or NATS JetStream with 3 nodes for more modest loads.

Kubernetes manifests

Indexer Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
 name: rag-indexer
 namespace: rag-pipeline
spec:
 replicas: 3
 selector:
 matchLabels:
 app: rag-indexer
 template:
 metadata:
 labels:
 app: rag-indexer
 spec:
 containers:
 - name: indexer
 image: registry.example.com/rag-indexer:1.0.0
 env:
 - name: POSTGRES_DSN
 valueFrom:
 secretKeyRef:
 name: pg-credentials
 key: dsn
 - name: QDRANT_HOST
 value: "qdrant-service.qdrant.svc.cluster.local"
 - name: EMBEDDING_ENDPOINT
 value: "http://tei-service.embeddings.svc.cluster.local:8080"
 - name: POLL_INTERVAL_MS
 value: "500"
 resources:
 requests:
 memory: "512Mi"
 cpu: "500m"
 limits:
 memory: "2Gi"
 cpu: "2"

Reconciler CronJob

apiVersion: batch/v1
kind: CronJob
metadata:
 name: rag-reconciler
 namespace: rag-pipeline
spec:
 schedule: "0 * * * *" # hourly
 concurrencyPolicy: Forbid
 jobTemplate:
 spec:
 template:
 spec:
 restartPolicy: OnFailure
 containers:
 - name: reconciler
 image: registry.example.com/rag-reconciler:1.0.0
 env:
 - name: POSTGRES_DSN
 valueFrom:
 secretKeyRef:
 name: pg-credentials
 key: dsn
 - name: QDRANT_HOST
 value: "qdrant-service.qdrant.svc.cluster.local"
 - name: SCROLL_PAGE_SIZE
 value: "1000"
 resources:
 requests:
 memory: "256Mi"
 cpu: "250m"
 limits:
 memory: "1Gi"
 cpu: "1"

Production gotchas

Reindexing when the embedding model changes

This is the most painful problem. If you move from bge-m3 to nomic-embed-text-v2, the vectors are incompatible: they live in different embedding spaces and the cosine distances between them are meaningless.

The answer is dual-index aliasing:

Create a new collection in Qdrant: corpus_v2.
Re-embed the whole corpus with the new model and load corpus_v2.
When the new collection is complete and validated (recall test), switch the corpus_prod alias from corpus_v1 to corpus_v2.
Delete corpus_v1 once traffic has migrated.

During the migration the two indexes coexist. The retriever uses the alias, not the collection’s direct name.

Index versioning

Store model_version in the payload of every point in Qdrant. This lets you:

Filter by version during retrieval (useful in model A/B testing).
Have the reconciler detect points with an old version and reprocess them selectively.
Audit: know which model generated each embedding.

# Filter by model version at retrieval time
results = qdrant.search(
 collection_name="corpus",
 query_vector=query_embedding,
 query_filter=Filter(
 must=[
 FieldCondition(key="model_version", match=MatchValue(value="1.0.0")),
 FieldCondition(key="tenant_id", match=MatchValue(value=tenant_id)),
 ]
 ),
 limit=10
)

Namespace per tenant (multi-tenancy)

There are two strategies in Qdrant:

Strategy	Pros	Cons
Collection per tenant	Total isolation, no extra filter	N collections = N HNSW indexes in RAM
Payload filter per tenant	A single collection, less RAM	The filter adds ~10-15% search latency

For fewer than 100 tenants with large corpora (> 1M chunks/tenant), use a collection per tenant. For hundreds or thousands of tenants with small corpora, use a payload filter with tenant_id indexed:

qdrant.create_payload_index(
 collection_name="corpus",
 field_name="tenant_id",
 field_schema=PayloadSchemaType.KEYWORD
)

What we have not covered

Streaming corpus updates with near-real-time CDC: selective chunk invalidation when only one section of a document changes (incremental chunking based on content diff, not full re-chunking).
Multi-tenant corpus isolation with per-chunk ACLs: going beyond the tenant_id filter towards group, role or even individual document permissions applied at retrieval time.
Federated corpus: corpora distributed across cross-border silos where regulation (GDPR, CCPA) prevents centralising the embeddings; federated search patterns without moving data.
Zero-downtime incremental reindexing using dual-index aliasing: the full model migration protocol with rollback, recall regression testing and progressive traffic splitting.

References

HuggingFace Text Embeddings Inference — benchmarks oficiales con modelos de la familia bge en hardware A100/H100. https://github.com/huggingface/text-embeddings-inference ↩︎

Ingestion on lo0 — Tech Blog

End-to-end document ingestion: from PDF to indexed chunk

TL;DR

The analogy: an archive’s cataloguing chain

The six-stage pipeline

Stage 1 — Parsing: layout-aware vs plain text

OCR for scans

Stage 2 — Cleaning, normalising and deduplicating

Deduplication: why it matters

Stage 3 — Chunking

What the 2026 benchmarks say (and why semantic disappoints)

A worked numerical example of chunking

Stage 4 — Enriching with metadata

Stages 5 and 6 — Embedding and indexing

A cross-cutting stage — Incremental ingestion and idempotency

The arithmetic: sizing the corpus and the index

Corpus sizing and ingestion time

Vector index size

Applied to the generic 4×H100 cluster

Closing: quality is decided upstream

See also

References

Taking RAG to the CPU: separating the data plane from the generation plane

TL;DR

The analogy: the library and the librarian

The three phases and their compute profiles

Why ingestion fits on CPU: the embedder is not an LLM

Runtimes that already do this effortlessly

Viability table: CPU for each component?

The numbers, with honest methodology

Decision tree: CPU or GPU for this piece?

Reference architecture (a): CPU-only

Reference architecture (b): the recommended hybrid

Applied to the generic 4×H100 cluster

The auditability angle: ENS / NIS2

When NOT to take it to CPU

See also

References

PostgreSQL + Qdrant in RAG ingestion: the postman who keeps two worlds in sync

The postman and the civil registry

The consistency gap problem

Outbox pattern: the outbox sheet

Mechanism

The outbox worker

CDC with Debezium: the direct phone line

Mechanism

Upsides and downsides

Comparison: outbox vs CDC

Microservice architecture

Microservice 1: Ingestor

Microservice 2: Indexer

Microservice 3: Reconciler

Throughput and ingestion maths

Embedding throughput

Total re-ingestion time

Storage cost in Qdrant

Recommended on-premise hardware

Kubernetes manifests

Indexer Deployment

Reconciler CronJob

Production gotchas

Reindexing when the embedding model changes

Index versioning

Namespace per tenant (multi-tenancy)

What we have not covered

See also

References