Dedup on lo0 — Tech Blog

End-to-end document ingestion: from PDF to indexed chunk

Thu, 11 Jun 2026 09:00:00 +0000

Fourth piece in an operational series about squeezing a generic on-premise LLM cluster of 4×H100 SXM 80 GB. Its siblings in this batch: serving embeddings and rerankers with TEI in production details the inference piece that this ingestion feeds; GitOps for the inference stack with Flux versions and deploys this whole pipeline; and hardening and secrets for the sovereign stack protects the corpus and the credentials this ingestion touches. The final consumer of what we build here, an end-to-end sovereign assistant, is assembled in a later instalment.

TL;DR

A RAG system inherits the quality of its corpus, and the corpus inherits the quality of the ingestion that manufactured it. This is the garbage-in/garbage-out of retrieval: a chunk badly extracted from a table, a page header repeated a thousand times, a scanned PDF from which you only got OCR noise; all of it gets embedded, indexed and reappears as poisoned context in the model’s answer. Document ingestion is not an afternoon’s script, it is a six-stage pipeline with engineering decisions at every one: (1) extract/parse, where almost everything is decided: layout-aware (Docling, which with Granite-Docling-258M preserves tables, formulas and structure, and which according to IBM avoids classic OCR and is up to ~30× faster) against plain text extraction (PyMuPDF), with OCR or a VLM for scans; (2) clean/normalise, removing boilerplate, normalising Unicode, rebuilding broken paragraphs; (3) chunk, and here 2026 has shifted the consensus: a February 2026 benchmark put recursive at 512 tokens in the lead (69% accuracy) ahead of semantic chunking (54%), and late chunking adds global context at no extra storage cost; (4) enrich metadata, with source, page, section, timestamp, ACL/tenant: for filtered retrieval, citation and auditability; (5) embed, through an embeddings server such as TEI; (6) index, in pgvector or Qdrant, with payload, dense + sparse. In between, exact deduplication (hash) and near-duplicate detection (MinHash/LSH or cosine with a threshold), because near-duplicates degrade retrieval recall and diversity. The numbers matter: a corpus of $N_{docs}\times$ pages $\times$ tokens/page gives the total tokens and, at a given CPU throughput, the ingestion time; and $N_{chunks}\times d\times\text{bytes}$ gives the index size, which in int8 falls 4× against fp32. The reading for the 4×H100: ingestion is CPU work, which links to the RAG-on-CPU piece; the GPU only comes in if you parse with a VLM (Granite-Docling) or use a 7B embedder.

The analogy: an archive’s cataloguing chain

Picture the cataloguing department of a large document archive. It is not one person putting papers into boxes; it is a chain of workstations, each with a different craft and its own quality standard. A document goes in raw at one end and comes out the other turned into a record you can find in seconds. If one station does its job badly, the ones downstream inherit the error and amplify it.

The first station is intake and reading. A heterogeneous box arrives: typed reports, crooked photocopies, folded tables, microfiche. An expert archivist really reads each item: distinguishes the body of the text from the margins, rebuilds a table that spans two pages, transcribes by hand what the scanner could not read. A novice archivist, by contrast, photocopies everything flat and hands over a slab of text where the table columns are interleaved into nonsense. That is exactly the difference between layout-aware parsing and plain text extraction, and it is where almost all of the quality is won or lost.

The second station is weeding out. Before filing anything, someone removes the duplicates (three copies of the same memo, two near-identical versions of a report) and cleans off the useless marks: “COPY” stamps, footers repeated on every sheet, coffee stains. If you do not weed, the archive fills up with copies that, when someone searches, return the same document six times and bury everything else. This is deduplication and cleaning.

The third station cuts it into records. A 400-page book is not catalogued as one giant record; it is broken down into manageable entries, by chapter, by section, of a size a reader can take in at a glance. Too big and the record mixes topics; too small and it loses context. This is chunking, and the size of the record is the decision that most shapes retrieval.

The fourth station labels. Each record carries a shelf mark, a date, its provenance, an access level (can anyone see this, or only the legal department?). Without those labels you cannot filter a search or say which document a statement came from. These are the metadata: source, page, section, timestamp, ACL.

The fifth and sixth stations place the record on the indexed shelf: they translate it into the catalogue’s language, a vector, and put it in the right drawer of the filing cabinet, so that a query finds the related records without walking through the whole thing. This is embedding and indexing in the vector store.

The moral runs through the whole post: RAG cannot retrieve better than what the cataloguing chain filed. If the first station chopped up a table badly, no reranker will fix it afterwards. Ingestion is the quality station for the entire system, and almost all of it, like the archive full of patient people working with nobody timing them, is background work that fits on CPU.

The six-stage pipeline

The temptation to treat ingestion as “read the PDF and chop it up” is the source of 80% of mediocre RAG systems. Each stage has a quality standard and a characteristic failure, and failures cascade downstream: if you parse badly, cleaning does not recover what was lost; if you chunk badly, embedding freezes the error into the vector; if you do not label, you will not be able to filter or cite. The rest of the post walks through the six stations and their decisions.

Stage 1 — Parsing: layout-aware vs plain text

Almost everything is won or lost here. A PDF is not text: it is a set of instructions for drawing glyphs on a canvas. “Extracting the text” from a PDF means reconstructing a reading order that the format does not guarantee, and tables, columns and figures break it systematically.

There are two philosophies, and the choice shapes the rest of the pipeline.

Plain text extraction. Tools such as PyMuPDF (fast, robust, no heavy dependencies) read the PDF’s text stream and dump it out. For single-column documents, running prose and no tables, it is perfect: extremely fast, faithful and cheap on CPU. Its limit shows up with structure: a two-column table comes out with the cells interleaved, a double-column document mixes the end of one with the start of the other, and a scanned invoice does not come out at all because there is no text layer. PyMuPDF can also segment by the table of contents (TOC) when the PDF carries one, which helps with chunking by section (Omdena, Document Parsing for RAG: A Complete Guide for 2026).

Layout-aware parsing. Tools such as Docling (an open source project driven by IBM Research) and unstructured.io first understand the layout, identifying titles, paragraphs, tables, figures, lists and formulas, and only then extract the content while respecting that structure. Docling captures table structure (rows, columns, multi-level headers) and, in its 2026 evolution, does so with a VLM: Granite-Docling-258M, released by IBM in January 2026 under Apache 2.0, a compact vision-language model (~258M parameters, Granite 3 backbone + SigLIP2 visual encoder) that converts pages, whether PDF, slides or scans, directly into a structured format called DocTags, preserving tables, code, inline and block mathematics, and the document’s hierarchy (IBM, Granite-Docling: End-to-end document understanding; model card on Hugging Face; Docling repo). IBM states that the VLM route avoids classic OCR and that this “reduces errors and speeds up the solution by up to 30×” against a traditional OCR pipeline. That is an IBM Research figure; I quote it and treat it as indicative, not as an independent benchmark reproduced here.

unstructured.io offers partitioning strategies graded by document complexity: fast (plain text, quick), hi_res (identifies the layout, recommended when classifying tables and elements properly matters), VLM and auto, balancing speed, cost and accuracy (Unstructured, PDF Parsing Strategies for RAG). The practical rule: fast for prose, hi_res or VLM for documents with tables and structure.

OCR for scans

When the document has no text layer (a scan, a photo, a digitised microfiche), you have to recognise the characters. Three routes:

Classic OCR (Tesseract, PaddleOCR, EasyOCR, which is what Docling integrates with when explicit OCR is needed). Mature, CPU-friendly, good with clean text; it struggles with tables, handwriting and complex layouts.
End-to-end VLM (Granite-Docling and similar). The model “looks” at the page and emits structure directly, with no separate OCR stage. Better with complex layouts; this is where the GPU does come in if the VLM is large or the volume is high.
Hybrid: OCR for character transcription, a layout model for the structure.

The honest criterion: for a corpus of native PDFs with text, PyMuPDF or unstructured fast handle it on CPU and cheaply. For a corpus with dense tables, forms or scans, layout-aware Docling/Granite-Docling earns its cost in chunk quality, and it is the only stage of the pipeline where the GPU can be justified.

Document case	Recommended tool	Silicon
Native PDF, single column, prose	PyMuPDF / unstructured `fast`	CPU
PDF with tables, double column, hierarchy	Docling / unstructured `hi_res`	CPU (layout models)
Scan, form, handwriting, complex layout	Granite-Docling (VLM) or OCR+layout	GPU if large VLM / high volume
HTML, DOCX, PPTX	Docling (multi-format) / native parsers	CPU

Stage 2 — Cleaning, normalising and deduplicating

Freshly parsed text comes in dirty. Cleaning means removing what adds nothing and normalising what is represented in a thousand ways:

Boilerplate: headers and footers repeated on every sheet, page numbers, watermarks, HTML navigation menus, cookie banners. If you do not remove them, they get embedded a thousand times over and contaminate both the index and the answers.
Unicode normalisation (NFC/NFKC), whitespace and dashes: the same character represented in several ways breaks exact matching and dirties the embeddings.
Paragraph reconstruction: undoing the hard line breaks the PDF inserted mid-sentence, without merging paragraphs that were meant to stay separate.

Deduplication: why it matters

RAG suffers two ills from duplicates. The exact kind, the same document uploaded three times, inflates the index and makes a search return the same answer repeatedly, wasting the top-k context slots. The near-duplicate kind, two near-identical versions of a report, a document and its draft, is worse: they look different to a hash but say the same thing, and they degrade retrieval recall and diversity. This is not theory: in the MS MARCO V2 collection a substantial overlap of near-duplicates has been documented which, left untreated, degrades retrieval precision and reduces the diversity of retrieved documents (Ragnarök / TREC RAG 2024).

Two levels of dedup, complementary:

Exact (hash). Compute a sha256 of the normalised content of each document (or chunk) and discard the matches. Cost $O(N)$, trivial. It catches byte-for-byte duplicates.
Near-dup (MinHash + LSH, or cosine of embeddings with a threshold). For those that differ slightly but mean the same thing. MinHash compresses each document into a compact signature such that the probability of two signatures matching at a position equals the Jaccard similarity of the original shingle sets; combined with Locality-Sensitive Hashing (LSH) it finds every near-duplicate pair without comparing all against all, reducing a quadratic problem to almost linear (Brenndoerfer, MinHash, Jaccard, LSH). It is the dominant technique in cleaning LLM training corpora (C4 and RefinedWeb use it) and applies just the same to a RAG corpus (Zilliz, Data Deduplication at Trillion Scale). The alternative, cosine of embeddings with a threshold (say > 0.97), catches semantic duplicates that MinHash cannot see (paraphrasing), but it requires you to already have the embeddings and is more expensive.

The practical rule: exact hash always (it is free); MinHash/LSH for large corpora with versions; cosine with a threshold if paraphrasing is a real problem. And deduplicate before embedding: re-embedding a duplicate means spending compute on rubbish you will then have to filter out.

Stage 3 — Chunking

Chunking is the decision that most shapes retrieval, and the one that carries the most myths. The trade-off is threefold: chunk size ↔ retrieval granularity ↔ context cost.

Large chunks: each one holds more context and there is less risk of cutting an idea in half, but the search is less precise (one vector represents too many topics) and, on retrieval, you put more tokens into the LLM’s prompt, which means more cost and more risk of diluting what is relevant.
Small chunks: very granular, precise retrieval, but each chunk loses context (a 43-token fragment may mean nothing outside its section) and you need to retrieve more of them to cover an answer.

The strategies, from least to most sophisticated:

Fixed size + overlap. Cut every $N$ tokens with an overlap of $k$ tokens between consecutive chunks so as not to slice a sentence dead. Simple, predictable, a reasonable baseline. The overlap is the insurance against cutting an idea right at the boundary.
Recursive (LangChain’s RecursiveCharacterTextSplitter). It tries to cut on separators in priority order (paragraph, then sentence, then word) to respect the natural structure as far as possible before falling back to a hard cut. It is the workhorse.
Semantic. Embed sentences and cut where the similarity between consecutive sentences falls below a threshold, grouping by coherence of meaning. It sounds better on paper; in 2026 practice it has disappointed (see below).
Structure/layout-aware (by headings). It exploits the hierarchy the layout-aware parser has already extracted: one chunk per section or subsection. unstructured offers the by_title strategy, which opens a new chunk when a title element appears, avoiding mixing text from different sections (Unstructured docs). If you parsed with Docling/hi_res, this strategy is almost free and is usually the best one for well-structured documents.
Late chunking. The twist of 2024–2026: instead of chunking and then embedding each chunk separately, embed the whole document first (with a long-context encoder) and then apply the chunk boundaries by mean-pooling the token embeddings inside each span. The result: every chunk keeps the document’s global context (a pronoun or a reference that only makes sense given the previous paragraph is encoded in the vector) and all of it with no extra storage cost, because you end up with one vector per chunk just as before (Jina AI, Late Chunking; arXiv:2409.04701).

What the 2026 benchmarks say (and why semantic disappoints)

It is worth being sceptical about fashion. A Vecta benchmark from February 2026 covering 7 strategies over 50 academic papers put recursive at 512 tokens in first place with 69% accuracy, while semantic chunking came in at 54%, partly because it produced tiny fragments, averaging 43 tokens, too small to mean anything (Firecrawl, Best Chunking Strategies for RAG in 2026). A systematic analysis from January 2026 also identified a “context cliff” around 2,500 tokens, where answer quality drops once you feed in contexts that are too long, an extra argument against giant chunks ([ibid.]). The honest reading: recursive at a moderate size with overlap remains the baseline that is hard to beat; late chunking is the improvement with the best cost/benefit ratio when the model supports it; semantic promises more than it delivers.

A worked numerical example of chunking

Take a technical document of 30 pages, with ~500 tokens of useful prose per page after cleaning (tables and figures are chunked separately). That is $30 \times 500 = 15{,}000$ tokens of text. We chunk with recursive at 512 tokens and 20% overlap ($0.20 \times 512 \approx 102$ tokens). The effective “step” between the start of one chunk and the next is:

$$\text{step} = \text{size} - \text{overlap} = 512 - 102 = 410 \text{ tokens}$$

The number of chunks in the document is then, approximately:

$$N_{chunks} \approx \left\lceil \frac{T_{doc} - \text{overlap}}{\text{step}} \right\rceil = \left\lceil \frac{15{,}000 - 102}{410} \right\rceil \approx \lceil 36.3 \rceil = 37 \text{ chunks}$$

Without overlap it would have been $\lceil 15{,}000 / 512 \rceil = 30$ chunks; the 20% overlap costs us 7 extra chunks (~23% more) in exchange for not splitting ideas at the boundaries. That is the concrete price of overlap: more vectors to embed, index and store, in exchange for robustness in retrieval. We will use this factor when sizing the full corpus.

Stage 4 — Enriching with metadata

A chunk with no metadata is a record with no shelf mark: it exists but it is no use. Each chunk gets a payload attached with, at a minimum:

Source and location: document_id, document name/URI, page number, section/heading (which the layout-aware parser already gave you). Essential for citation: being able to say “this comes from document X, page 12, section 3.2” is what separates an auditable RAG from one that hallucinates with no traceability.
Timestamp: when it was ingested and the document’s date. It lets you filter by freshness and detect stale content.
ACL / tenant: who is allowed to see this chunk. It is critical and it is applied as a filter at retrieval time: a user from department A must not retrieve chunks marked for B only. Without this, the RAG system is a data leak waiting to happen.
Embedding model version (model_version): so you know which embedder generated each vector and can migrate without mixing incompatible spaces.

These metadata are not decoration: they enable filtered retrieval (searching only in what the user can see, or only in documents after a given date), citation (reconstructing the origin of every statement) and auditability (knowing what was retrieved, from where and when). All of it lives in the point’s payload in the vector store.

Stages 5 and 6 — Embedding and indexing

The last two stations translate the chunk into a vector and place it on the shelf.

Embedding. The chunks are sent in batches to an embeddings server, typically Hugging Face’s TEI (Text Embeddings Inference), which exposes the OpenAI /v1/embeddings contract and runs on CPU or GPU. It is throughput-bound work with no latency SLA: its natural home is the CPU with a small encoder in int8 (the sibling piece in this batch, serving embeddings and rerankers with TEI, details the how). It is worth emitting dense + sparse at the same time: the dense vector captures semantics, the sparse one (SPLADE/BM25-like) exact lexical overlap, and together they make hybrid retrieval more robust.

Indexing. The vectors, with their payload, are upserted into the vector store. Two reference options, both current in 2026:

pgvector (a PostgreSQL extension). Its great virtue is living inside Postgres: ACID transactions, joins with the relational metadata, a single database to operate. Version 0.8 added halfvec (half-precision vectors, 2× less storage) and 0.9 (early 2026) added support for sparse vectors and speed improvements. Its known limit: it has no native int8 quantisation, so high-dimensional embeddings consume RAM linearly (Encore, pgvector vs Qdrant 2026; Katz, Scalar and binary quantization for pgvector).
Qdrant (a dedicated vector engine). It supports int8 scalar quantisation (float32 → int8, 4× less memory) and product quantisation, native sparse vectors and RRF fusion for hybrid search. It is more efficient in memory and in quantisation; the cost is operating one more system alongside Postgres (Markaicode, pgvector vs Qdrant 2026).

The practical rule: pgvector if you already have Postgres and the corpus fits comfortably in RAM (a single database to operate and back up); Qdrant if memory efficiency and int8 quantisation are critical because of the corpus size. Synchronisation between the relational truth (Postgres) and the index (Qdrant) when both are used is detailed in PostgreSQL + Qdrant in microservices.

A cross-cutting stage — Incremental ingestion and idempotency

A living corpus changes: documents are added, edited, deleted. Re-indexing everything each night is expensive and causes windows of unavailability. The alternative is incremental ingestion with two pillars:

Idempotency by doc-id + hash. Each chunk is identified deterministically ({doc_id}_{chunk_index}) and each document carries a content hash. On reprocessing, if the hash has not changed, nothing is re-embedded: the compute is saved. If it changed, the old chunks for that doc_id are deleted and the new ones are upserted. An upsert with a deterministic id is idempotent: reprocessing an event twice does not create duplicates.
CDC (Change Data Capture). Instead of polling, Debezium reads the PostgreSQL WAL and propagates additions, edits and deletions to the index in near real time. A deletion in Postgres triggers the deletion of that document’s chunks in the vector store, avoiding the “ghost documents” that contaminate retrieval. The deep dive is in PostgreSQL + Qdrant in microservices and in the one on Debezium and CDC.

The arithmetic: sizing the corpus and the index

Two calculations you need to be able to do before provisioning anything.

Corpus sizing and ingestion time

Suppose a corporate corpus of $N_{docs} = 50{,}000$ documents, averaging 20 pages and 500 useful tokens per page after cleaning. The total tokens in the corpus:

$$T_{corpus} = N_{docs} \times \text{pages} \times \text{tokens/page} = 50{,}000 \times 20 \times 500 = 5 \times 10^{8} \text{ tokens}$$

Five hundred million tokens. Applying the overlap factor from the chunking example (~1.23×, the 23% extra chunks from the 20% overlap) and an effective size of 512 tokens/chunk, the number of chunks is:

$$N_{chunks} \approx \frac{T_{corpus}}{\text{step}} = \frac{5 \times 10^{8}}{410} \approx 1.22 \times 10^{6} \text{ chunks}$$

That is, ~1.22 million chunks to embed. At a conservative CPU embedding throughput of 3,000 tok/s per Xeon server in int8, the same figure we use in the RAG-on-CPU piece, the ingestion time for the first full corpus on one box is:

$$t_{ingesta} = \frac{T_{corpus}}{\text{throughput}} = \frac{5 \times 10^{8}}{3{,}000} \approx 1.67 \times 10^{5} \text{ s} \approx 46 \text{ hours}$$

46 hours on a single box sounds bad, but ingestion is embarrassingly parallel: the corpus is shared out. With 8 CPU servers it drops to ~6 hours, comfortably within a weekend window for the initial load; and subsequent incremental ingestions (only what changed) take minutes. Layout-aware parsing adds its own cost, since Docling with a VLM is slower than PyMuPDF, but it is also batch and parallelises the same way.

Vector index size

Each vector has dimension $d = 1024$ (that of bge-m3). In float32 (4 bytes/dimension), each vector takes:

$$\text{bytes}_{fp32} = d \times 4 = 1024 \times 4 = 4096 \text{ B} = 4 \text{ KB}$$

For the 1.22 M chunks, the dense vectors alone in fp32:

$$\text{size}_{fp32} = N_{chunks} \times d \times 4 = 1.22 \times 10^{6} \times 4096 \text{ B} \approx 5.0 \text{ GB}$$

In int8 (1 byte/dimension), each vector takes 1 KB and the total falls 4×:

$$\text{size}_{int8} = N_{chunks} \times d \times 1 = 1.22 \times 10^{6} \times 1024 \text{ B} \approx 1.25 \text{ GB}$$

To this you must add the HNSW index (~1.2× the size of the vectors for $m=16$) and the payload (metadata + chunk text, ~500 B/chunk → ~0.6 GB). In round numbers:

Configuration	Vectors	HNSW (~1.2×)	Payload	Total
fp32	~5.0 GB	~6.0 GB	~0.6 GB	~11.6 GB
int8	~1.25 GB	~1.5 GB	~0.6 GB	~3.4 GB

The reading: a corpus of 50,000 documents fits in the RAM of a single node even in fp32, and in int8 (Qdrant) it fits with room to spare, which keeps search latency in single-digit milliseconds. int8 quantisation is almost always the balance point, saving 4× with a recall loss typically below 1%. (These numbers are order-of-magnitude, with the constants and assumptions stated; they are for sizing, not for pinning down an invoice.)

Applied to the generic 4×H100 cluster

Let us bring this down to the series’ cluster: 4×H100 SXM 80 GB plus a generic CPU fleet (Xeon with AMX, NUCs). The correct split for ingestion is almost all CPU, with the GPU as an occasional exception:

Cleaning, dedup, chunking, metadata, ingestion embedding and indexing → the CPU fleet. All of this is batch work, throughput-bound, with no latency SLA. It is exactly the “data plane” the RAG on CPU piece talks about: no H100 should spend a cycle chunking documents or building an HNSW index (which was always CPU by design). The nightly incremental re-indexing of a corpus that changes on the scale of hours is the textbook case of “CPU work in no hurry”.
The GPU only comes in at two points. First, in VLM parsing: if the corpus has scans, dense tables or forms and you choose Granite-Docling-258M or another vision-language model, that parsing can be accelerated on GPU, although at 258M parameters it is light and, at moderate volumes, runs on CPU without drama. Second, in the large embedder: if retrieval quality demands a 7B embedder (gte-Qwen2, NV-Embed) instead of bge-m3 (568M), that embedder is once again an LLM-class model and lives where the 7B models live, on the GPU.
The 4×H100 are reserved for generation. As throughout the series, the expensive and scarce silicon is kept for what is latency-bound, the LLM producing the answer, and at most for the peaks of VLM parsing or 7B embedding the CPU cannot absorb. To give a sense of the ceiling: a 4×H100 node serving bge-m3 through TEI runs at around ~2,000 chunks/s, against the thousands of tok/s of a Xeon in int8; but using the H100s for daily ingestion means spending the resource the whole organisation fights over on a job the CPU fleet does overnight without anyone missing it.

The sentence that sums up the split: ingestion is the archive’s cataloguing chain, and almost all of it is done by patient, cheap staff (CPU); the star writer (GPU) is only disturbed when there is a page no classic OCR can decipher.

Closing: quality is decided upstream

The recurring error in mediocre RAG is not in the reranker or the prompt: it is in an ingestion that parsed a table badly, did not weed out the duplicates, or chunked at a size that destroys context. Garbage-in, garbage-out: no downstream component fixes what ingestion spoiled upstream. Investing in the cataloguing chain, with layout-aware parsing where it is needed, real dedup, measured chunking, complete metadata, is what moves the needle on system quality the most, and almost all of it fits on the CPU fleet. The GPU, like the star writer, should only touch the corpus when its brain is genuinely required.

References

IBM — Granite-Docling: End-to-end document understanding (Granite-Docling-258M, Apache 2.0, DocTags, enero 2026). https://www.ibm.com/new/announcements/granite-docling-end-to-end-document-conversion
ibm-granite — granite-docling-258M model card (VLM ~258M, Granite 3 + SigLIP2). https://huggingface.co/ibm-granite/granite-docling-258M
Docling project — Docling: Get your documents ready for gen AI (layout, tablas, multi-formato). https://github.com/docling-project/docling
Unstructured — PDF Parsing Strategies for RAG (fast / hi_res / VLM / auto; chunking by_title). https://unstructured.io/blog/mastering-pdf-transformation-strategies-with-unstructured-part-2 · https://docs.unstructured.io/
Omdena — Document Parsing for RAG: A Complete Guide for 2026 (PyMuPDF, TOC, pipeline). https://www.omdena.com/blog/document-parsing-for-rag
Firecrawl — Best Chunking Strategies for RAG (and LLMs) in 2026 (benchmark Vecta feb-2026: recursive 512 → 69%, semántico 54%; context cliff ~2.500 tok). https://www.firecrawl.dev/blog/best-chunking-strategies-rag
Günther, M., et al. — Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models. arXiv 2409.04701. https://arxiv.org/abs/2409.04701 · Jina AI: https://jina.ai/news/late-chunking-in-long-context-embedding-models/
Brenndoerfer, M. — MinHash: Jaccard Similarity, LSH, and Near-Duplicate Detection. https://mbrenndoerfer.com/writing/minhash-algorithm-jaccard-similarity-lsh-deduplication
Zilliz — Data Deduplication at Trillion Scale (MinHash LSH dominante en limpieza de corpus). https://zilliz.com/blog/data-deduplication-at-trillion-scale-solve-the-biggest-bottleneck-of-llm-training
Ragnarök / TREC RAG 2024 — near-duplicados en MS MARCO V2 degradan recuperación y diversidad. https://arxiv.org/pdf/2406.16828
Encore — pgvector vs Qdrant in 2026 (halfvec, sparse, límites de cuantización de pgvector). https://encore.dev/articles/pgvector-vs-qdrant
Markaicode — pgvector vs Qdrant (2026 Benchmarks) (Qdrant int8 scalar quantization 4×). https://markaicode.com/vs/pgvector-vs-qdrant/
Katz, J. — Scalar and binary quantization for pgvector. https://jkatz05.com/post/postgres/pgvector-scalar-binary-quantization/
Hugging Face — Text Embeddings Inference (TEI), backends CPU/GPU, endpoints OpenAI-compatibles. https://github.com/huggingface/text-embeddings-inference

RAG corpus curation: the active librarian who decides what goes in, what goes out and what gets signed

Mon, 25 May 2026 11:00:00 +0200

This is the curation layer inside stage 1 (Data) of the six-stage LLMOps pipeline. It complements the other Data posts: dataset versioning for the four versionable artefacts, PostgreSQL + Qdrant ingestion in microservices for the outbox + CDC pattern, and RAG over Kafka and a datalake for streaming transport. This post is not about moving data: it is about what to do with data before you let a model read it.

TL;DR

A RAG system serving mediocre answers is rarely the retriever’s fault, nor the model’s. The root cause is usually the corpus: three nearly identical versions of the same PDF that make top-k return the same thing three times, an old manual nobody removed that contradicts the current one, a free-text field with customer numbers that the model quotes verbatim, a scanned PDF with dirty OCR that the chunker cut in the middle of a sentence. None of that is fixed by changing the model, the embedder, the reranker or the prompt. It is fixed by curating the corpus. This post takes apart the five operational layers of curation (schema-validated ingest, three-level deduplication, PII anonymisation measured with precision/recall, anti-contamination against the golden eval set, chunk→trace lineage), the minimum maths needed to avoid fooling yourself, the 2026 stack (Presidio, Unstructured, Argilla, LangChain text splitters, OpenLineage, Marquez, Great Expectations), the seven traps that reduce the stage to theatre, and the on-premise hardware to sustain all of it without sending anything sensitive to external APIs.

The analogy: the active librarian

A serious librarian does not accept books by the kilo. When someone offers a new volume:

They look at the spine, the ISBN and the stamp: is it legible? Is it catalogued correctly? Does it belong to a recognised collection? Without valid metadata, it does not get in. This is the schema check.
They check whether they already hold a copy: is it exactly the same book? Is it a later edition of the same one? Is it a translated version of something already on the shelves? If so, they decide explicitly what to do (replace, archive the old one, withdraw both from lending). This is dedup at its three levels.
They flag what is restricted: if the book contains identifiable personal data, some pages cannot be lent as they are, so they must be redacted, anonymised or moved to the reserved section. This is PII anonymisation.
They verify it is not this year’s final exam book: if it is, it stays out of the public collection until the syllabus changes, because if students can consult it the exam stops measuring what it is meant to measure. This is anti-contamination against the golden eval set.
They write it into the register: this book, this edition, this provenance, this date, this person who approved the entry. This is lineage.

If the book passes all five, it joins the collection. If it fails any of them, it goes to an auditable quarantine shelf with the reason for rejection. The difference between a good collection and a mediocre one is not size: it is how much discipline you apply across the five layers, every day, to every new book that arrives.

A RAG corpus is exactly that. The only differences are scale (thousands or millions of documents per month) and the fact that the “readers” are LLMs that cannot tell a duplicate from a reinforced truth, nor a PII value from a synthetic example, nor a contaminated fragment from an authentic one.

The four data artefacts and where the RAG corpus fits

Before going down to the five layers it is worth being clear about which corpus we are curating. The Data stage of the pipeline manages four distinct artefacts, each with its own discipline. The data versioning post lists them; here I reorder them from a curation perspective:

Artefact	Who consumes it	Dominant curation
Training dataset	Tune (fine-tuning the model or an adapter)	aggressive dedup + quality filters + label balancing
RAG corpus	Deploy (retrieval at request time)	the 5 layers in this post
Golden eval set	Eval (promotion gates)	strict hold-out + stratification + maintenance driven by incidents
Retrain enriched dataset	Retrain (closing the loop)	production feedback + human triage

The RAG corpus is the most volatile of the four and the one most exposed to the end user: every answer the system serves literally contains fragments of it. A duplicate in the training dataset degrades learning but ends up buried in the weights; a duplicate in the RAG corpus shows up in today’s answer and tomorrow’s. That justifies the extra discipline that follows.

Layer 1 — Schema-validated ingest

Every piece entering the corpus has to arrive with structured metadata validated against a schema. This is not bureaucracy: it is the only way to make the following layers (dedup, PII, lineage) work without friction.

The canonical pattern is to define a schema in JSON Schema or Pydantic that every document must satisfy:

class CorpusDocument(BaseModel):
 source_system: str # e.g. "confluence", "salesforce", "manual_pdf"
 source_id: str # unique ID in the source system
 version: str # document version (semver or date)
 language: str # ISO 639-1: "es", "en"
 title: str
 body: str
 captured_at: datetime
 captured_by: str # pipeline or human
 sensitivity: Literal["public", "internal", "restricted"]
 schema_version: str # version of the schema itself, not of the document

Any piece that fails this contract is rejected at ingest and never reaches the following layers. Validation is done with Great Expectations (declarative suites), Pandera (more pythonic, integrates with pandas) or Soda (oriented towards continuous data quality). The choice is a matter of style; what matters is:

Validation suites live in code and are versioned with the pipeline, not in a notebook off to one side.
Rejection generates an auditable event (quarantine) with the specific schema failure as its reason, not a log line lost in stdout.
The schema itself is versioned. When it changes, previous documents are reprocessed or explicit backward compatibility is maintained.

The RAG over Kafka post covers the Schema Registry pattern (Confluent Schema Registry or Apicurio) that materialises this in streaming: every message on the topic is validated against the registered schema before propagating downstream. For batch or pull, Great Expectations is the equivalent.

Common trap: leaving the body field free with no further validation. You have to tighten it up with minimum/maximum length (a PDF that yields 12 characters after extraction is almost certainly broken), valid encoding (UTF-8 with no control characters), proportion of alphanumeric characters (dirty OCR returns a soup of symbols). These are simple rules that filter 80% of the noise with no need for AI.

Layer 2 — Three-level deduplication

The most expensive and most silent mistake in a RAG corpus is the duplicate. A document that appears three times in the corpus makes retrieval’s top-k return it three times, wasting two slots and reinforcing a single source. The LLM reads it as if three independent sources agreed, when in fact it is the same thing repeated.

Deduplication is done at three levels, in this order by cost:

Level A — Exact dedup (SHA-256 hash)

Compute the hash of the normalised content (trim, lower-case where applicable, remove redundant whitespace) and compare it against an index of already ingested hashes. On a match, discard or replace. Cost: $O(1)$ per document. It catches literal duplicates (the same PDF uploaded twice, two byte-for-byte copies of the same HTML).

Level B — Near-duplicate (MinHash + LSH)

Nearly identical documents with minor differences (a different header, an updated date, one version in Castilian and another in Galician with minimal changes). The canonical algorithm is MinHash with Locality-Sensitive Hashing (LSH), which approximates Jaccard similarity over shingles of k tokens. For n documents, comparing all against all is $O(n^2)$, which is unworkable for large corpora. LSH cuts the cost to $O(n)$ most-likely buckets.

A typical threshold is Jaccard ≥ 0.80 over shingles of 5 tokens. The standard libraries are datasketch (Python, MIT) or dedup (Python, MIT). A numerical example: for 1 M short documents (300 tokens each), datasketch.MinHashLSH with 128 permutations and threshold 0.8 takes about 2 GB of RAM and processes the full corpus in around 30 minutes on a modern CPU. The fraction of duplicates found in a real enterprise corpus is usually between 5% and 25%; removing them cuts storage and improves retrieval quality at the same time.

Level C — Semantic dedup (cosine over embeddings)

Documents saying the same thing in different words, such as paraphrases, translations or rewritten versions, are not caught by MinHash. This is where semantic similarity comes in: compute the embedding of each document and compare the cosine between pairs.

The problem is quadratic cost: for n documents, computing every pair is $O(n^2)$. For n = 1 M and 768-dimensional embeddings (a typical model such as BAAI/bge-base-en-v1.5), that is 5×10^11 dot products, which is unworkable. The solution is the same idea as LSH but over dense vectors: HNSW (Hierarchical Navigable Small World) or IVF (Inverted File) to build an approximate search index. For each new document you run a k-NN query against the index and examine only the k nearest neighbours.

A sensible threshold for treating something as a semantic duplicate: cosine ≥ 0.95. Below 0.95 the documents are related but distinct; above it, they are almost always the same information rewritten. The exact threshold is calibrated by observing precision/recall over a human-annotated sample; 100 pairs confirmed by a reviewer is reasonable for setting it.

A numerical example: with qdrant or pgvector as the HNSW index and k=10 neighbours per query, deduplicating 1 M documents against the existing corpus takes on the order of 2-4 hours on an RTX 4090 (including embedding computation). If the embedder is self-hosted with vLLM, the cost per token is negligible against compute time.

Policy on what to do with a duplicate

Detection is not enough, you have to decide. Three common policies, in order of complexity:

Drop: discard the more recent one, keep the older one. Simple, with no extra lineage.
Replace: discard the old one, index the new one. More volatility but it reflects the update.
Merge with provenance: mark the new one as a “shadow” of the old one, keep both in lineage but index only one. Better for regulated audit.

The policy has to be explicit and applied uniformly, not an ad-hoc decision per document.

Layer 3 — PII anonymisation with measured precision/recall

This layer is the one that most easily turns into theatre. The typical mistake: install Presidio, run it over the corpus, assume the output is clean. Without measuring the detector’s precision and recall against an annotated golden set, you know nothing.

A PII detector can fail in two ways:

False negative (low recall): it does not detect a national ID written as “12345678-A” because your model was trained on the 12345678A format without a hyphen. The RAG serves personal data unredacted.
False positive (low precision): it redacts a number from a configuration manual thinking it is a phone number. The RAG loses useful information.

Both are problems; regulation (GDPR, ENS, NIS2) penalises the first, and user experience degrades with the second. The acceptable ratio depends on the domain. In medical data, practically zero false negatives is non-negotiable; in internal technical documentation you can tolerate more recall in exchange for less precision.

The standard metric is F1 over an annotated golden set:

\[ \text{precision} = \frac{TP}{TP + FP}, \quad \text{recall} = \frac{TP}{TP + FN}, \quad F_1 = 2 \cdot \frac{\text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}} \]

To build the PII golden set, annotate around 200 documents by hand with every entity marked (national ID, IBAN, email, phone, address, personal name). Then run the detector and compute the metrics per category, not just aggregated, because a global F1 of 0.90 can hide a recall of 0.55 on IBANs.

2026 stack for this layer:

Microsoft Presidio (MIT, Microsoft): the most complete OSS option. Configurable detectors, recognises around 50 entities by default, extensible with your own regex patterns or with fine-tuned NER models.
spaCy NER (MIT, Explosion AI): a base for custom detectors; useful when Presidio does not cover a domain entity.
Llama Guard 4 (Llama Community License, Meta): a safety classifier that also detects PII in one pass, an option when you already have a GPU for inference and prefer a single pass.
DataFog (Apache 2.0): a more recent alternative, specialised in streaming pipelines.

Recommended hybrid pattern: Presidio for rule-based + regex detection (fast, deterministic) → Llama Guard as a second pass over what Presidio did not flag (an ensemble that raises recall without killing throughput). This is measured and reported as aggregate and per-category F1 on every release of the detector.

Common fallacy: trusting that a detector with F1 0.95 “is very good”. If you have 1 M documents and each contains on average 1 PII entity, F1 0.95 means 50,000 mishandled entities (between false positives and negatives). With sensitive data you have to design so that false negatives go to human quarantine, not to the public corpus.

Layer 4 — Anti-contamination against the golden eval set

If the RAG corpus contains fragments of the golden eval set, Eval metrics measure memorisation. The model returns the exact answer because it has it literally in its context, not because it generalised anything. Deploy then promotes models that shine in the exam and fail in production.

This layer is the easiest to implement and the easiest to forget:

The golden eval set has its own versioned hash.
Before indexing any new document into the RAG corpus, run a token-by-token overlap check (or by shingles, similar to MinHash) against the golden set.
If overlap exceeds a threshold (typically ≥ 30% of 5-token n-grams), the document is not indexed. It stays in quarantine flagged as “contamination risk vs golden_v12”.
A human reviews the rejections. Sometimes they are false positives (a short quotation, a boilerplate sentence). Sometimes they are real contamination that a supplier introduced without realising.

The deeper reason: the RAG corpus and the golden set are enemies by design. The golden set measures how well the system generalises to questions it has not seen. If those questions are in the RAG, the system “sees” them on every query. The metric stops measuring generalisation.

This check is computationally trivial, a hash join over n-grams. The difficulty is keeping it up: every time the golden set changes (monthly or quarterly), the full corpus has to be re-validated against the new golden set. Without that discipline, contamination sneaks in through the back door when someone updates the golden set with real cases that the RAG was already serving.

Layer 5 — End-to-end lineage: from document to trace

The last layer is the one that closes the auditable chain. Every chunk indexed in the vector store carries metadata that lets you answer the forensic question:

“The system generated this answer on 14 March at 16:23. Which exact document did the quoted fragment come from? When did that document enter the corpus? Which version of the embedder processed it? Who approved its ingest?”

Without lineage, that question is unanswerable. With lineage done properly, it is four queries.

The canonical pattern:

Every indexed chunk carries in its metadata: source_system, source_id, document_version, chunk_index, embedder_version, dataset_hash, ingested_at, ingested_by, schema_version.
Every RAG answer in production emits a trace span that includes the retrieved chunk_id values.
The central tracing system (Langfuse, Phoenix or OpenLLMetry) joins chunk_id → chunk metadata → document metadata → corpus dataset_hash → embedder version → and so on.

The tools that standardise this glue are OpenLineage (Apache 2.0, LF AI & Data) and Marquez (Apache 2.0, its server implementation). They define a lineage event schema that interoperates between systems; an ingest job emits an event “produced corpus_v12.3 from source X with embedder bge-base-v1.5”; a retrieval job emits “consumed corpus_v12.3 with query Q produced response R”. The graph is reconstructed automatically.

This layer is the only way to pass real audits under regulations such as the EU AI Act, GDPR or ENS, where traceability of which data went into which answer is a requirement, not an option. Without it, the answer “we do not know which document this came from” is not acceptable, and it is the default answer if lineage is not built from day one.

The maths that matter

Beyond dedup thresholds and PII F1 scores, there are three mathematical pieces that any serious team ends up using.

Chunk size vs retrieval quality. Chunk size affects retrieval quality in a non-monotonic way: chunks that are too small fragment ideas (retrieval returns a piece without context), and chunks that are too large dilute the signal (the embedding mixes several topics and similarity drops). The empirical sweet spot for technical text in 2026 is between 256 and 768 tokens per chunk, with 15-25% overlap between contiguous chunks to preserve continuity.

Numerically, for a corpus of 1 M documents with a mean length of 2,000 tokens, chunking at 512 tokens with overlap 100 gives $\frac{2000}{512 - 100} \approx 5$ chunks per document, a total of roughly 5 M indexed chunks. With 768-dimensional embeddings in float32, that occupies $5 \cdot 10^6 \cdot 768 \cdot 4 \approx 15$ GB of vector memory, manageable in any modern vector store.

PII golden set coverage. To know whether the annotated PII golden set is representative enough, compute the proportion of categories covered: if your golden set of 200 documents has 5 IBAN examples and production has 12,000 IBANs per day, the measured F1 on IBANs is statistical noise. Rule of thumb: a minimum of 30 examples per category for per-category metrics to mean anything.

Cost of re-embedding when rotating the model. Changing the embedder invalidates the whole index. For a corpus of 5 M chunks with a model such as BAAI/bge-base-en-v1.5 (768 dim, around 110 M parameters) served in vLLM on 1× H100, throughput is on the order of 8,000-15,000 chunks/second. Full re-embedding: about 5-10 minutes. For a larger embedder (bge-large, 1024 dim, around 335 M parameters), a factor of 3× worse, around 15-30 minutes. The bottleneck is usually vector store I/O, not GPU compute. The dual-index pattern, keeping the old index serving while the new one is built and doing an atomic swap at the end, avoids downtime and allows rollback.

Applied to typical on-premise hardware

For an on-premise deployment that keeps all curation in-house without sending data to external APIs:

RTX 4090 (24 GB): covers layer 1 (the schema check with Great Expectations is CPU-bound), layer 2 levels A and B (hash + MinHash are CPU-bound), and layer 2 level C semantic dedup with a bge-base-class embedder (8-15k chunks/s, enough for corpora of up to 5-10 M chunks in hours). For Presidio in NER mode (layer 3) it runs comfortably. It is the sensible GPU for the whole curation pipeline on a mid-size corpus.
Generic 4×H100 SXM configuration (320 GB total, NVLink): only needed if the corpus exceeds around 50 M chunks or if you want frequent re-embeddings with large models (bge-large, e5-mistral). In practice, two GPUs serve the embedder at TP=2 with throughput above 50k chunks/s, and the other two go to the PII judge (Llama Guard 4) or to serving the main inference model. Capacity for corpora of hundreds of millions of chunks.

The stubborn arithmetic: with a 4090, corpus curation is an overnight job; with 4×H100, it is minutes. The decision depends on corpus size and on how often you rotate the embedder or the PII rules.

The seven traps that kill this stage

Trap 1 — No validated schema at ingest. Malformed documents reach the chunker, the chunker slices them into nonsense, garbage embeddings enter the index. The RAG answer quotes incoherent text and nobody knows why.

Trap 2 — Dedup only at exact hash level. The corpus fills up with paraphrases and translations of the same document. Retrieval’s top-k returns the same source 3 times. The LLM reads it as three confirmations.

Trap 3 — PII detector without precision/recall measurement. Everyone assumes Presidio “works”. IBANs in non-standard formats slip through. The RAG serves personal data.

Trap 4 — Golden eval set contaminated with the corpus. Eval metrics measure memorisation. Promotions approve models that fail in real production.

Trap 5 — No lineage down to the chunk. The question “where did this quotation come from?” has no answer. Regulatory audit fails. Incidents cannot be investigated.

Trap 6 — Maintenance as a one-off event. The corpus is curated once when the system is initialised, and afterwards everyone assumes it is fine. Six months later the documents are out of date, the new PII rules are not applied retrospectively, dedup has not been re-run after adding new sources. The corpus degrades silently.

Trap 7 — Quarantine without human review. Rejected documents go into a table nobody looks at. False positives pile up, real contamination cases go uninvestigated, the team’s trust in curation erodes and the pressure to “relax the thresholds” begins.

All seven are operational, not technical. Corpus curation does not break because of an algorithm bug: it breaks because discipline slackens. It is the exact equivalent of the kind of degradation that kills Eval suites, and in both cases the symptom is the same: the metrics improve or hold steady while the real experience gets worse.

What we have not covered (upcoming posts)

Vector store versioning proper: an embedding index is not versioned like a raw dataset because it depends on the embedding model. Changing the embedder rewrites the whole index. It is a different animal with its own patterns (index branching, selective reembedding, recall-aware ANN parameters).
Streaming corpus updates with CDC: when the corpus has to be updated in near-real time from an OLTP system. The Postgres + Qdrant ingestion post covers the mechanics; still pending is the pattern for selectively invalidating chunks that depend on deleted rows.
Multi-tenant corpus isolation: how to set up a shared corpus versus one with per-tenant namespaces, with ACLs over individual chunks. Especially relevant for multi-customer RAG under data sovereignty constraints.
Federated corpus: corpora distributed across silos that the system queries without centralising the content. An emerging pattern for companies with several sites and cross-border restrictions.
Reranking-aware curation: how curation discipline changes when there is a reranker (Cohere Rerank, ColBERTv2, BGE-Reranker) reordering the top-k after retrieval. Some duplicates you would tolerate without a reranker become intolerable when the reranker pushes them up the ranking.

References

Presidio: https://microsoft.github.io/presidio/ — official docs, list of supported entities, guide to extending it with custom NER.
OpenLineage: https://openlineage.io/ — event schema spec and libraries per language.
Marquez: https://marquezproject.ai/ — server implementation of OpenLineage.
datasketch (MinHash + LSH): https://ekzhu.com/datasketch/ — the reference Python library for near-duplicate deduplication at scale.
Great Expectations: https://docs.greatexpectations.io/ — declarative data quality suites.
Unstructured: https://docs.unstructured.io/ — parsing and normalisation of heterogeneous documents (PDF, HTML, DOCX, eml) before chunking.
Argilla: https://docs.argilla.io/ — human annotation UI for building the PII golden set and other calibration sets.
Llama Guard 4: Meta technical paper, multimodal safety classifier — useful as a second PII detection layer.
GDPR, EU AI Act, ENS, NIS2 — the regulatory frameworks whose compliance depends, in practice, on the discipline of layers 3 (PII) and 5 (lineage). The CEN/CENELEC technical standards for conformity assessment of GenAI systems under the EU AI Act are still pending final publication.

Dedup on lo0 — Tech Blog

End-to-end document ingestion: from PDF to indexed chunk

TL;DR

The analogy: an archive’s cataloguing chain

The six-stage pipeline

Stage 1 — Parsing: layout-aware vs plain text

OCR for scans

Stage 2 — Cleaning, normalising and deduplicating

Deduplication: why it matters

Stage 3 — Chunking

What the 2026 benchmarks say (and why semantic disappoints)

A worked numerical example of chunking

Stage 4 — Enriching with metadata

Stages 5 and 6 — Embedding and indexing

A cross-cutting stage — Incremental ingestion and idempotency

The arithmetic: sizing the corpus and the index

Corpus sizing and ingestion time

Vector index size

Applied to the generic 4×H100 cluster

Closing: quality is decided upstream

See also

References

RAG corpus curation: the active librarian who decides what goes in, what goes out and what gets signed

TL;DR

The analogy: the active librarian

The four data artefacts and where the RAG corpus fits

Layer 1 — Schema-validated ingest

Layer 2 — Three-level deduplication

Level A — Exact dedup (SHA-256 hash)

Level B — Near-duplicate (MinHash + LSH)

Level C — Semantic dedup (cosine over embeddings)

Policy on what to do with a duplicate

Layer 3 — PII anonymisation with measured precision/recall

Layer 4 — Anti-contamination against the golden eval set

Layer 5 — End-to-end lineage: from document to trace

The maths that matter

Applied to typical on-premise hardware

The seven traps that kill this stage

What we have not covered (upcoming posts)

See also

References