Qdrant on lo0 — Tech Blog

End-to-end document ingestion: from PDF to indexed chunk

Thu, 11 Jun 2026 09:00:00 +0000

Fourth piece in an operational series about squeezing a generic on-premise LLM cluster of 4×H100 SXM 80 GB. Its siblings in this batch: serving embeddings and rerankers with TEI in production details the inference piece that this ingestion feeds; GitOps for the inference stack with Flux versions and deploys this whole pipeline; and hardening and secrets for the sovereign stack protects the corpus and the credentials this ingestion touches. The final consumer of what we build here, an end-to-end sovereign assistant, is assembled in a later instalment.

TL;DR

A RAG system inherits the quality of its corpus, and the corpus inherits the quality of the ingestion that manufactured it. This is the garbage-in/garbage-out of retrieval: a chunk badly extracted from a table, a page header repeated a thousand times, a scanned PDF from which you only got OCR noise; all of it gets embedded, indexed and reappears as poisoned context in the model’s answer. Document ingestion is not an afternoon’s script, it is a six-stage pipeline with engineering decisions at every one: (1) extract/parse, where almost everything is decided: layout-aware (Docling, which with Granite-Docling-258M preserves tables, formulas and structure, and which according to IBM avoids classic OCR and is up to ~30× faster) against plain text extraction (PyMuPDF), with OCR or a VLM for scans; (2) clean/normalise, removing boilerplate, normalising Unicode, rebuilding broken paragraphs; (3) chunk, and here 2026 has shifted the consensus: a February 2026 benchmark put recursive at 512 tokens in the lead (69% accuracy) ahead of semantic chunking (54%), and late chunking adds global context at no extra storage cost; (4) enrich metadata, with source, page, section, timestamp, ACL/tenant: for filtered retrieval, citation and auditability; (5) embed, through an embeddings server such as TEI; (6) index, in pgvector or Qdrant, with payload, dense + sparse. In between, exact deduplication (hash) and near-duplicate detection (MinHash/LSH or cosine with a threshold), because near-duplicates degrade retrieval recall and diversity. The numbers matter: a corpus of $N_{docs}\times$ pages $\times$ tokens/page gives the total tokens and, at a given CPU throughput, the ingestion time; and $N_{chunks}\times d\times\text{bytes}$ gives the index size, which in int8 falls 4× against fp32. The reading for the 4×H100: ingestion is CPU work, which links to the RAG-on-CPU piece; the GPU only comes in if you parse with a VLM (Granite-Docling) or use a 7B embedder.

The analogy: an archive’s cataloguing chain

Picture the cataloguing department of a large document archive. It is not one person putting papers into boxes; it is a chain of workstations, each with a different craft and its own quality standard. A document goes in raw at one end and comes out the other turned into a record you can find in seconds. If one station does its job badly, the ones downstream inherit the error and amplify it.

The first station is intake and reading. A heterogeneous box arrives: typed reports, crooked photocopies, folded tables, microfiche. An expert archivist really reads each item: distinguishes the body of the text from the margins, rebuilds a table that spans two pages, transcribes by hand what the scanner could not read. A novice archivist, by contrast, photocopies everything flat and hands over a slab of text where the table columns are interleaved into nonsense. That is exactly the difference between layout-aware parsing and plain text extraction, and it is where almost all of the quality is won or lost.

The second station is weeding out. Before filing anything, someone removes the duplicates (three copies of the same memo, two near-identical versions of a report) and cleans off the useless marks: “COPY” stamps, footers repeated on every sheet, coffee stains. If you do not weed, the archive fills up with copies that, when someone searches, return the same document six times and bury everything else. This is deduplication and cleaning.

The third station cuts it into records. A 400-page book is not catalogued as one giant record; it is broken down into manageable entries, by chapter, by section, of a size a reader can take in at a glance. Too big and the record mixes topics; too small and it loses context. This is chunking, and the size of the record is the decision that most shapes retrieval.

The fourth station labels. Each record carries a shelf mark, a date, its provenance, an access level (can anyone see this, or only the legal department?). Without those labels you cannot filter a search or say which document a statement came from. These are the metadata: source, page, section, timestamp, ACL.

The fifth and sixth stations place the record on the indexed shelf: they translate it into the catalogue’s language, a vector, and put it in the right drawer of the filing cabinet, so that a query finds the related records without walking through the whole thing. This is embedding and indexing in the vector store.

The moral runs through the whole post: RAG cannot retrieve better than what the cataloguing chain filed. If the first station chopped up a table badly, no reranker will fix it afterwards. Ingestion is the quality station for the entire system, and almost all of it, like the archive full of patient people working with nobody timing them, is background work that fits on CPU.

The six-stage pipeline

The temptation to treat ingestion as “read the PDF and chop it up” is the source of 80% of mediocre RAG systems. Each stage has a quality standard and a characteristic failure, and failures cascade downstream: if you parse badly, cleaning does not recover what was lost; if you chunk badly, embedding freezes the error into the vector; if you do not label, you will not be able to filter or cite. The rest of the post walks through the six stations and their decisions.

Stage 1 — Parsing: layout-aware vs plain text

Almost everything is won or lost here. A PDF is not text: it is a set of instructions for drawing glyphs on a canvas. “Extracting the text” from a PDF means reconstructing a reading order that the format does not guarantee, and tables, columns and figures break it systematically.

There are two philosophies, and the choice shapes the rest of the pipeline.

Plain text extraction. Tools such as PyMuPDF (fast, robust, no heavy dependencies) read the PDF’s text stream and dump it out. For single-column documents, running prose and no tables, it is perfect: extremely fast, faithful and cheap on CPU. Its limit shows up with structure: a two-column table comes out with the cells interleaved, a double-column document mixes the end of one with the start of the other, and a scanned invoice does not come out at all because there is no text layer. PyMuPDF can also segment by the table of contents (TOC) when the PDF carries one, which helps with chunking by section (Omdena, Document Parsing for RAG: A Complete Guide for 2026).

Layout-aware parsing. Tools such as Docling (an open source project driven by IBM Research) and unstructured.io first understand the layout, identifying titles, paragraphs, tables, figures, lists and formulas, and only then extract the content while respecting that structure. Docling captures table structure (rows, columns, multi-level headers) and, in its 2026 evolution, does so with a VLM: Granite-Docling-258M, released by IBM in January 2026 under Apache 2.0, a compact vision-language model (~258M parameters, Granite 3 backbone + SigLIP2 visual encoder) that converts pages, whether PDF, slides or scans, directly into a structured format called DocTags, preserving tables, code, inline and block mathematics, and the document’s hierarchy (IBM, Granite-Docling: End-to-end document understanding; model card on Hugging Face; Docling repo). IBM states that the VLM route avoids classic OCR and that this “reduces errors and speeds up the solution by up to 30×” against a traditional OCR pipeline. That is an IBM Research figure; I quote it and treat it as indicative, not as an independent benchmark reproduced here.

unstructured.io offers partitioning strategies graded by document complexity: fast (plain text, quick), hi_res (identifies the layout, recommended when classifying tables and elements properly matters), VLM and auto, balancing speed, cost and accuracy (Unstructured, PDF Parsing Strategies for RAG). The practical rule: fast for prose, hi_res or VLM for documents with tables and structure.

OCR for scans

When the document has no text layer (a scan, a photo, a digitised microfiche), you have to recognise the characters. Three routes:

Classic OCR (Tesseract, PaddleOCR, EasyOCR, which is what Docling integrates with when explicit OCR is needed). Mature, CPU-friendly, good with clean text; it struggles with tables, handwriting and complex layouts.
End-to-end VLM (Granite-Docling and similar). The model “looks” at the page and emits structure directly, with no separate OCR stage. Better with complex layouts; this is where the GPU does come in if the VLM is large or the volume is high.
Hybrid: OCR for character transcription, a layout model for the structure.

The honest criterion: for a corpus of native PDFs with text, PyMuPDF or unstructured fast handle it on CPU and cheaply. For a corpus with dense tables, forms or scans, layout-aware Docling/Granite-Docling earns its cost in chunk quality, and it is the only stage of the pipeline where the GPU can be justified.

Document case	Recommended tool	Silicon
Native PDF, single column, prose	PyMuPDF / unstructured `fast`	CPU
PDF with tables, double column, hierarchy	Docling / unstructured `hi_res`	CPU (layout models)
Scan, form, handwriting, complex layout	Granite-Docling (VLM) or OCR+layout	GPU if large VLM / high volume
HTML, DOCX, PPTX	Docling (multi-format) / native parsers	CPU

Stage 2 — Cleaning, normalising and deduplicating

Freshly parsed text comes in dirty. Cleaning means removing what adds nothing and normalising what is represented in a thousand ways:

Boilerplate: headers and footers repeated on every sheet, page numbers, watermarks, HTML navigation menus, cookie banners. If you do not remove them, they get embedded a thousand times over and contaminate both the index and the answers.
Unicode normalisation (NFC/NFKC), whitespace and dashes: the same character represented in several ways breaks exact matching and dirties the embeddings.
Paragraph reconstruction: undoing the hard line breaks the PDF inserted mid-sentence, without merging paragraphs that were meant to stay separate.

Deduplication: why it matters

RAG suffers two ills from duplicates. The exact kind, the same document uploaded three times, inflates the index and makes a search return the same answer repeatedly, wasting the top-k context slots. The near-duplicate kind, two near-identical versions of a report, a document and its draft, is worse: they look different to a hash but say the same thing, and they degrade retrieval recall and diversity. This is not theory: in the MS MARCO V2 collection a substantial overlap of near-duplicates has been documented which, left untreated, degrades retrieval precision and reduces the diversity of retrieved documents (Ragnarök / TREC RAG 2024).

Two levels of dedup, complementary:

Exact (hash). Compute a sha256 of the normalised content of each document (or chunk) and discard the matches. Cost $O(N)$, trivial. It catches byte-for-byte duplicates.
Near-dup (MinHash + LSH, or cosine of embeddings with a threshold). For those that differ slightly but mean the same thing. MinHash compresses each document into a compact signature such that the probability of two signatures matching at a position equals the Jaccard similarity of the original shingle sets; combined with Locality-Sensitive Hashing (LSH) it finds every near-duplicate pair without comparing all against all, reducing a quadratic problem to almost linear (Brenndoerfer, MinHash, Jaccard, LSH). It is the dominant technique in cleaning LLM training corpora (C4 and RefinedWeb use it) and applies just the same to a RAG corpus (Zilliz, Data Deduplication at Trillion Scale). The alternative, cosine of embeddings with a threshold (say > 0.97), catches semantic duplicates that MinHash cannot see (paraphrasing), but it requires you to already have the embeddings and is more expensive.

The practical rule: exact hash always (it is free); MinHash/LSH for large corpora with versions; cosine with a threshold if paraphrasing is a real problem. And deduplicate before embedding: re-embedding a duplicate means spending compute on rubbish you will then have to filter out.

Stage 3 — Chunking

Chunking is the decision that most shapes retrieval, and the one that carries the most myths. The trade-off is threefold: chunk size ↔ retrieval granularity ↔ context cost.

Large chunks: each one holds more context and there is less risk of cutting an idea in half, but the search is less precise (one vector represents too many topics) and, on retrieval, you put more tokens into the LLM’s prompt, which means more cost and more risk of diluting what is relevant.
Small chunks: very granular, precise retrieval, but each chunk loses context (a 43-token fragment may mean nothing outside its section) and you need to retrieve more of them to cover an answer.

The strategies, from least to most sophisticated:

Fixed size + overlap. Cut every $N$ tokens with an overlap of $k$ tokens between consecutive chunks so as not to slice a sentence dead. Simple, predictable, a reasonable baseline. The overlap is the insurance against cutting an idea right at the boundary.
Recursive (LangChain’s RecursiveCharacterTextSplitter). It tries to cut on separators in priority order (paragraph, then sentence, then word) to respect the natural structure as far as possible before falling back to a hard cut. It is the workhorse.
Semantic. Embed sentences and cut where the similarity between consecutive sentences falls below a threshold, grouping by coherence of meaning. It sounds better on paper; in 2026 practice it has disappointed (see below).
Structure/layout-aware (by headings). It exploits the hierarchy the layout-aware parser has already extracted: one chunk per section or subsection. unstructured offers the by_title strategy, which opens a new chunk when a title element appears, avoiding mixing text from different sections (Unstructured docs). If you parsed with Docling/hi_res, this strategy is almost free and is usually the best one for well-structured documents.
Late chunking. The twist of 2024–2026: instead of chunking and then embedding each chunk separately, embed the whole document first (with a long-context encoder) and then apply the chunk boundaries by mean-pooling the token embeddings inside each span. The result: every chunk keeps the document’s global context (a pronoun or a reference that only makes sense given the previous paragraph is encoded in the vector) and all of it with no extra storage cost, because you end up with one vector per chunk just as before (Jina AI, Late Chunking; arXiv:2409.04701).

What the 2026 benchmarks say (and why semantic disappoints)

It is worth being sceptical about fashion. A Vecta benchmark from February 2026 covering 7 strategies over 50 academic papers put recursive at 512 tokens in first place with 69% accuracy, while semantic chunking came in at 54%, partly because it produced tiny fragments, averaging 43 tokens, too small to mean anything (Firecrawl, Best Chunking Strategies for RAG in 2026). A systematic analysis from January 2026 also identified a “context cliff” around 2,500 tokens, where answer quality drops once you feed in contexts that are too long, an extra argument against giant chunks ([ibid.]). The honest reading: recursive at a moderate size with overlap remains the baseline that is hard to beat; late chunking is the improvement with the best cost/benefit ratio when the model supports it; semantic promises more than it delivers.

A worked numerical example of chunking

Take a technical document of 30 pages, with ~500 tokens of useful prose per page after cleaning (tables and figures are chunked separately). That is $30 \times 500 = 15{,}000$ tokens of text. We chunk with recursive at 512 tokens and 20% overlap ($0.20 \times 512 \approx 102$ tokens). The effective “step” between the start of one chunk and the next is:

$$\text{step} = \text{size} - \text{overlap} = 512 - 102 = 410 \text{ tokens}$$

The number of chunks in the document is then, approximately:

$$N_{chunks} \approx \left\lceil \frac{T_{doc} - \text{overlap}}{\text{step}} \right\rceil = \left\lceil \frac{15{,}000 - 102}{410} \right\rceil \approx \lceil 36.3 \rceil = 37 \text{ chunks}$$

Without overlap it would have been $\lceil 15{,}000 / 512 \rceil = 30$ chunks; the 20% overlap costs us 7 extra chunks (~23% more) in exchange for not splitting ideas at the boundaries. That is the concrete price of overlap: more vectors to embed, index and store, in exchange for robustness in retrieval. We will use this factor when sizing the full corpus.

Stage 4 — Enriching with metadata

A chunk with no metadata is a record with no shelf mark: it exists but it is no use. Each chunk gets a payload attached with, at a minimum:

Source and location: document_id, document name/URI, page number, section/heading (which the layout-aware parser already gave you). Essential for citation: being able to say “this comes from document X, page 12, section 3.2” is what separates an auditable RAG from one that hallucinates with no traceability.
Timestamp: when it was ingested and the document’s date. It lets you filter by freshness and detect stale content.
ACL / tenant: who is allowed to see this chunk. It is critical and it is applied as a filter at retrieval time: a user from department A must not retrieve chunks marked for B only. Without this, the RAG system is a data leak waiting to happen.
Embedding model version (model_version): so you know which embedder generated each vector and can migrate without mixing incompatible spaces.

These metadata are not decoration: they enable filtered retrieval (searching only in what the user can see, or only in documents after a given date), citation (reconstructing the origin of every statement) and auditability (knowing what was retrieved, from where and when). All of it lives in the point’s payload in the vector store.

Stages 5 and 6 — Embedding and indexing

The last two stations translate the chunk into a vector and place it on the shelf.

Embedding. The chunks are sent in batches to an embeddings server, typically Hugging Face’s TEI (Text Embeddings Inference), which exposes the OpenAI /v1/embeddings contract and runs on CPU or GPU. It is throughput-bound work with no latency SLA: its natural home is the CPU with a small encoder in int8 (the sibling piece in this batch, serving embeddings and rerankers with TEI, details the how). It is worth emitting dense + sparse at the same time: the dense vector captures semantics, the sparse one (SPLADE/BM25-like) exact lexical overlap, and together they make hybrid retrieval more robust.

Indexing. The vectors, with their payload, are upserted into the vector store. Two reference options, both current in 2026:

pgvector (a PostgreSQL extension). Its great virtue is living inside Postgres: ACID transactions, joins with the relational metadata, a single database to operate. Version 0.8 added halfvec (half-precision vectors, 2× less storage) and 0.9 (early 2026) added support for sparse vectors and speed improvements. Its known limit: it has no native int8 quantisation, so high-dimensional embeddings consume RAM linearly (Encore, pgvector vs Qdrant 2026; Katz, Scalar and binary quantization for pgvector).
Qdrant (a dedicated vector engine). It supports int8 scalar quantisation (float32 → int8, 4× less memory) and product quantisation, native sparse vectors and RRF fusion for hybrid search. It is more efficient in memory and in quantisation; the cost is operating one more system alongside Postgres (Markaicode, pgvector vs Qdrant 2026).

The practical rule: pgvector if you already have Postgres and the corpus fits comfortably in RAM (a single database to operate and back up); Qdrant if memory efficiency and int8 quantisation are critical because of the corpus size. Synchronisation between the relational truth (Postgres) and the index (Qdrant) when both are used is detailed in PostgreSQL + Qdrant in microservices.

A cross-cutting stage — Incremental ingestion and idempotency

A living corpus changes: documents are added, edited, deleted. Re-indexing everything each night is expensive and causes windows of unavailability. The alternative is incremental ingestion with two pillars:

Idempotency by doc-id + hash. Each chunk is identified deterministically ({doc_id}_{chunk_index}) and each document carries a content hash. On reprocessing, if the hash has not changed, nothing is re-embedded: the compute is saved. If it changed, the old chunks for that doc_id are deleted and the new ones are upserted. An upsert with a deterministic id is idempotent: reprocessing an event twice does not create duplicates.
CDC (Change Data Capture). Instead of polling, Debezium reads the PostgreSQL WAL and propagates additions, edits and deletions to the index in near real time. A deletion in Postgres triggers the deletion of that document’s chunks in the vector store, avoiding the “ghost documents” that contaminate retrieval. The deep dive is in PostgreSQL + Qdrant in microservices and in the one on Debezium and CDC.

The arithmetic: sizing the corpus and the index

Two calculations you need to be able to do before provisioning anything.

Corpus sizing and ingestion time

Suppose a corporate corpus of $N_{docs} = 50{,}000$ documents, averaging 20 pages and 500 useful tokens per page after cleaning. The total tokens in the corpus:

$$T_{corpus} = N_{docs} \times \text{pages} \times \text{tokens/page} = 50{,}000 \times 20 \times 500 = 5 \times 10^{8} \text{ tokens}$$

Five hundred million tokens. Applying the overlap factor from the chunking example (~1.23×, the 23% extra chunks from the 20% overlap) and an effective size of 512 tokens/chunk, the number of chunks is:

$$N_{chunks} \approx \frac{T_{corpus}}{\text{step}} = \frac{5 \times 10^{8}}{410} \approx 1.22 \times 10^{6} \text{ chunks}$$

That is, ~1.22 million chunks to embed. At a conservative CPU embedding throughput of 3,000 tok/s per Xeon server in int8, the same figure we use in the RAG-on-CPU piece, the ingestion time for the first full corpus on one box is:

$$t_{ingesta} = \frac{T_{corpus}}{\text{throughput}} = \frac{5 \times 10^{8}}{3{,}000} \approx 1.67 \times 10^{5} \text{ s} \approx 46 \text{ hours}$$

46 hours on a single box sounds bad, but ingestion is embarrassingly parallel: the corpus is shared out. With 8 CPU servers it drops to ~6 hours, comfortably within a weekend window for the initial load; and subsequent incremental ingestions (only what changed) take minutes. Layout-aware parsing adds its own cost, since Docling with a VLM is slower than PyMuPDF, but it is also batch and parallelises the same way.

Vector index size

Each vector has dimension $d = 1024$ (that of bge-m3). In float32 (4 bytes/dimension), each vector takes:

$$\text{bytes}_{fp32} = d \times 4 = 1024 \times 4 = 4096 \text{ B} = 4 \text{ KB}$$

For the 1.22 M chunks, the dense vectors alone in fp32:

$$\text{size}_{fp32} = N_{chunks} \times d \times 4 = 1.22 \times 10^{6} \times 4096 \text{ B} \approx 5.0 \text{ GB}$$

In int8 (1 byte/dimension), each vector takes 1 KB and the total falls 4×:

$$\text{size}_{int8} = N_{chunks} \times d \times 1 = 1.22 \times 10^{6} \times 1024 \text{ B} \approx 1.25 \text{ GB}$$

To this you must add the HNSW index (~1.2× the size of the vectors for $m=16$) and the payload (metadata + chunk text, ~500 B/chunk → ~0.6 GB). In round numbers:

Configuration	Vectors	HNSW (~1.2×)	Payload	Total
fp32	~5.0 GB	~6.0 GB	~0.6 GB	~11.6 GB
int8	~1.25 GB	~1.5 GB	~0.6 GB	~3.4 GB

The reading: a corpus of 50,000 documents fits in the RAM of a single node even in fp32, and in int8 (Qdrant) it fits with room to spare, which keeps search latency in single-digit milliseconds. int8 quantisation is almost always the balance point, saving 4× with a recall loss typically below 1%. (These numbers are order-of-magnitude, with the constants and assumptions stated; they are for sizing, not for pinning down an invoice.)

Applied to the generic 4×H100 cluster

Let us bring this down to the series’ cluster: 4×H100 SXM 80 GB plus a generic CPU fleet (Xeon with AMX, NUCs). The correct split for ingestion is almost all CPU, with the GPU as an occasional exception:

Cleaning, dedup, chunking, metadata, ingestion embedding and indexing → the CPU fleet. All of this is batch work, throughput-bound, with no latency SLA. It is exactly the “data plane” the RAG on CPU piece talks about: no H100 should spend a cycle chunking documents or building an HNSW index (which was always CPU by design). The nightly incremental re-indexing of a corpus that changes on the scale of hours is the textbook case of “CPU work in no hurry”.
The GPU only comes in at two points. First, in VLM parsing: if the corpus has scans, dense tables or forms and you choose Granite-Docling-258M or another vision-language model, that parsing can be accelerated on GPU, although at 258M parameters it is light and, at moderate volumes, runs on CPU without drama. Second, in the large embedder: if retrieval quality demands a 7B embedder (gte-Qwen2, NV-Embed) instead of bge-m3 (568M), that embedder is once again an LLM-class model and lives where the 7B models live, on the GPU.
The 4×H100 are reserved for generation. As throughout the series, the expensive and scarce silicon is kept for what is latency-bound, the LLM producing the answer, and at most for the peaks of VLM parsing or 7B embedding the CPU cannot absorb. To give a sense of the ceiling: a 4×H100 node serving bge-m3 through TEI runs at around ~2,000 chunks/s, against the thousands of tok/s of a Xeon in int8; but using the H100s for daily ingestion means spending the resource the whole organisation fights over on a job the CPU fleet does overnight without anyone missing it.

The sentence that sums up the split: ingestion is the archive’s cataloguing chain, and almost all of it is done by patient, cheap staff (CPU); the star writer (GPU) is only disturbed when there is a page no classic OCR can decipher.

Closing: quality is decided upstream

The recurring error in mediocre RAG is not in the reranker or the prompt: it is in an ingestion that parsed a table badly, did not weed out the duplicates, or chunked at a size that destroys context. Garbage-in, garbage-out: no downstream component fixes what ingestion spoiled upstream. Investing in the cataloguing chain, with layout-aware parsing where it is needed, real dedup, measured chunking, complete metadata, is what moves the needle on system quality the most, and almost all of it fits on the CPU fleet. The GPU, like the star writer, should only touch the corpus when its brain is genuinely required.

References

IBM — Granite-Docling: End-to-end document understanding (Granite-Docling-258M, Apache 2.0, DocTags, enero 2026). https://www.ibm.com/new/announcements/granite-docling-end-to-end-document-conversion
ibm-granite — granite-docling-258M model card (VLM ~258M, Granite 3 + SigLIP2). https://huggingface.co/ibm-granite/granite-docling-258M
Docling project — Docling: Get your documents ready for gen AI (layout, tablas, multi-formato). https://github.com/docling-project/docling
Unstructured — PDF Parsing Strategies for RAG (fast / hi_res / VLM / auto; chunking by_title). https://unstructured.io/blog/mastering-pdf-transformation-strategies-with-unstructured-part-2 · https://docs.unstructured.io/
Omdena — Document Parsing for RAG: A Complete Guide for 2026 (PyMuPDF, TOC, pipeline). https://www.omdena.com/blog/document-parsing-for-rag
Firecrawl — Best Chunking Strategies for RAG (and LLMs) in 2026 (benchmark Vecta feb-2026: recursive 512 → 69%, semántico 54%; context cliff ~2.500 tok). https://www.firecrawl.dev/blog/best-chunking-strategies-rag
Günther, M., et al. — Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models. arXiv 2409.04701. https://arxiv.org/abs/2409.04701 · Jina AI: https://jina.ai/news/late-chunking-in-long-context-embedding-models/
Brenndoerfer, M. — MinHash: Jaccard Similarity, LSH, and Near-Duplicate Detection. https://mbrenndoerfer.com/writing/minhash-algorithm-jaccard-similarity-lsh-deduplication
Zilliz — Data Deduplication at Trillion Scale (MinHash LSH dominante en limpieza de corpus). https://zilliz.com/blog/data-deduplication-at-trillion-scale-solve-the-biggest-bottleneck-of-llm-training
Ragnarök / TREC RAG 2024 — near-duplicados en MS MARCO V2 degradan recuperación y diversidad. https://arxiv.org/pdf/2406.16828
Encore — pgvector vs Qdrant in 2026 (halfvec, sparse, límites de cuantización de pgvector). https://encore.dev/articles/pgvector-vs-qdrant
Markaicode — pgvector vs Qdrant (2026 Benchmarks) (Qdrant int8 scalar quantization 4×). https://markaicode.com/vs/pgvector-vs-qdrant/
Katz, J. — Scalar and binary quantization for pgvector. https://jkatz05.com/post/postgres/pgvector-scalar-binary-quantization/
Hugging Face — Text Embeddings Inference (TEI), backends CPU/GPU, endpoints OpenAI-compatibles. https://github.com/huggingface/text-embeddings-inference

Function calling and tool-augmented retrieval: the detective who knows which file to ask for

Thu, 04 Jun 2026 10:00:00 +0200

This post goes deeper into the agentic reasoning mechanism that extends the RAG described in RAG with reranker and hybrid retrieval. The retriever invoked when the LLM chooses vector_search is exactly the pipeline in that article. The JSON Schema that defines each tool call is structured output applied to the tool interface. And the agent’s requests pass through the L7 inference gateway before reaching the model.

TL;DR

A naive RAG always queries the same source. Function calling gives the LLM the ability to decide which tool to invoke (vector store, SQL, web search) according to what the query actually needs. The ReAct pattern chains those invocations in a reasoned loop until there is enough evidence. A 3-iteration pipeline with Llama-3.1-70B on on-premise hardware takes ≈ 1.1 s against the ≈ 300 ms of a single-step RAG; the gain is not in speed but in queries that a naive RAG simply cannot answer. The critical eval metric is tool selection accuracy: the percentage of turns in which the model picks the right tool, measured over a synthetic eval set.

The analogy: the detective who knows which file to ask for

A detective in a novel does not go to the same filing cabinet regardless of the case that lands on his desk. When he takes a case, he reasons first: what kind of evidence do I need? If there are witnesses, he asks for the statements (vector search over unstructured documents). If there are financial transactions, he asks the bank for the accounting records (SQL over the structured database). If the suspect has recent activity that the company cannot possibly have indexed, he goes to the press archive (web search). He does not consult all three sources at once on every case: he picks the one the evidence requires, receives the result, reasons again if he needs more, and only writes the report when he has enough evidence.

A bad detective always goes to the same filing cabinet. A naive RAG is that bad detective: it vectorises the query, goes to the vector store, and returns whatever it finds even if the question was “how many contracts?”, something no PDF chunk can answer better than a COUNT(*) in SQL.

Function calling is giving the LLM the ability to reason about which source to ask for, and to invoke it in a structured way. The analogy has three edges worth keeping in mind:

The detective does not improvise the file he asks for: there is a catalogue of available sources with a description of what each one contains. The tool description in the system prompt does that job.
The detective can ask for several pieces of evidence at once if they are independent (parallel tool calling).
The detective knows when to stop: if after N rounds he reaches no conclusion, he declares that he does not have enough evidence. The agent has an iteration limit for the same reason.

What function calling is: the anatomy of a tool call

Function calling, also called tool use, is a mechanism by which the LLM, instead of generating free text as its answer, generates a structured JSON object representing a tool invocation. The system intercepts that JSON, executes the real tool, and returns the result as a message with the tool role in the conversation.

Defining tools in the system prompt

Each tool is defined through a JSON Schema that specifies name, description and parameters. This JSON Schema is exactly the same mechanism described in structured output, applied here to the tool interface:

{
 "tools": [
 {
 "name": "vector_search",
 "description": "Search internal company documents about policies, contracts and procedures. Use when the query requires unstructured text, document context or semantic similarity.",
 "parameters": {
 "type": "object",
 "properties": {
 "query": {"type": "string", "description": "Natural language search query"},
 "top_k": {"type": "integer", "default": 5}
 },
 "required": ["query"]
 }
 },
 {
 "name": "sql_query",
 "description": "Query the SQL database for structured metrics, counts, aggregations and financial data. Use when the query requires exact numbers, filters, sums or joins over structured records.",
 "parameters": {
 "type": "object",
 "properties": {
 "query": {"type": "string", "description": "Parameterized SQL query with $1, $2 placeholders"},
 "params": {"type": "array", "items": {}, "description": "Parameter values for the placeholders"}
 },
 "required": ["query"]
 }
 },
 {
 "name": "web_search",
 "description": "Search public web for real-time information, recent news or current prices. Use only when data is public and not covered by internal sources.",
 "parameters": {
 "type": "object",
 "properties": {
 "query": {"type": "string"}
 },
 "required": ["query"]
 }
 }
 ]
}

The life cycle of a tool call

When the LLM decides to invoke a tool, the message it generates in place of free text has this structure (OpenAI-compatible format, the same one vLLM supports):

{
 "role": "assistant",
 "tool_calls": [
 {
 "id": "call_01",
 "type": "function",
 "function": {
 "name": "sql_query",
 "arguments": "{\"query\": \"SELECT COUNT(*), SUM(amount) FROM contracts WHERE amount > $1 AND year = $2 AND provider_region = $3\", \"params\": [100000, 2025, \"EU\"]}"
 }
 }
 ]
}

The system executes the tool and returns:

{"role": "tool", "tool_call_id": "call_01", "content": "{\"count\": 47, \"total\": 8300000}"}

The LLM receives that message as a continuation of the conversation and decides whether it needs more information or can generate the final answer.

Support in OSS models

In 2026, native function calling support (not emulated via system prompt) is available in:

Llama 3.1 / 3.3+: native tool call format, supported in vLLM with --enable-auto-tool-choice --tool-call-parser llama3_json
Qwen 2.5+: native support with --tool-call-parser hermes
Mistral NeMo / Mistral 7B Instruct v0.3+: native format with --tool-call-parser mistral
Qwen3 (2025+): extended native support with parallel tool calling

Without --enable-auto-tool-choice, vLLM can force the tool call format via guided decoding (JSON Schema), but the ability to decide when to invoke a tool versus generating free text requires specific model training.

The ReAct pattern: Reasoning + Acting

ReAct (Yao et al., 2022) is the loop that turns an LLM with tools into an agent with reasoning. The central idea is that the model interleaves thoughts (explicit reasoning about what to do) with actions (tool invocations) and observations (results received). This loop repeats until the model reckons it has enough information to answer, or until the iteration limit is reached.

Thought → Action → Observation → Thought → Action → Observation → ... → Response

A full ReAct trace: a real production query

Query: “How many contracts above 100k€ did we sign in 2025 with EU providers, and which special conditions recur?”

This query requires two different sources: the exact numbers live in SQL, and the context of special conditions lives in the contract PDFs indexed in the vector store.

Iteration 1

Thought: The query asks for aggregated data (count and sum) with exact filters over structured fields (amount, year, region). This is a SQL query, not a semantic search.

Action: sql_query

SELECT COUNT(*) AS num_contratos, SUM(amount) AS total_eur
FROM contracts
WHERE amount > $1 AND year = $2 AND provider_region = $3

params: [100000, 2025, "EU"]

Observation: {"num_contratos": 47, "total_eur": 8300000}

Iteration 2

Thought: I have the numbers. Now I need the special conditions that recur in these contracts. That kind of information is in the free text of the PDFs, not in the structured table. I go to the vector store.

Action: vector_search

"EU provider contracts 2025 amount above 100k special conditions recurring clauses"

top_k: 5

Observation:

[chunk_1] Provider A contract — clause 8.3: late-delivery penalty of 2% per month on the invoiced amount...
[chunk_2] Provider C contract — clause 7.1: price revision linked to the euro area CPI capped at 4% per year...
[chunk_3] Provider B contract — clause 8.3: late-delivery penalty of 2% per month...

Thought: I have enough data from both sources. Clause 8.3 on late-delivery penalties appears in at least two of the retrieved contracts. I can synthesise the answer.

Response: “In 2025 you signed 47 contracts with EU providers above 100,000 €, for a cumulative total of 8.3 M€. The special condition that appears most often in the higher-value contracts is the late-delivery penalty clause (8.3 in the contract templates of Provider A and B): 2% per month on the invoiced amount. The price revision linked to the euro area CPI, capped at 4% per year (clause 7.1), also recurs.”

This trace has 2 iterations. A more complex query (comparing with the previous year, cross-referencing with payment status) would add further iterations. Common practice is to set a limit of 5–10 iterations maximum to avoid loops.

The three sources and when to use each one

The choice of source is neither arbitrary nor left solely to the LLM: the architecture defines which tools exist and how they are described. The table below summarises the selection criteria:

Characteristic	Vector store (Qdrant)	SQL / structured (PostgreSQL)	Web search
Data type	Free text, documents, PDFs	Tables with a fixed schema	Public pages, news
Structure	Unstructured	Highly structured	Semi-structured
Temporality	Static index (periodic refresh)	Real time (transactional)	Real time (crawl)
Typical latency	5–50 ms	10–200 ms	200–2,000 ms
Privacy	Internal data, full sovereignty	Internal data, full sovereignty	Public data only
Natural query	Yes (natural language → embedding)	No (parameterised SQL)	Yes
Exact aggregations	No	Yes (`COUNT`, `SUM`, `GROUP BY`)	No
When to use	Document context, semantics, PDFs	Metrics, counts, exact filters, joins	Data that does not exist internally and is public

The most important rule of thumb: if the question contains words like “how many”, “sum”, “total”, “more than X”, “in year Y” and the data is in a structured table, the right answer is sql_query. If the question asks for context, explanations, clauses, procedures or examples from documents, the answer is vector_search. If it asks for the current price of something or recent news about a third party, web_search, but only if no data sovereignty is involved.

Tool routing: how the LLM picks the right tool

The description of each tool in the system prompt is the most critical factor for routing accuracy. An LLM with good function calling capabilities can still choose badly if the descriptions are ambiguous or overlap.

Descriptions that work versus those that do not

Weak description (leads the LLM to use the wrong tool):

"search_docs" — Searches for information in the available sources.
"query_data" — Gets data from the system.

Strong description (delimits precisely when to use each one):

"vector_search" — Search internal company documents about policies, contracts and procedures.
Use when the query requires unstructured text, document context or semantic
similarity. NOT for counts, sums or exact filters.
"sql_query" — Query the SQL database for structured metrics, counts, aggregations and
financial data. Use when the query requires exact numbers, filters, sums or
joins over structured records. NOT for finding document context.

The difference lies in two elements: (1) positive use-case examples, and (2) explicit exclusions with NOT for. Both reduce the semantic overlap between tools and improve tool selection accuracy.

Parallel tool calling

When two tools are independent of each other, that is, when the result of one does not affect the query of the other, the LLM can invoke them simultaneously in the same turn:

{
 "tool_calls": [
 {"id": "call_01", "function": {"name": "sql_query", "arguments": "..."}},
 {"id": "call_02", "function": {"name": "vector_search", "arguments": "..."}}
 ]
}

The system executes both in parallel and returns both observations before the LLM’s next turn. This cuts total latency when the queries are independent: instead of 2 sequential iterations (2 × tool_latency), you pay 1 × max(sql_latency, vector_latency). For the detective example: if he needs both the accounting records and the witness statements to answer, he can ask for them at the same time.

Tool selection accuracy: the eval metric

Tool selection accuracy is the percentage of turns in which the LLM picks the right tool given a set of evaluated queries:

\[ \text{TSA} = \frac{\text{turns with correct tool chosen}}{\text{total turns with expected tool call}} \]

It is measured over a synthetic eval set built with (query, expected_tool, expected_args) triples. An example of a minimal eval set:

Query	Expected tool	Failure indicator
“How many orders in March?”	`sql_query`	LLM uses `vector_search`
“What does the returns policy say?”	`vector_search`	LLM uses `sql_query`
“What is the price of copper today?”	`web_search`	LLM uses `vector_search`
“Sum the Q3 contracts”	`sql_query`	LLM uses `vector_search`

A TSA < 0.85 in a production agent is a sign that the tool descriptions need revising before the model does. For more detail on how to build these evals, see LLM evals.

Error handling in tool calls

SQL injection via prompt

The most serious risk in tool-augmented retrieval is that the LLM generates malicious SQL, either because a user induced it via prompt injection, or because the model hallucinated a destructive query. This attack vector is covered in detail in LLM Guard, but the minimum rules on the tool side are:

Rule 1: parameterised queries always, never direct interpolation.

# NEVER this:
cursor.execute(f"SELECT * FROM contracts WHERE provider = '{llm_output}'")

# Always this:
cursor.execute("SELECT * FROM contracts WHERE provider = $1", [llm_output])

Rule 2: DB user with minimum permissions. The user the agent runs SQL as must have SELECT on the necessary tables and nothing else. No DROP, INSERT, UPDATE or DELETE.

Rule 3: table allowlist. The system validates that the query generated by the LLM only references tables on an allowlist before executing it.

Rule 4: per-query timeout. Queries that block for too long, potentially induced to DoS the database, are cancelled with a configured timeout.

Rate limits, timeouts and tool errors

When a tool fails, the error is returned to the LLM as an observation:

{"role": "tool", "tool_call_id": "call_01", "content": "{\"error\": \"timeout after 5s\", \"tool\": \"web_search\"}"}

The system prompt must instruct the LLM on what to do in this case:

If a tool returns an error or is unavailable, acknowledge the limitation in your response.
Do not retry more than once. If web_search is unavailable, state that real-time data
is not accessible at this moment and answer with available internal sources only.

This stops the agent from entering retry loops and handles graceful degradation: if web_search is unavailable, it answers with what it has in the internal sources.

Diagram of the ReAct loop with the three sources

Latency maths of the ReAct pipeline

Each iteration of the ReAct loop has three latency components:

\[ T_{\text{iter}} = \text{TTFT}_{\text{LLM}} + T_{\text{tool}} + \Delta_{\text{context}} \]

where:

TTFT_LLM: time to the LLM’s first token (dominated by the prefill of the accumulated context)
T_tool: tool execution time
Δ_context: overhead of the growing context window (each iteration adds the previous output to the context)

Reference values: Llama-3.1-70B on 4×H100 SXM (320 GB, NVLink)

With Llama-3.1-70B in FP8 on a node with 4×H100 SXM (320 GB HBM3, NVLink 900 GB/s), the typical values in production are:

Component	Value
TTFT with context < 4k tokens	≈ 150 ms
TTFT with context 8k tokens	≈ 220 ms
`sql_query` (simple query, indexed)	≈ 50 ms
`vector_search` (top-5, Qdrant in RAM)	≈ 20 ms
`web_search` (external API)	≈ 600 ms

Total latency by number of iterations

1-iteration pipeline (simple query, a single tool):

\[ T_1 = 150 + 50 = 200 \text{ ms} + \text{final synthesis} \approx 200 + 300 = 500 \text{ ms} \]

2-iteration pipeline (SQL + vector_search in sequence):

\[ T_2 = (150 + 50) + (180 + 20) + 400 = 800 \text{ ms} \]

The context in the second iteration already includes the result of the first, so TTFT rises slightly to ≈ 180 ms.

3-iteration pipeline (the most common case in complex queries):

\[ T_3 = (150 + 50) + (180 + 20) + (200 + 50) + 450 \approx 1,100 \text{ ms} \]

Parallel tool calling (SQL + vector_search in parallel, a single iteration):

\[ T_{\text{parallel}} = 150 + \max(50, 20) + 400 = 600 \text{ ms} \]

When the two queries are independent, parallel tool calling trims latency from ≈ 800 ms to ≈ 600 ms: a 25% improvement over the 2 sequential iterations case.

Comparison with a naive RAG

Configuration	Latency	Queries it can answer
Naive RAG (1 retriever, 1 step)	≈ 300 ms	Document context queries
ReAct 1 iteration (SQL)	≈ 500 ms	Structured aggregation queries
ReAct 2 iterations (SQL + vector)	≈ 800 ms	Hybrid numeric + context queries
ReAct 3 iterations	≈ 1,100 ms	Complex multi-source queries
ReAct with web_search	≈ 1,500 ms	Queries requiring real-time data

The practical conclusion: multi-hop ReAct is 3–5× slower than a single-step naive RAG. The gain is not in speed but in query coverage: a naive RAG cannot answer “how many contracts?” because that answer is in no text chunk at all. For applications with a strict latency SLO (< 500 ms), you have to work out whether the use case really needs ReAct or whether a well-configured RAG with hybrid retrieval covers 90% of the queries.

On-premise hardware for ReAct agents

A ReAct agent with Llama-3.1-70B in production has different requirements from a naive RAG, because the context grows with each iteration and prefill throughput is more critical.

Recommended configuration: 4×H100 SXM (320 GB HBM3, NVLink 900 GB/s)

Llama-3.1-70B in FP8: fits in 2×H100 (70B params × 1 byte/param ≈ 70 GB + KV cache). With 4×H100 you can serve it with tensor parallelism TP=4, cutting prefill TTFT by ≈ 2×.
Qdrant instance: it can sit on the same node (if the collection fits in RAM) or on a dedicated node. For collections < 50M vectors of 768 dims: ≈ 150 GB, which fits in the RAM of a dual-socket server.
PostgreSQL: separate node or managed instance. The agent adds no unusual load to SQL; the queries are simple and bounded by timeout.
vLLM with --enable-auto-tool-choice --tool-call-parser llama3_json --max-model-len 16384: a 16k token context comfortably covers the 5–10 turns of a ReAct pipeline.

Minimum viable configuration: 2×H100 SXM (160 GB)

Llama-3.1-70B in FP8 with TP=2. TTFT ≈ 250–300 ms for 4k token contexts (a 60–100% increase over TP=4).
Good enough for internal workloads with < 20 concurrent requests.
Not advisable for an SLO < 1 s with more than 5 concurrent users and a long context.

What we have not covered

Agentic retrieval loops with planning. ReAct is the simplest agent pattern. When a query requires decomposition into sub-tasks with dependencies, you need orchestration frameworks such as LangGraph (state graphs), smolagents (Hugging Face, agents with Python code as actions) or llama-index Agents (planning + retrieval pipeline). These introduce a planning step before the execution loop.

MCP (Model Context Protocol). Anthropic’s emerging standard, with OSS implementations, for defining tools portably across frameworks and hosts. Instead of defining each tool’s JSON Schema separately in every application, MCP centralises those definitions in an MCP server that any compatible client can discover and invoke. Adoption in 2025–2026 is fast among OSS frameworks (LangChain, smolagents, OpenWebUI).

Tool caching. If the same tool call (same arguments, same tool) is going to be invoked several times within the same context or in very similar contexts, the result can be cached. The mechanism is analogous to the semantic cache described for RAG: before executing the tool, the hash of the arguments (or its embedding, for semantic matching) is compared against a cache with a TTL. Especially valuable for sql_query with frequent queries and data that changes little.

Multi-agent. When an orchestrating agent delegates sub-tasks to specialised agents, one for SQL, one for document retrieval, one for code generation, you enter multi-agent system territory. Each sub-agent can have its own tool set and its own LLM (possibly smaller and more specialised). Coordination between agents introduces additional tracing and observability complexity.

References

Yao, S. et al. (2022). ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629. https://arxiv.org/abs/2210.03629
vLLM documentation. Tool calling. https://docs.vllm.ai/en/stable/features/tool_calling.html
Qdrant documentation. Search. https://qdrant.tech/documentation/concepts/search/
OpenAI. Function calling. https://platform.openai.com/docs/guides/function-calling
Meta AI. Llama 3.1 Model Card. https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md
Qwen Team (Alibaba). Qwen2.5 Technical Report. arXiv:2412.15115. https://arxiv.org/abs/2412.15115
Anthropic. Model Context Protocol. https://modelcontextprotocol.io
OWASP. LLM Top 10 for Large Language Model Applications. LLM01: Prompt Injection. https://owasp.org/www-project-top-10-for-large-language-model-applications/

Semantic cache in RAG: the receptionist with a photographic memory

Thu, 04 Jun 2026 08:00:00 +0200

TL;DR

In a RAG system with real traffic, 30–70% of queries are semantically equivalent to some earlier one even when the text differs. The semantic cache intercepts those queries ahead of the retriever and the LLM, returning the answer already computed if the cosine similarity with a previous query exceeds a threshold θ (typically 0.92–0.95). At 10,000 requests/day and a 45% hit rate, that amounts to not running 4,500 LLM generations: roughly 0.62 GPU hours saved every day on a cluster running Llama-3.1-70B. The fundamental trade-off is that a high θ gives more accurate answers but less saving; a low θ maximises saving but can return incorrect answers for subtly different queries.

The analogy: the receptionist with a notebook

Picture the front desk of a 400-room hotel. Over the course of the day, the receptionist takes hundreds of questions. But if you look through the log book, you will see that 60% of those questions are variants of the same ten:

“Where is the gym?”
“What time is breakfast?”
“Do you have parking?”
“How do I connect to the WiFi?”

By the third day, the receptionist has built a mental notebook of answers. When someone asks “where can I go to exercise?”, they do not call the concierge (retrieval) or open the hotel’s 300-page internal manual (LLM): they look at the notebook, work out that the question is the same as “where is the gym?”, and answer in two seconds.

But when someone asks “what time does the gym close today?”, the receptionist knows the notebook cannot be trusted: the schedule may have changed because of a private event. They have to call the concierge.

That is exactly the mechanism of a semantic cache:

The notebook is the cache store (Redis with a vector index, or a Qdrant collection).
Working out that “exercise” ≈ “gym” is the cosine similarity search with threshold θ.
Calling the concierge is retrieval over the corpus.
Consulting the manual is LLM generation.
“Today” is the temporal query signal that invalidates the cache.

The threshold θ is exactly what separates “where is it” (semantically the same) from “what time is it open today” (semantically different). It is not magic: it is vector arithmetic over learned representations.

The problem in production

A typical RAG pipeline has three layers of latency and compute: embedding the query, vector search over the corpus, and generation with the LLM. In development, that cost is irrelevant. In production with 50 concurrent users, each of those layers scales linearly with the number of requests.

The problem is that users ask the same questions over and over, with slightly different wording:

Original query	Equivalent query
“How do I configure the agent?”	“What is the process for configuring the agent?”
“error installing the dependency”	“the dependency installation fails”
“what is an embedding?”	“explain to me what embeddings are”

Running the full pipeline for each of these variants is pure waste. Empirical studies in technical support and corporate Q&A systems report that between 30% and 70% of a day’s queries are semantically redundant with respect to earlier queries from the same week.

The query distribution in real systems follows a power law similar to the Zipf distribution: the 100 most frequent topics account for roughly 60% of total traffic. A well-calibrated cache captures exactly that concentration.

How the semantic cache works

The full flow is shown in the diagram below. Let us describe it in prose first.

When a new query $q$ arrives:

Embedding the query: $q$ is embedded with the same model used to index the corpus. This is critical: if the corpus was indexed with text-embedding-3-large and the cache uses a different embedder, the vector spaces are not comparable.
Search in the cache store: an ANN (Approximate Nearest Neighbor) search is run over the vectors of previously cached queries. The most similar query $q^*$ and its cosine similarity $s$ are retrieved.
Threshold decision:
$$ \text{answer} = \begin{cases} r^* & \text{if } s(q, q^*) \geq \theta \\ \text{pipeline}(q) & \text{if } s(q, q^*) < \theta \end{cases} $$
where $r^*$ is the cached answer associated with $q^*$.
On a miss: the full pipeline runs (retrieval + LLM). The generated answer is stored in the cache with a configurable TTL for future similar queries.

The cache store is not an ordinary key-value database. It is a vector index over the query embeddings, with the values being the generated answers. Each entry has the structure:

{vector: embed(q), response: r, ttl: T, metadata: {...}}

Flow diagram

The semantic cache flow as middleware between the gateway and the retriever. Cache hits skip the vector search over the corpus and the LLM generation entirely.

The threshold θ and its trade-off

The threshold θ is the most sensitive parameter in the system. It works exactly like the receptionist’s recognition threshold: if it is too demanding, it will only identify textually identical questions and the notebook will not be much use. If it is too lax, it will return the answer to “where is the gym?” to someone who asked “what time does the gym close?”.

The cosine similarity between two vectors $\mathbf{a}$ and $\mathbf{b}$ is:

$$ s(\mathbf{a}, \mathbf{b}) = \frac{\mathbf{a} \cdot \mathbf{b}}{\|\mathbf{a}\| \cdot \|\mathbf{b}\|} $$

For prose text (Spanish or English), modern embedders such as text-embedding-3-large or nomic-embed-text assign cosine similarities of around 0.90–0.96 to semantically equivalent paraphrases and similarities of 0.75–0.88 to related but non-equivalent queries.

The cache quality metric is not just hit rate: it is precision@cache, defined as the fraction of cached answers that remain correct for the new query. A cached answer is “correct” if an evaluator (another LLM, or metrics such as BERTScore) judges it equivalent to the one the full pipeline would have generated for that specific query.

θ	Estimated hit rate	Estimated precision@cache	Effective saving
0.85	~65%	~72%	~47%
0.90	~55%	~85%	~47%
0.92	~48%	~91%	~44%
0.93	~45%	~94%	~42%
0.95	~35%	~98%	~34%
0.97	~18%	~99.5%	~18%

The effective saving is defined as $\text{hit rate} \times \text{precision@cache}$, since a hit with an incorrect answer is not a saving: it is an error that can cost more in lost trust than what was saved in GPU.

The empirical sweet spot for most corporate Q&A applications in Spanish or English lies between θ = 0.92 and θ = 0.95. In highly specialised domains where small nuances change the answer (medicine, law, finance), θ ≥ 0.95 is advisable.

The arithmetic of the saving

Let us put concrete numbers on a real system.

Base configuration:

10,000 requests/day
Technical corpus of 1 million chunks in a Qdrant index
LLM: Llama-3.1-70B on 4×H100 SXM (320 GB, NVLink)
Average answer: 200 output tokens
LLM throughput on this hardware: ~400 tokens/s/GPU with batching (continuous batching enabled, see continuous-batching-fundamentos)

Cost of a request without cache:

Embedding the query takes ~2 ms on a GPU. The ANN vector search over 1 M chunks in Qdrant takes ~5 ms (measured empirically with HNSW, ef=128). Generating 200 tokens at 400 tok/s in total (4 GPUs) comes to:

$$ t_{\text{LLM}} = \frac{200 \text{ tokens}}{400 \text{ tok/s}} = 0.5 \text{ s per request} $$

If 10,000 requests reach the LLM in a day, the total GPU time spent on generation is:

$$ T_{\text{GPU}} = 10{,}000 \times 0.5 \text{ s} = 5{,}000 \text{ s} \approx 1.38 \text{ GPU hours per day} $$

With a semantic cache at θ = 0.93 and a hit rate of ~45%:

Only 5,500 requests (55%) reach the LLM:

$$ T_{\text{GPU,cache}} = 5{,}500 \times 0.5 \text{ s} = 2{,}750 \text{ s} \approx 0.76 \text{ GPU hours per day} $$

Saving:

$$ \Delta T_{\text{GPU}} = 1.38 - 0.76 = 0.62 \text{ GPU hours/day} $$

In inference compute, this amounts roughly to being able to serve 45% more users without adding hardware, or cutting inference costs by 45% if you work with external APIs billed per token.

The cost of the semantic cache itself (embedding the query + ANN search over the cache store) is ~7 ms per request, negligible against the 500 ms of generation avoided on hits.

Zipf distribution of topics:

The reason it works is the Zipf distribution of traffic. If we number topics by frequency (topic 1 = most frequent), the frequency of topic $k$ is proportional to $1/k$. With 1,000 distinct topics:

$$ \text{fraction of traffic covered by top-}N = \frac{\sum_{k=1}^{N} 1/k}{\sum_{k=1}^{1000} 1/k} \approx \frac{\ln N}{\ln 1000} = \frac{\ln N}{6.9} $$

For the top-100 topics: $\ln(100)/6.9 \approx 4.6/6.9 \approx 67\%$ of traffic. The cache does not need to cover every topic: it captures 67% of traffic while covering only 10% of the topics.

OSS stack 2026

GPTCache

GPTCache is the reference library for a standalone semantic cache. Its architecture is modular:

Embedder: ONNX Runtime with converted models (onnx/all-MiniLM-L6-v2 by default), with no GPU dependency for the cache layer.
Vector store: Faiss (local), Milvus, or Qdrant.
Scalar store: SQLite (development) or Redis (production) for metadata, TTL, and answers.
Similarity evaluation: cosine by default, configurable.

Minimal configuration in Python:

from gptcache import cache
from gptcache.adapter import openai
from gptcache.embedding import Onnx
from gptcache.manager import CacheBase, VectorBase, get_data_manager
from gptcache.similarity_evaluation.distance import SearchDistanceEvaluation

onnx = Onnx()
data_manager = get_data_manager(
 CacheBase("redis", url="redis://localhost:6379"),
 VectorBase("qdrant", host="localhost", collection_name="query_cache")
)
cache.init(
 embedding_func=onnx.to_embeddings,
 data_manager=data_manager,
 similarity_evaluation=SearchDistanceEvaluation(),
)
cache.set_openai_key()

GPTCache intercepts calls to the OpenAI API (or to compatible proxies) transparently. The TTL is configured at the data_manager level.

MeanCache

MeanCache (2024) extends GPTCache to multi-turn conversations. The problem with standard GPTCache is that in dialogue the “relevant” query is not just the last message but the whole context window. MeanCache computes the query embedding as the weighted mean of the embeddings of the last $k$ turns:

$$ \mathbf{e}_{\text{query}} = \frac{\sum_{i=1}^{k} w_i \cdot \mathbf{e}_{q_i}}{\sum_{i=1}^{k} w_i} $$

where $w_i$ decreases with the age of the turn. This reduces false positives in dialogues where the topic keeps shifting.

Qdrant as a dual cache store

If the RAG corpus is already in Qdrant, you can use the same instance with a separate collection for the cache. The advantages are operational: a single service to manage, the same backup and monitoring infrastructure.

The cache collection uses payload filters to implement TTL:

from qdrant_client import QdrantClient
from qdrant_client.models import Filter, FieldCondition, Range
import time

client = QdrantClient("localhost", port=6333)

# Search the cache with a TTL filter
hits = client.search(
 collection_name="semantic_cache",
 query_vector=query_embedding,
 query_filter=Filter(
 must=[FieldCondition(
 key="expires_at",
 range=Range(gt=time.time())
 )]
 ),
 limit=1,
 score_threshold=0.93
)

You have to run a periodic cleanup of expired entries, since Qdrant has no native TTL (unlike Redis).

Langfuse for traceability

Langfuse is the OSS standard for LLM pipeline observability (see tracing-llm-otel-genai). Every request should be tagged with whether it was a cache hit or a miss:

from langfuse import Langfuse
from langfuse.decorators import observe

langfuse = Langfuse()

@observe()
def process_query(query: str) -> dict:
 cache_result = semantic_cache.lookup(query)
 if cache_result:
 langfuse.update_current_observation(
 metadata={"cache_hit": True, "cache_score": cache_result.score}
 )
 return cache_result.response
 # full pipeline...
 langfuse.update_current_observation(
 metadata={"cache_hit": False}
 )

With this metadata, Langfuse lets you compute the real hit rate, the distribution of similarity scores, and detect whether the threshold θ needs adjusting.

On-premise hardware: a reference configuration

For an on-premise deployment with this stack, a suitable configuration for RAG with a semantic cache is:

Inference node: 4×H100 SXM (320 GB NVLink in total) for Llama-3.1-70B in FP8. Throughput ~400 tok/s in generation with continuous batching (vLLM or TGI).

Vector services node: CPU with 256 GB RAM. Qdrant for the corpus (1–10 M chunks) and for the cache store (up to 500K entries in memory). Redis 7.x for metadata and an exact-match cache as a first layer.

Embedding node: CPU or a mid-range GPU (A10G). The cache embedder can run on ONNX Runtime on CPU with no perceptible impact on latency (~2 ms per embedding).

Separating the cache store from the corpus matters: the corpus holds millions of chunks with large HNSW indexes; the cache store holds at most tens of thousands of queries with a much smaller index and search times of 1–2 ms.

Cases where the cache fails

The receptionist with a notebook fails in three well-defined scenarios:

1. Queries with a temporal context

“What is the current status of the incident?” or “What changed in the latest release?” are questions whose correct answer changes over time. A cache with a 24-hour TTL could return stale information.

The solution is to detect temporal markers in the query (regular expressions over “today”, “now”, “current”, “latest”, “yesterday”, and their Spanish equivalents) and force a cache miss for these queries, whatever the similarity score.

2. Personalised queries over private data

If the RAG has access to user data (account history, private documents), two different users asking the same question must get different answers. A shared cache that ignores user context is a privacy risk.

The solution is a cache partitioned by user_id or tenant_id. This lowers the hit rate (each user’s cache is smaller) but it is the only safe option in multi-tenant architectures.

3. TTL and a stale corpus

When the corpus is updated (new documents are ingested, errors are corrected), cached answers can go out of date. A fixed TTL (24–48 hours) mitigates the problem but does not remove it.

For corpora with frequent updates, the solution is an active invalidation mechanism: when the corpus is updated in Qdrant, a job runs that identifies which cache entries might be affected (by semantic overlap with the updated chunks) and deletes them. This is the “selective cache invalidation” mentioned in the section on topics not covered.

Integration into the pipeline as middleware

The semantic cache is implemented as middleware between the API gateway and the retriever. It does not change the API contract: the client still sends queries and receives answers in the same format.

class SemanticCacheMiddleware:
 def __init__(self, cache_store, retriever, llm, threshold=0.93):
 self.cache = cache_store
 self.retriever = retriever
 self.llm = llm
 self.threshold = threshold

 async def process(self, query: str, context: dict) -> dict:
 # First layer: exact-match cache (Redis GET, O(1))
 exact = await self.cache.exact_lookup(query)
 if exact:
 return {**exact, "cache_type": "exact"}

 # Second layer: semantic cache (ANN search)
 query_embedding = await self.embed(query)
 semantic = await self.cache.semantic_lookup(
 query_embedding, threshold=self.threshold
 )
 if semantic:
 return {**semantic, "cache_type": "semantic"}

 # Miss: full pipeline
 chunks = await self.retriever.retrieve(query_embedding)
 response = await self.llm.generate(query, chunks)

 # Store for future queries
 await self.cache.store(
 embedding=query_embedding,
 query=query,
 response=response,
 ttl=context.get("ttl", 86400)
 )
 return {**response, "cache_type": "miss"}

The first exact-match layer (Redis GET) is an extra optimisation: for textually identical queries, the embedding is not even computed. The cost is a Redis operation taking microseconds. Only if there is no exact match does it fall through to the semantic lookup.

What we have not covered

Selective cache invalidation: when a subset of the corpus is updated (for example, the documents for a specific product are reindexed), you would need to identify which cache entries overlap semantically with the updated chunks and mark them stale. The mechanism involves computing similarity between the embeddings of the updated chunks and the embeddings of the cached queries, which is expensive at scale.
Multi-tenant cache: isolation vs sharing: in a SaaS product with multiple customers, a shared cache maximises the hit rate but can expose one tenant’s answers to another if filtering is not done correctly. A cache partitioned by tenant is safe but has much lower hit rates. The middle ground is a shared cache with ACL filtering applied over Qdrant’s payload filters.
Semantic cache for streaming responses: when the LLM emits tokens in streaming mode (SSE), the cache cannot easily intercept the complete answer. The options are: cache on the first miss and return the whole answer at once on hits (breaking the streaming experience), or implement a “fake streaming” that emits the tokens of the cached answer at a controlled rate.
Exact-match cache as a first layer: ahead of the semantic cache, an O(1) lookup in Redis with the query as the key can catch textually identical queries at negligible cost. The code in the previous section already shows this two-layer architecture.

References

Bang, J. et al. (2024). MeanCache: User-Centric Semantic Cache for Large Language Model Based Web Applications. arXiv:2403.02694.
Zilliz. (2023). GPTCache: A Library for Creating Semantic Cache for LLM Queries. GitHub: zilliztech/GPTCache.
Qdrant Team. (2024). Qdrant Documentation: Filtering with payload. qdrant.tech/documentation.
Manning, C. D., Raghavan, P., Schütze, H. (2008). Introduction to Information Retrieval. Cambridge University Press. Cap. 19: Web search (distribución Zipf).
Langfuse. (2024). Observability for LLM Applications. langfuse.com/docs.
Meta AI. (2024). Llama 3.1 Model Card. ai.meta.com.
Guo, Y. et al. (2023). Evaluating the Factual Consistency of Large Language Models Through Summarization. Referencia para BERTScore como métrica de evaluación de respuestas cacheadas.

PostgreSQL + Qdrant in RAG ingestion: the postman who keeps two worlds in sync

Thu, 04 Jun 2026 07:00:00 +0200

TL;DR — In a production RAG system, PostgreSQL holds the official truth about documents and Qdrant holds the vectors for search. Keeping them in sync is not trivial: if you delete a document from Postgres and do not invalidate its chunks in Qdrant, the system returns answers from ghost documents. There are two patterns to avoid this: the outbox pattern (atomic transaction plus asynchronous worker, at-least-once) and CDC with Debezium (reading the Postgres WAL directly, low latency, higher complexity). This article explains when to use each one, how to orchestrate them as microservices and what numbers to expect with bge-m3 on on-premise hardware.

The postman and the civil registry

Picture a town with two complementary offices.

The first is the civil registry: it keeps the official census. Every time someone is born, dies or moves house, the registry is the first to know. It is slow, structured, transactional. If the registry says someone exists, they exist. If it says they died, they are dead. PostgreSQL is the civil registry of your documents.

The second is the postman’s notebook: a copy optimised for finding any resident in seconds, organised by area, phonetic names and habitual routes. The postman cannot update the registry, but can search at speeds the registry would never reach. Qdrant is the postman’s notebook.

The problem is synchronisation. If the registry records a death but nobody tells the postman, he will keep trying to deliver letters to an address that no longer exists. In RAG, that translates into indexed chunks of documents that have already been deleted, edited or replaced: ghost documents that pollute the results.

How does the registry tell the postman?

Outbox pattern: every time the registry updates its ledger, it notes the change on an outbox sheet. A messenger clerk reads that sheet periodically and updates the postman’s notebook. Guaranteed, asynchronous, fault tolerant.
CDC with Debezium: the postman has a direct phone line to the registry. Every time the clerk writes something new, the phone rings and the postman updates his notebook in near real time.

The consistency gap problem

In a naive RAG architecture, the flow is:

The user uploads a document → it is inserted into Postgres with metadata.
A worker splits it into chunks, generates embeddings and upserts into Qdrant.
Retrieval uses Qdrant to find relevant chunks and Postgres to hydrate metadata.

So far so good. The problem shows up in updates and deletes:

The user edits a document → Postgres updates the row, but the vectors of the old chunks are still in Qdrant. Retrieval returns stale context.
The user deletes a document → Postgres removes the row, but the chunks remain in Qdrant. Retrieval returns chunks from a document that should no longer exist.
The permission system revokes a tenant’s access → Qdrant has no way of knowing unless there is explicit synchronisation.

This is not a theoretical problem. In living corpora (corporate wikis, knowledge bases updated daily), the consistency gap accumulates noise progressively. An internal study on production pipelines shows that without active reconciliation, 3-8% of indexed chunks correspond to documents that no longer exist in the source of truth after 30 days of operation.

The answer is not “reindex everything every night”. With 10M chunks and a non-trivial embedding model, that costs hours of compute and causes unavailability windows. The answer is change propagation with guarantees.

Outbox pattern: the outbox sheet

Mechanism

The outbox pattern solves the problem of “writing to two systems in the same operation” without distributed transactions (which are expensive and fragile).

The idea is simple: PostgreSQL is the single coordinator. When the ingestion microservice processes a document, it performs two writes in the same local transaction:

It inserts or updates the document in the documents table.
It inserts an event into the outbox_events table.

If the transaction fails, both writes are rolled back. If it succeeds, both are committed atomically. There is no inconsistent intermediate state.

-- Relevant tables
CREATE TABLE documents (
 id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
 tenant_id TEXT NOT NULL,
 title TEXT NOT NULL,
 content TEXT NOT NULL,
 checksum TEXT NOT NULL,
 updated_at TIMESTAMPTZ DEFAULT now()
);

CREATE TABLE outbox_events (
 id BIGSERIAL PRIMARY KEY,
 aggregate_id UUID NOT NULL, -- document id
 event_type TEXT NOT NULL, -- 'document.created' | 'document.updated' | 'document.deleted'
 payload JSONB NOT NULL,
 created_at TIMESTAMPTZ DEFAULT now(),
 processed_at TIMESTAMPTZ -- NULL = pending
);

-- Example of an atomic insert
BEGIN;
 INSERT INTO documents (tenant_id, title, content, checksum)
 VALUES ($1, $2, $3, $4)
 RETURNING id INTO _doc_id;

 INSERT INTO outbox_events (aggregate_id, event_type, payload)
 VALUES (_doc_id, 'document.created', jsonb_build_object(
 'tenant_id', $1,
 'title', $2,
 'checksum', $4
 ));
COMMIT;

The outbox worker

A separate process (the outbox worker) polls outbox_events where processed_at IS NULL, processes each event (chunking, embedding, upsert into Qdrant) and marks the row as processed:

UPDATE outbox_events
 SET processed_at = now()
 WHERE id = $1;

Guarantee: at-least-once. If the worker fails between the Qdrant upsert and the UPDATE, the event will be reprocessed. Qdrant tolerates idempotent upserts (same point id = overwrite), so reprocessing does not create duplicates.

Latency: it depends on the polling interval. With polling every 500ms, p50 latency is ~250ms; p99, ~500ms. Acceptable for most RAG cases, where the user does not expect to see a document indexed in under a second.

CDC with Debezium: the direct phone line

Mechanism

Change Data Capture (CDC) reads PostgreSQL’s Write-Ahead Log (WAL) directly. Postgres writes every change to the WAL before applying it to the tables; it is the mechanism it uses for replication and recovery. Debezium subscribes to a logical replication slot and turns those events into structured messages.

-- Enable logical replication in postgresql.conf
-- wal_level = logical

-- Create a replication slot for Debezium
SELECT pg_create_logical_replication_slot('debezium_slot', 'pgoutput');

The full flow:

Postgres WAL → Debezium connector → Kafka/NATS → Indexer consumer → Qdrant

Debezium emits events with the record’s before/after structure:

{
 "op": "d",
 "before": {
 "id": "550e8400-e29b-41d4-a716-446655440000",
 "tenant_id": "acme",
 "checksum": "sha256:abc123"
 },
 "after": null
}

With "op": "d" (delete), the consumer knows it must delete every point in Qdrant whose payload contains that document_id.

# Consumer: delete by payload filter
qdrant_client.delete(
 collection_name="corpus",
 points_selector=FilterSelector(
 filter=Filter(
 must=[
 FieldCondition(
 key="document_id",
 match=MatchValue(value=event["before"]["id"])
 )
 ]
 )
 )
)

Upsides and downsides

CDC removes polling and cuts latency to tens of milliseconds (WAL propagation time plus consumer processing). But it adds operational complexity: you need to manage the replication slot (unconsumed slots retain WAL indefinitely, which can fill the disk), the message broker and the consumer offset state.

Comparison: outbox vs CDC

Criterion	Outbox pattern	CDC with Debezium
Typical latency	250ms – 2s	20ms – 200ms
Delivery guarantee	At-least-once	At-least-once
Operational complexity	Low (Postgres only)	High (Debezium + broker)
WAL retention risk	None	High if the slot stalls
Idempotency required	Yes (in the indexer)	Yes (in the consumer)
Multi-table support	Manual	Automatic (any table)
Backpressure	Natural (polling)	Needs explicit design
When to choose it	Corpus < 100k docs/day, small team	Corpus > 1M docs/day, low latency critical

Rule of thumb: start with outbox. Move to CDC when the change volume goes past ~50k events/hour or when latency measured in seconds becomes unacceptable for the use case (for example, real-time news indexing).

Microservice architecture

The ingestion pipeline is made up of three microservices with cleanly separated responsibilities:

Microservice 1: Ingestor

Responsibilities: receive documents, split them into chunks and request embeddings. It does not write to Qdrant directly.

# ingestor/main.py (simplified)
from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
 chunk_size=512, # tokens, not characters
 chunk_overlap=64,
 length_function=token_count,
)

def ingest_document(doc: Document, db: Session) -> None:
 chunks = splitter.split_text(doc.content)
 with db.begin():
 db.execute(
 "UPDATE documents SET checksum=$1 WHERE id=$2",
 [doc.checksum, doc.id]
 )
 db.execute(
 """INSERT INTO outbox_events (aggregate_id, event_type, payload)
 VALUES ($1, 'document.updated', $2)""",
 [doc.id, {"chunks": chunks, "tenant_id": doc.tenant_id,
 "model": "bge-m3", "model_version": "1.0.0"}]
 )

Microservice 2: Indexer

It reads the outbox, generates embeddings by calling the inference server (vLLM or TEI) and upserts into Qdrant.

# indexer/worker.py
import asyncio
from qdrant_client import QdrantClient
from qdrant_client.models import PointStruct, VectorParams, Distance

qdrant = QdrantClient(host="qdrant-service", port=6333)

async def process_event(event: dict) -> None:
 payload = event["payload"]
 chunks = payload["chunks"]
 doc_id = str(event["aggregate_id"])

 # Batch embedding on TEI (Text Embeddings Inference)
 embeddings = await embed_batch(chunks, model="bge-m3")

 points = [
 PointStruct(
 id=f"{doc_id}_{i}",
 vector=emb,
 payload={
 "document_id": doc_id,
 "tenant_id": payload["tenant_id"],
 "chunk_index": i,
 "text": chunks[i],
 "model_version": payload["model_version"],
 }
 )
 for i, emb in enumerate(embeddings)
 ]

 # If it is an update, delete the previous chunks first
 if event["event_type"] in ("document.updated", "document.deleted"):
 qdrant.delete(
 collection_name="corpus",
 points_selector=filter_by_doc_id(doc_id)
 )

 if event["event_type"] != "document.deleted":
 qdrant.upsert(collection_name="corpus", points=points)

Microservice 3: Reconciler

The reconciler is the safety net. Periodically (every hour, for example) it compares the set of document_id values in Postgres with the set of document_id values in Qdrant. IDs present in Qdrant but absent from Postgres are ghosts: they get deleted.

# reconciler/diff.py
async def reconcile(tenant_id: str) -> int:
 pg_ids = set(await fetch_all_doc_ids(tenant_id))
 qdrant_ids = set(await scroll_all_doc_ids(tenant_id)) # paginated scroll

 orphans = qdrant_ids - pg_ids
 if orphans:
 logger.warning("Orphan chunks for %d documents", len(orphans))
 for doc_id in orphans:
 qdrant.delete("corpus", filter_by_doc_id(doc_id))

 return len(orphans)

Throughput and ingestion maths

Embedding throughput

The bge-m3 model (1024 dimensions, dense + sparse + colbert support) on a node with 4×H100 SXM (320 GB NVLink) running via vLLM or HuggingFace TEI reaches roughly 2,000 chunks/second with batch size = 256 and 512-token sequences.

$$\text{throughput} = 4 \times 500 \text{ chunks/s/GPU} = 2{,}000 \text{ chunks/s}$$

The figure of 500 chunks/s per GPU comes from public TEI benchmarks with bge-m3 on H100 SXM5, batch=256, seq_len=512 ¹.

Total re-ingestion time

For a corpus of 10M chunks:

$$t = \frac{10{,}000{,}000 \text{ chunks}}{2{,}000 \text{ chunks/s}} = 5{,}000 \text{ s} \approx 83 \text{ minutes}$$

That is pure embedding time. Adding Qdrant write latency (~0.5ms per upsert batch of 100 points):

$$t_{\text{qdrant}} = \frac{10{,}000{,}000}{100} \times 0.5\text{ ms} = 50{,}000 \text{ ms} = 50 \text{ s}$$

Estimated total for a full re-ingestion: ~85-90 minutes on a 4×H100 node.

Storage cost in Qdrant

Each bge-m3 vector has 1024 dimensions in float32 (4 bytes):

$$\text{size per vector} = 1{,}024 \times 4 \text{ B} = 4{,}096 \text{ B} = 4 \text{ KB}$$

For 10M chunks (dense vectors only):

$$\text{total vectors} = 10^7 \times 4{,}096 \text{ B} = 40.96 \text{ GB}$$

Adding the JSON payload (estimated at ~500 bytes/chunk):

$$\text{payload} = 10^7 \times 500 \text{ B} = 5 \text{ GB}$$

HNSW index (roughly 1.2× the vector size for $m=16$):

$$\text{HNSW} \approx 40.96 \text{ GB} \times 1.2 = 49.15 \text{ GB}$$

Estimated total on disk: ~95 GB for 10M chunks with dense bge-m3.

With scalar quantization (int8), the vector size shrinks 4×:

$$\text{with int8 quantization} \approx \frac{40.96}{4} + 5 + \frac{49.15}{4} \approx 27.5 \text{ GB}$$

Configuration	Vectors	HNSW	Payload	Total
float32, no quantization	40.96 GB	49.15 GB	5 GB	~95 GB
int8 scalar quantization	10.24 GB	12.29 GB	5 GB	~28 GB
binary quantization	1.28 GB	1.54 GB	5 GB	~8 GB

Binary quantization loses recall precision (~2-5% in NDCG@10), but it lets you host much larger corpora in RAM. For production where recall is critical, int8 is the usual balance point.

Recommended on-premise hardware

For a continuous ingestion pipeline in production:

Embedding node: 4×H100 SXM (320 GB, NVLink), 2× 64-core CPU (EPYC 9654), 1 TB DDR5 RAM, 100 GbE. Runs vLLM or TEI serving bge-m3. Sustained throughput: ~2,000 chunks/s with an asynchronous batch pipeline.

Qdrant node: 32-core CPU, 256 GB RAM (to keep the HNSW index in memory with 10M chunks and no quantization), 2 TB NVMe (Qdrant snapshot and WAL writes). Qdrant recommends that the HNSW index fit in RAM for p99 latency < 5ms.

PostgreSQL node: 16-core CPU, 128 GB RAM, 4 TB NVMe for the WAL (especially relevant if you use CDC with a logical replication slot; the slot retains WAL until Debezium consumes it).

Broker (if CDC): 3-broker Kafka with 500 GB NVMe per node, or NATS JetStream with 3 nodes for more modest loads.

Kubernetes manifests

Indexer Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
 name: rag-indexer
 namespace: rag-pipeline
spec:
 replicas: 3
 selector:
 matchLabels:
 app: rag-indexer
 template:
 metadata:
 labels:
 app: rag-indexer
 spec:
 containers:
 - name: indexer
 image: registry.example.com/rag-indexer:1.0.0
 env:
 - name: POSTGRES_DSN
 valueFrom:
 secretKeyRef:
 name: pg-credentials
 key: dsn
 - name: QDRANT_HOST
 value: "qdrant-service.qdrant.svc.cluster.local"
 - name: EMBEDDING_ENDPOINT
 value: "http://tei-service.embeddings.svc.cluster.local:8080"
 - name: POLL_INTERVAL_MS
 value: "500"
 resources:
 requests:
 memory: "512Mi"
 cpu: "500m"
 limits:
 memory: "2Gi"
 cpu: "2"

Reconciler CronJob

apiVersion: batch/v1
kind: CronJob
metadata:
 name: rag-reconciler
 namespace: rag-pipeline
spec:
 schedule: "0 * * * *" # hourly
 concurrencyPolicy: Forbid
 jobTemplate:
 spec:
 template:
 spec:
 restartPolicy: OnFailure
 containers:
 - name: reconciler
 image: registry.example.com/rag-reconciler:1.0.0
 env:
 - name: POSTGRES_DSN
 valueFrom:
 secretKeyRef:
 name: pg-credentials
 key: dsn
 - name: QDRANT_HOST
 value: "qdrant-service.qdrant.svc.cluster.local"
 - name: SCROLL_PAGE_SIZE
 value: "1000"
 resources:
 requests:
 memory: "256Mi"
 cpu: "250m"
 limits:
 memory: "1Gi"
 cpu: "1"

Production gotchas

Reindexing when the embedding model changes

This is the most painful problem. If you move from bge-m3 to nomic-embed-text-v2, the vectors are incompatible: they live in different embedding spaces and the cosine distances between them are meaningless.

The answer is dual-index aliasing:

Create a new collection in Qdrant: corpus_v2.
Re-embed the whole corpus with the new model and load corpus_v2.
When the new collection is complete and validated (recall test), switch the corpus_prod alias from corpus_v1 to corpus_v2.
Delete corpus_v1 once traffic has migrated.

During the migration the two indexes coexist. The retriever uses the alias, not the collection’s direct name.

Index versioning

Store model_version in the payload of every point in Qdrant. This lets you:

Filter by version during retrieval (useful in model A/B testing).
Have the reconciler detect points with an old version and reprocess them selectively.
Audit: know which model generated each embedding.

# Filter by model version at retrieval time
results = qdrant.search(
 collection_name="corpus",
 query_vector=query_embedding,
 query_filter=Filter(
 must=[
 FieldCondition(key="model_version", match=MatchValue(value="1.0.0")),
 FieldCondition(key="tenant_id", match=MatchValue(value=tenant_id)),
 ]
 ),
 limit=10
)

Namespace per tenant (multi-tenancy)

There are two strategies in Qdrant:

Strategy	Pros	Cons
Collection per tenant	Total isolation, no extra filter	N collections = N HNSW indexes in RAM
Payload filter per tenant	A single collection, less RAM	The filter adds ~10-15% search latency

For fewer than 100 tenants with large corpora (> 1M chunks/tenant), use a collection per tenant. For hundreds or thousands of tenants with small corpora, use a payload filter with tenant_id indexed:

qdrant.create_payload_index(
 collection_name="corpus",
 field_name="tenant_id",
 field_schema=PayloadSchemaType.KEYWORD
)

What we have not covered

Streaming corpus updates with near-real-time CDC: selective chunk invalidation when only one section of a document changes (incremental chunking based on content diff, not full re-chunking).
Multi-tenant corpus isolation with per-chunk ACLs: going beyond the tenant_id filter towards group, role or even individual document permissions applied at retrieval time.
Federated corpus: corpora distributed across cross-border silos where regulation (GDPR, CCPA) prevents centralising the embeddings; federated search patterns without moving data.
Zero-downtime incremental reindexing using dual-index aliasing: the full model migration protocol with rollback, recall regression testing and progressive traffic splitting.

References

HuggingFace Text Embeddings Inference — benchmarks oficiales con modelos de la familia bge en hardware A100/H100. https://github.com/huggingface/text-embeddings-inference ↩︎

Embeddings in 2026: the three families (dense, sparse, multi-vector), the model zoo and the decision that matters in production

Wed, 03 Jun 2026 03:00:00 +0200

This post opens the data sub-saga inside the six-stage LLMOps pipeline by going into the piece that holds up three-layer retrieval: the embedder. If the librarian of curation decided what enters the index and the faculty committee decided what reaches the model’s face, this post looks at the one in the middle: the cartographer who draws the map you search over.

TL;DR

The conversation about embeddings has been simplified in production to the point where “which embedder do you use” always gets the same answer: OpenAI text-embedding-3-large in the demo, bge-m3 in the “sovereign-ready” version. That simplification hides the fact that an embedder is three different models at once, dense single-vector, learned sparse (SPLADE) and multi-vector late-interaction (ColBERT), and that in 2026 the leading models do not compete within the same family: gte-Qwen2-7B-instruct and NV-Embed-v2 break MTEB in dense single-vector, SPLADE-v3 and the sparse head of bge-m3 dominate the learned lexical descriptor, Jina-ColBERT-v2 and ColNomic-7B are the strongest in multilingual multi-vector, and Snowflake Arctic Embed L 2.0 has slipped in as the small multilingual favourite with decent Matryoshka. This post takes apart the three families with their maths (InfoNCE with τ, MaxSim, SPLADE’s FLOPS regularization, Matryoshka Representation Learning), goes through the open source model zoo with the dimension, licence and niche of each one, poses the specific problem of multilingual Spanish, which cuts the list of viable models to fewer than six, gives the real storage cost per million chunks with int8 / binary / TurboQuant, describes how they are served on-premise with TEI, Infinity and vLLM --task embed, sets the minimum hardware at an RTX 4090 and the good hardware at a 4×H100 cluster, lists the seven operational traps that sink quality without warning (corpus drift, forgotten normalisation, badly chosen Matryoshka dimension, missing hard negatives, chat template slipped into the embedder, tokenizer drift, MTEB overfit) and closes with a license-clean stack for sovereign production.

The analogy: three librarians cataloguing the same book

A technical library receives a new book and, before putting it on the shelves, has to generate a searchable identifier for it. Three librarians with three different trades live in the library, and all three catalogue the same book at once:

Librarian A, the thematic one. He reads the whole book and gives it a single rich RFID tag. That tag is a vector of 1,024 numbers where each coordinate encodes a latent semantic axis (one the librarian never verbalises: he has learned it by reading a hundred million previous books). Two books about Kubernetes in production will end up with RFID tags very close together in the space even if one talks about Linkerd and the other about Cilium, because they share thematic axes. To search, you compare the question’s tag with the book’s and return the nearest ones by cosine. It is fast, it scales to millions, and it loses fine nuance. This is the dense single-vector embedder: bge-m3 in dense mode, gte-Qwen2-7B-instruct, Snowflake Arctic Embed L 2.0, multilingual-e5-large-instruct.

Librarian B, the lexical one. He summarises nothing. What he does is write on a card the weighted list of relevant terms from the book, expanded with synonyms from the field. The Kubernetes book carries on its card “kubernetes 4.2, linkerd 3.8, cilium 3.7, service-mesh 4.1, sidecar 3.2, mtls 2.9, ebpf 2.6, k8s 4.0…”, each term with a weight. The trick is that the model does the expansion: if your question says “service mesh” and the original book only said “Linkerd”, librarian B’s card did record “service-mesh 4.1” because he understood the relationship. To search, you intersect the question’s card with the book’s the old-fashioned way: inverted index, posting lists. It is decisive when the reader writes few, very specific words (product names, errors, jargon). This is the learned sparse embedder: SPLADE-v3 or the sparse head of bge-m3. It is the modern successor to BM25, not its rival; we will see why.

Librarian C, the copyist. He gives up on summarising. He takes every word on every page of the book and gives it a 128-dimension mini-RFID. He ends up with a 30,000-token book turned into 30,000 mini-RFIDs. When you search for something, the librarian compares every word of your question with every word of the book and keeps the maximum for each word of the question, adding them up. It captures nuance the other two lose by construction (proper nouns, numbers, specific phrasings), but its filing system is an order of magnitude larger in disk space. This is ColBERT-v2 / Jina-ColBERT-v2 / ColNomic-7B: late interaction, MaxSim.

And then there is the all-rounder librarian who does all three jobs in a single pass: bge-m3. A single 568 M parameter model that simultaneously returns the thematic RFID tag (1,024-d), the weighted list of terms (sparse) and the set of per-token mini-RFIDs (128-d). That is why it has settled in as the standard embedder for multilingual on-premise RAG: a single model.encode(chunk) produces the three outputs that feed hybrid retrieval without orchestrating three different models.

The same chunk indexed by three librarians. The all-rounder bge-m3 runs all three in a single 568 M parameter pass.

What an embedding really is

A text embedding is a function f : text → ℝᵈ trained so that two “semantically similar” texts produce nearby vectors in that space. The important part is not “vector”, any fixed-length hash is one, but what closeness means. Closeness is defined implicitly by the loss the model is trained with.

Almost every modern embedder is trained with InfoNCE (also called Multiple-Negatives Ranking Loss in sentence-transformers):

\mathcal{L}_{\text{InfoNCE}} = -\log \frac{\exp(\text{sim}(q, d^+)/\tau)}{\sum_{d \in \mathcal{B}} \exp(\text{sim}(q, d)/\tau)}

For each (query, doc⁺) pair the model has to assign the positive more similarity than all the other documents in the batch B, where the other documents act as free in-batch negatives. The temperature τ (typically 0.02–0.07, almost always 0.05) controls how sharp the distribution gets: lower τ means the model becomes more demanding about the positive but less stable. A large batch size |B| means many more negatives per gradient, so a better informed model. That is why embedders are trained with batch ≥ 1,024 on H100 clusters with AllGather across nodes to stack every negative in the cluster into a single effective batch.

On top of the in-batch negatives comes hard negatives mining: documents deliberately picked because they are “almost in the answer” (typically the next 10-100 neighbours from a previous BM25 / dense retrieval). Without hard negatives the model learns to discriminate the trivial and real quality on BEIR / MTEB drops 5-10 points.

This matters because the embedder family depends on what you feed into sim(·,·):

Family	`sim(q, d)`	Model output
Dense single-vector	dot product of two normalised 1024-d vectors	`f(q), f(d) ∈ ℝ^d`
Learned sparse	dot product of two 30,522-d vectors (≈80 non-zero each)	`f(q), f(d) ∈ ℝ^V`, `V =
Multi-vector	`Σᵢ maxⱼ ⟨qᵢ, dⱼ⟩` (MaxSim)	`f(q) ∈ ℝ^{

The loss is the same in all three cases, InfoNCE, but the geometry of the space changes, and with it quality, storage cost and search latency.

The three families in detail

Dense single-vector, the cartographer

The embedder reads the full chunk, passes it through an encoder transformer (XLM-RoBERTa, BERT, a Mistral decoder with a prompt) and aggregates the token representations into a single vector by:

CLS pooling: uses the embedding of the [CLS] token. Standard in BERT-base.
Mean pooling: plain average of the embeddings of all tokens. Standard in multilingual-e5-large-instruct, bge-m3 dense, Snowflake Arctic Embed L 2.0.
Last-token pooling: for decoder-LLM based embedders (e5-mistral-7b-instruct, gte-Qwen2-7B-instruct, NV-Embed-v2) that take the final token as the aggregate. It works because the model is causal and the last token “has seen” the whole context.
Latent-attention pooling: new in NV-Embed-v2. A learnable attention layer that weights tokens instead of averaging them. +2-3 MTEB points over mean pooling.

After pooling the vector is normalised to norm 1: v ← v / ‖v‖₂. With normalised vectors, cosine and dot product coincide:

\cos(q, d) = \frac{q \cdot d}{\|q\|\|d\|} = q \cdot d, \qquad \|q - d\|^2 = 2 - 2\,(q \cdot d)

That is why almost every vector DB indexes by inner product and leaves it to the user to normalise beforehand (Qdrant, Faiss IP, Milvus). If you forget to normalise the query vector but the corpus vectors are normalised, retrieval degrades silently: the query’s magnitude distorts the ranking. It is the number one bug in RAG in production.

A minimal numerical example, two normalised 4-d vectors:

q = [0.5, 0.5, 0.5, 0.5] ‖q‖ = 1
d₁ = [0.6, 0.4, 0.5, 0.5] ‖d₁‖ = 1.005, normalised [0.597, 0.398, 0.498, 0.498]
q · d₁ = 0.5·0.597 + 0.5·0.398 + 0.5·0.498 + 0.5·0.498 = 0.995

Closeness of nearly 1, as expected. If d₁ had not been normalised, q · d₁ = 1.0 and it would appear closer than any d that is perfectly aligned but with a norm < 1.005. Normalisation is not a detail: it is the contract of the space.

Learned sparse, the lexical descriptor

SPLADE-v3 (Naver, March 2024) has consolidated the modern version of the lexical librarian. Internally it is a small BERT (~110 M parameters, DistilBERT/BERT base) that produces, for each input token, a distribution over the whole vocabulary (30,522 dimensions in BERT WordPiece), and then does max-pool over the tokens:

w_j = \max_{i \in \text{seq}} \log\bigl(1 + \text{ReLU}(W_{ij})\bigr)

where Wᵢⱼ is the logit of input token i for vocabulary term j. The log(1+ReLU) saturates high logits (it stops a single word from dominating the vector) and the ReLU cuts the negatives. The result is a 30,522-dimension vector of which typically 50-200 entries stay non-zero.

The key part is the FLOPS regularization added to the loss during training:

\mathcal{L}_{\text{FLOPS}} = \lambda \cdot \sum_{j=1}^{V} \bar{w}_j^2, \qquad \bar{w}_j = \frac{1}{|B|}\sum_{i \in B} w_{ij}

It penalises the expected cost of the posting lists: if a vocabulary word appears on average in many documents, adding it to a new document costs double. The model learns to generate sparse vectors by construction.

What does this do to the text “Linkerd 3.8 brings mTLS by default”? The model does not only write the literal terms, it also writes, with a smaller but non-zero weight, “service-mesh”, “kubernetes”, “tls”, “sidecar”, “envoy”, “istio” (its competitor, also semantically related), “encryption”, “k8s”. That semantic expansion of the document is what sets SPLADE apart from BM25. BM25 only knows what was literally in the text; SPLADE knows what an expert would add as a descriptor.

In practice SPLADE-v3 beats BM25 by 3-6 MRR@10 points on MS MARCO and dominates BEIR zero-shot. The cost is ~2-4× the query latency of BM25 over the same inverted index, which static pruning can mitigate.

For the multilingual case, bge-m3 with its sparse head is the only maintainable option: SPLADE-v3 is trained in English and the multilingual ports are in an experimental state.

Multi-vector, the copyist

ColBERT-v2 (Stanford, NAACL 2022) introduced the late interaction paradigm. Instead of compressing the document to a single vector, it leaves it as a matrix (|d|, k) with a k-dimension vector per token. The similarity between query and document is computed token by token and aggregated with MaxSim:

s(q, d) = \sum_{i \in q} \max_{j \in d} \langle q_i, d_j \rangle

What is being computed: for each word of the query, find the document token that fits it best and add that similarity. The sum is over the query, not over the document. This lets a 30,000-token document compete fairly with a 200-token one, because the query always adds |q| terms.

Why does it give more quality than dense single-vector? Because compressing to one vector loses information about where each idea was. If the query is “which version brought mTLS by default in Linkerd”, the dense summary of the document only knows that the chunk is about “Linkerd and mTLS”; ColBERT’s copyist can match “which version” with “3.8” because it stores the embedding of the 3.8 token separately. On BEIR / out-of-domain, late interaction beats single-vector by between +2 and +6 nDCG@10 with the same backbone.

The price is storage. Per document, a 1024-dimension dense single-vector in fp16 takes 1024 × 2 = 2 KB. ColBERT-v2 with 128-dimension tokens for a 256-token chunk takes 256 × 128 × 2 = 65,536 B ≈ 64 KB: 32× more space. With ColBERT-v2’s nbits=2 residual compression it drops to ~16 KB (8×). Jina-ColBERT-v2 adds Matryoshka over the token dimensions (truncatable to 128 / 96 / 64), cutting another 50%.

For 1 million chunks:

Family	Per doc	Total
Dense fp32 (1024-d)	4,096 B	4.0 GB
Dense fp16 / halfvec	2,048 B	2.0 GB
Dense int8 (SQ)	1,024 B	1.0 GB
Dense binary (1 bit/d)	128 B	128 MB
SPLADE (≈ 80 terms × 8 B)	~640 B	~640 MB
ColBERT fp16 (256 tok × 128 d)	65,536 B	64 GB
ColBERT residual `nbits=2`	~16,000 B	~16 GB
Jina-ColBERT-v2 (MRL token 64)	~8,000 B	~8 GB

ColBERT in on-premise production is reserved for corpora of up to a few million chunks, or applied only as a reranker over the top-100 from the first committee (dense + sparse), as described in the reranker post.

Matryoshka, the truncatable dimension

An operational lever that changed the conversation about embeddings between 2024 and 2026 is Matryoshka Representation Learning (Kusupati et al., NeurIPS 2022). The trick: during training, on top of the loss over the full D-dimension vector, the same loss is computed over prefixes of the vector:

\mathcal{L}_{\text{MRL}} = \sum_{k \in \{64, 128, 256, 512, 1024\}} \alpha_k \cdot \mathcal{L}_{\text{InfoNCE}}\bigl(\text{emb}[:k]\bigr)

The first k dimensions of the embedding are trained to be, on their own, a valid embedding. At inference time, if you want a cheaper embedding, you truncate the vector: the first quarter is already a usable embedding. Without Matryoshka, truncating wrecks the geometry: the first 256 dimensions of an embedding trained only at 1024-d encode nothing coherent.

Typical degradation in MTEB nDCG@10 when truncating an MRL embedder:

Truncation	Average loss
1024 → 512	-1 to -2 points
1024 → 256	-3 to -5 points
1024 → 128	-5 to -8 points
1024 → 64	-8 to -12 points

Native MRL models in 2026 (the ones that let you pick the dimension at runtime without retraining):

jina-embeddings-v3 (1024 → 32, fine steps, CC-BY-NC-4.0)
jina-embeddings-v4 (2048 → 128, multimodal text+image, CC-BY-NC-4.0)
nomic-embed-text-v2-moe (768 → 256, Apache 2.0)
Snowflake-arctic-embed-l-v2.0 (1024 → 256, Apache 2.0)
mxbai-embed-large-v1 and mxbai-embed-2d-large-v1 (the latter also truncatable in layer depth)
Stella_en_1.5B_v5 (multiple steps 512 / 768 / 1024 / 2048 / 4096 / 6144 / 8192, MIT, English only)
text-embedding-3-large (OpenAI, 3072 → 256, API only)
voyage-3 family (1024 → 256 / 512 / 1024 / 2048, API only)

Production recommendation: always use an MRL model even if you do not truncate at the start, because it simplifies the future quantisation decision. And above all, evaluate truncation on your real corpus: the average MTEB degradation of “-3 points” becomes “-12 points” in a niche domain.

The 2026 open source model zoo

What follows is a spec sheet per model. Verified as of 2026-06: HuggingFace cards + reference papers + MTEB / MMTEB leaderboard.

Dense single-vector

Model	Params	Dim	Tokens	Languages	Licence	Distinctive
`BAAI/bge-m3` (dense)	568 M	1024	8192	100+	MIT	Tri-mode (dense + sparse + colbert) in one forward. De facto standard for multilingual on-prem.
`Snowflake/snowflake-arctic-embed-l-v2.0`	568 M	1024 (MRL → 256)	8192	~100	Apache 2.0	Trained for multilingual + English without degrading either. MIRACL 55.8.
`intfloat/multilingual-e5-large-instruct`	560 M	1024	512	~100	MIT	Veteran multilingual baseline. Short window.
`intfloat/e5-mistral-7b-instruct`	7.1 B	4096	4096	English	MIT	First decoder-as-embedder to break MTEB. English.
`Alibaba-NLP/gte-Qwen2-7B-instruct`	7 B	3584	32,768	100+	Apache 2.0	The only one with 32k context. Top MTEB-en, strong multilingual.
`nvidia/NV-Embed-v2`	7.85 B	4096	32,768	English	CC-BY-NC-4.0	Latent-attention pooling. Top quality. Licence blocker in prod.
`Linq-AI-Research/Linq-Embed-Mistral`	7 B	4096	4096	English	CC-BY-NC-4.0	Top MTEB retrieval Aug 2024. Non-commercial.
`NovaSearch/stella_en_1.5B_v5`	1.54 B	8192 (multiple MRL)	8192	English	MIT	Small + rich MRL. English.
`BAAI/bge-multilingual-gemma2`	9 B	3584	8192	100+	Gemma	High quality, the Gemma licence restricts redistribution.
`BAAI/bge-en-icl`	7 B	4096	8192	English	MIT-style	In-context learning from examples in the prompt.
`mixedbread-ai/mxbai-embed-large-v1`	335 M	1024 (MRL)	512	English	Apache 2.0	MRL + native binary. Short window.
`jinaai/jina-embeddings-v3`	570 M	1024 (MRL → 32)	8192	89	CC-BY-NC-4.0	Per-task LoRA. Non-commercial without a licence.
`nomic-ai/nomic-embed-text-v2-moe`	475 M / 305 M active	768 (MRL → 256)	512	~100	Apache 2.0	First general-purpose MoE in embeddings.

Learned sparse

Model	Params	Vocab	Tokens	Languages	Licence	Distinctive
`naver/splade-v3`	110 M	30,522	512	English	CC-BY-NC-SA-4.0	SOTA learned sparse. Non-commercial.
`BAAI/bge-m3` (sparse head)	568 M	XLM-R vocab	8192	100+	MIT	The only license-clean multilingual option.

Multi-vector (late interaction)

Model	Params	Dim/token	Tokens	Languages	Licence	Distinctive
`colbert-ir/colbertv2.0`	110 M	128	512	English	MIT	The original paper, the basis for everything.
`jinaai/jina-colbert-v2`	560 M	128 / 96 / 64 (MRL)	8192	89	Apache 2.0	The license-clean multilingual multi-vector.
`nomic-ai/colnomic-embed-multimodal-7b`	7 B	3584 (Qwen2-VL)	—	~100	Apache 2.0	Multimodal text+image multi-vector. Vidore-v2 open SOTA.

The leaderboard, with caution

MTEB / MMTEB (Massive Multilingual Text Embedding Benchmark, Enevoldsen et al., arxiv 2502.13595) is the standard thermometer. Top retrieval on MMTEB in mid-2026 is dominated by Qwen3-Embedding-8B (~70.6 multilingual avg) and Llama-Embed-Nemotron-8B. Below them, the 7B models (gte-Qwen2-7B, NV-Embed-v2) and the 568M ones (bge-m3, Snowflake-Arctic-L-2.0) compete task by task.

Trap: MTEB has started to saturate through dataset contamination. The higher the ranking, the more likely it is that the model saw subsets of the evaluation datasets during training. The rule in production: the leaderboard is for discarding bad models, not for picking the best one. The final decision is made on your own domain eval set, generated with the LLM-as-judge or evals recipe.

The specific problem of multilingual Spanish

For a Spanish customer serving corporate, legal or support documentation in Spanish (and often with Catalan / Portuguese / English mixed in), the embedder zoo narrows to fewer than six viable models. The operational exclusions:

English-only models: e5-mistral-7b-instruct, stella-en-1.5B-v5, Linq-Embed-Mistral, mxbai-embed-large-v1, NV-Embed-v2, SPLADE-v3. They cut Spanish performance below an acceptable level: translating the query into English before searching is one route, but it introduces latency, tokenisation drift and another model dependency.
Models with a non-commercial licence: jina-embeddings-v3, jina-embeddings-v4, NV-Embed-v2, Linq-Embed-Mistral. They work for a PoC, but commercial production needs an explicit agreement with the vendor. Unless you have the licence signed, they have to be excluded.
Models with a Gemma licence: bge-multilingual-gemma2. Allowed for internal use, awkward to redistribute weights to a customer.

The ones left, ordered by practical choice in sovereign production:

BAAI/bge-m3 — MIT, 568 M, 100+ languages (including Spanish and Catalan, explicitly trained), 8,192 tokens, tri-mode dense+sparse+colbert. A reasonable default. Fits in an RTX 4090. TEI and Infinity serve it natively.
Snowflake/snowflake-arctic-embed-l-v2.0 — Apache 2.0, same size, explicit Matryoshka, better MIRACL/CLEF than bge-m3 on some multilingual tasks, no sparse head. If the priority is pure MMTEB in Spanish.
intfloat/multilingual-e5-large-instruct — MIT, 560 M, veteran baseline. Its 512-token window is the big limitation: long documents have to be split first. If what you already have in production works, do not migrate out of fashion.
Alibaba-NLP/gte-Qwen2-7B-instruct — Apache 2.0, 32k context, high quality in Spanish (Qwen2 is well trained on Spanish). If the chunks are long (more than 4k tokens) and you have the GPU to serve it (it does not fit in a 4090; it does in an H100). It fits alongside an LLM on an H100 80GB with care.
nomic-ai/nomic-embed-text-v2-moe — Apache 2.0, 305 M active, MRL, ~100 languages. If latency and cost per token rule: the MoE gives it disproportionate throughput for its quality.
jinaai/jina-colbert-v2 — Apache 2.0, multilingual multi-vector, as a reranker or as the main retrieval on a small corpus (< 1 M chunks). The only license-clean multi-vector in Spanish.

The rule of thumb: bge-m3 as dense+sparse on the front line, jina-colbert-v2 as a third reranking layer when the use case warrants it, and Snowflake Arctic L 2.0 as the alternative if the corpus-specific eval prefers its geometry.

Serving embeddings on-premise

Three engines split the on-prem embedding serving landscape in 2026, with different profiles.

Text Embeddings Inference (TEI), the standard

huggingface/text-embeddings-inference is a server written in Rust with a Candle / ONNX backend, integrated FlashAttention and dynamic batching by tokens. It exposes an OpenAI-compatible /v1/embeddings API and supports the three bge-m3 modes simultaneously from version 1.5 onwards. For multilingual production it is the obvious default.

# values.yaml — TEI serving multilingual bge-m3 on an RTX 4090
image: ghcr.io/huggingface/text-embeddings-inference:1.5
args:
 - --model-id=BAAI/bge-m3
 - --pooling=cls
 - --max-batch-tokens=16384
 - --max-concurrent-requests=512
 - --dtype=float16
resources:
 limits:
 nvidia.com/gpu: 1

Indicative throughput with bge-m3, fp16, 512-token sequence, batch 32:

RTX 4090 (24 GB): ~8–15 k tokens/s
A100 80 GB: ~60 k tokens/s sustained
H100 80 GB: ~40–80 k tokens/s, with fp8 about 50% more

(The ranges are approximate and depend on the real batch, the average sequence length and compilation with FA2/FA3.)

Infinity, the flexible one

michaelfeil/infinity (MIT) is a multi-model FastAPI server able to load bge-m3, Snowflake Arctic, Jina-v3, Nomic, ColPali, CLAP and rerankers simultaneously behind the same OpenAI-style API. Backend PyTorch + Optimum (ONNX/TensorRT) or CTranslate2. Useful when you need to serve several different embedders (one for text, another for code, another for images) behind a single endpoint, or when the model does not yet have TEI support.

vLLM `--task embed`, for the 7B embedders

When the embedder is really an LLM decoder turned into an embedder (e5-mistral-7b-instruct, gte-Qwen2-7B-instruct, NV-Embed-v2, Stella-1.5B), the natural place to serve it is vLLM, which already has the PagedAttention and continuous batching stack in production:

vllm serve Alibaba-NLP/gte-Qwen2-7B-instruct \
 --task embed \
 --dtype bfloat16 \
 --max-model-len 32768 \
 --trust-remote-code

vLLM detects the correct pooling (last-token on the Qwen / Mistral based ones) and exposes an OpenAI-compatible /v1/embeddings. For inference clusters already running vLLM with an LLM on another port, it is the natural way to serve the embedder without standing up another stack.

fastembed, the lightweight one

qdrant/fastembed loads bge-small, MiniLM, ColBERT and BM25/SPLADE sparse in ONNX-CPU. It is not competitive on throughput against TEI/Infinity with a GPU, but it is the right option when you have to serve embeddings on a NUC node without a GPU (see mixed NVIDIA + Intel environments) or when the embedder is part of the client (a preview in a UI, prior scoring at the edge).

Storage, quantisation and the corpus arithmetic

The embedding does not stay in the embedder’s memory: it lives in the vector DB index and is materialised every time you ingest a new chunk. The storage cost calculation is what decides the final dimension, not MTEB quality. For 1 million chunks with a dense embedder at 1024-d:

fp32 : 1,024 dims × 4 B × 1 M = 4,096 MB ≈ 4.0 GB
fp16/halfvec: 1,024 dims × 2 B × 1 M = 2,048 MB ≈ 2.0 GB
int8 (SQ) : 1,024 dims × 1 B × 1 M = 1,024 MB ≈ 1.0 GB
binary : 128 B × 1 M ≈ 128 MB

The quantisation options in order of real use (mid-2026):

halfvec (fp16): the default in pgvector 0.7+ and in any serious vector DB. Zero MTEB loss, 2× compression. Always turn it on.
Scalar Quantization int8 (SQ): each vector component is mapped to int8 with a global min/max. Typical recall@10 loss: 0–1%. 4× compression. The Qdrant default, supported in Milvus and Weaviate.
Binary quantisation: bit = sign(v_i - μ_i). 32× raw compression. Cold loss 5–15%. Mitigated with a Hadamard / TurboQuant pre-rotation (Qdrant 1.18, Dec 2025): it pre-multiplies by a random orthogonal matrix that spreads the energy across dimensions before binarising. After TurboQuant the loss drops to 1–3%. It also combines with rescoring over the original fp16 for the top-100 candidates.
Product Quantization (PQ): the FAISS classic. Up to 64× compression, 2–5% loss. More complex to operate (it needs a trained codebook); in 2026 it has given ground to binary + rescoring.

A corpus of 100 million chunks (a real figure for a large corporate RAG) with dense bge-m3:

Format	Total
fp32	400 GB
fp16	200 GB
int8	100 GB
binary + Hadamard	12.5 GB

The difference between 200 GB and 12.5 GB is the difference between needing a dedicated vector DB node with 8 NVMe in RAID and fitting in the RAM of a single node. For large corpora, quantisation is no longer an optimisation: it is the only way to operate.

The vector DB integration

The vector DBs of 2026 have become hybrid DBs that index all three types at once. The quick map:

Vector DB	Native hybrid	Multi-vector / ColBERT	Quantisation
Qdrant ≥1.10	RRF/DBSF in `query_points` with dense + sparse + colbert in one collection	Yes, native (one-shot MaxSim)	SQ int8, binary, TurboQuant 1.18
Weaviate	`hybrid(alpha=0.75)` BM25 + dense, multi-target named vectors	Yes, as a multi-vector named vector	PQ, SQ, rotational 8-bit BBQ
Milvus ≥2.4	Multi-vector + sparse in the schema; 2.5 adds native BM25 full-text	Multi-vector field, MaxSim orchestrated from the client	SQ, PQ, CAGRA GPU
pgvector 0.7+/0.8	`halfvec`, `sparsevec`, `bit`; HNSW for all three	Not native (separate-table workaround)	`binary_quantize()`, halfvec, rescoring with exact `<#>`
Elasticsearch / OpenSearch	`sparse_vector` (ELSER, SPLADE) + `dense_vector` HNSW; RRF	OpenSearch 3.x yes	ES 9: `int8_hnsw` by default, BBQ binary quantization

For sovereign on-prem production in Spanish, the easiest combination to operate in 2026 is Qdrant + bge-m3: a single collection indexes the three modes of the same model, the hybrid query with RRF is done in one call, TurboQuant quantisation brings the corpus down to manageable levels, and the operator is a Go binary with simple backups to S3/MinIO. pgvector + bge-m3 is the other reasonable option when you already have Postgres with HA and do not want to add a second DB to the operational inventory; you lose native multi-vector, but you gain cross-cutting SQL over the chunks.

The HNSW parameters you absolutely have to touch:

M: connections per node in the graph. 16–32 typical. Higher means more recall, more RAM. For small corpora (<1 M) M=16; for medium corpora (10 M) M=24; for large corpora M=32+IVF-PQ or M=32+binary.
ef_construction: search width during construction. 100–400. Higher means a better graph, slower construction. Build with ef_construction=400 even if it is slow; you pay it once.
ef_search: width during the query. 50–200. The main knob for the recall/latency trade-off at runtime. Start at 64 and measure.

Implications for on-premise inference

The embedder does not share hardware with the LLM as comfortably as it might seem. The numbers:

bge-m3 (568 M) takes about 568 × 2 = 1,136 MB in fp16 for the weights, plus the batch KV cache, plus temporary activations. In practice it is served comfortably in 6–8 GB of VRAM even at a high batch. It fits alongside a 7B-Q4 LLM on an RTX 4090.
gte-Qwen2-7B-instruct needs ~14 GB fp16 for weights alone. It does not fit next to a 7B LLM on a 4090; on an H100 80 GB it does, with care over simultaneous batching.
jina-colbert-v2 (560 M) takes ~1.1 GB of weights, but the storage of the multi-vector index is the real cost: 8 GB per million chunks even with Matryoshka and compression.

On the RTX 4090 (24 GB)

A realistic minimum stack for a Spanish RAG with a corpus under 1 M chunks:

GPU 24 GB ┐
 ├─ TEI bge-m3 (dense + sparse + colbert) │ ~6 GB VRAM, ~12 k tok/s
 └─ vLLM Qwen2.5-7B-Instruct AWQ Q4 │ ~8 GB VRAM, ~80 tok/s
CPU/RAM ┐
 ├─ Qdrant with bge-m3 dense + sparse + colbert │ ~3 GB RAM per M chunks
 └─ FastAPI gateway (LiteLLM)

It serves a few tens of RAG QPS with decent multilingual quality. It is the PoC configuration and the deployment for a small site.

On the 4×H100 80 GB cluster

For the production case with several million chunks and an SLO of p99 < 500 ms:

H100 #1 (80 GB) ── vLLM Qwen3-72B-Instruct AWQ + Qwen2.5-7B speculative ┐
H100 #2 (80 GB) ── vLLM gte-Qwen2-7B-instruct (embedding 32k ctx) │ LLM + large embed
H100 #3 (80 GB) ── TEI bge-m3 multi-tenant + jina-colbert-v2 reranker │ medium embed
H100 #4 (80 GB) ── Hold-out for canary / shadow │ see canary post
 ┘
Qdrant cluster (3 CPU nodes + NVMe) ── 100 M chunks indexed (binary + TurboQuant + rescoring)

This configuration separates the large LLM from the large embedder (they share the Qwen2 architecture but compete for VRAM if put on the same GPU) and leaves a whole H100 for variants in canary. bge-m3 fits with room to spare alongside the reranker on a single H100, serving tens of thousands of requests/min.

The seven operational traps of the embedder

Not normalising the query vector. Cosine and dot product coincide only when both vectors are unit vectors. If you forget v ← v / ‖v‖₂ in the client, the results are “almost right”, the top-1 is still correct on trivial queries, the top-10 no longer is, and nobody notices until RAG quality drops 8 points. Fix: bake normalisation into the embedder adapter, not into the client.
Chat template slipped into the embedder. Some LLM-based embedders (e5-mistral-7b-instruct, gte-Qwen2-7B) expect a specific instruction prompt before the text to embed ("Instruct: Retrieve relevant passages\nQuery: ..."). Forgetting it leaves performance ~5 MTEB points lower. Fix: read the usage_template in the model card and put it into the embedder wrapper.
Badly chosen Matryoshka dimension. The default in many Qdrant / pgvector clients is dim=768. If your embedder is natively MRL at 1024 → 768, fine. If it is 1024 without MRL, truncating to 768 wrecks the space (typical loss -8 MTEB points). Fix: use the model’s native dim and truncate only when storage demands it, and only on MRL models.
Missing hard negatives in fine-tuning. When the embedder is fine-tuned with your own data (which should be standard practice for corporate RAG), if the mini-batch only carries positives and in-batch negatives from the same domain, the model learns that anything outside the domain is negative, but inside the domain it does not discriminate. Fix: mine hard negatives with BM25 / dense from your own corpus before fine-tuning.
Corpus drift without reindexing. When you retrain or replace the embedder but only apply the new model to new chunks, you end up with the index mixing two incompatible geometries. Chunks from the old and the new model are not comparable by cosine. Fix: every change of embedder is a full reindexing of the corpus, planned as an operational retrain.
Tokenizer drift between client and model. The Python client that prepares the queries uses its own tokenizer (sometimes tiktoken by default) and truncates at 8,192 tokens. The embedder uses XLM-R with sentencepiece and truncates at 8,192 of its own tokenizer. Long queries are truncated differently; the corpus embeddings are consistent but the query ones are not. Fix: use the model’s tokenizer in the client or in the wrapper.
MTEB overfit as a selection guide. The MTEB leaderboard has become a contaminated metric: there is evidence that leading models saw subsets of the evaluation datasets during training. The model that is +0.5 points above the second is not necessarily better for your domain. Fix: your own domain eval set (100-300 labelled query-doc pairs) run with the evals recipe decides.

License-clean stack for sovereign production

Let us write down the recommended ending. For a Spanish organisation serving corporate RAG on-prem under ENS / ISO 42001 / EU AI Act, with a corpus of 1-50 M chunks in Spanish + English + Catalan:

Layer	Component	Licence	Rationale
Dense embedder	`BAAI/bge-m3`	MIT	Robust multilingual, 8k tokens, license-clean, served by TEI
Sparse embedder	`bge-m3` sparse head	MIT	Same pass as the dense one, no second model needed
Layer 2 reranker	`BAAI/bge-reranker-v2-m3`	MIT	Multilingual cross-encoder from the same team
Layer 3 reranker (optional)	`jinaai/jina-colbert-v2`	Apache 2.0	License-clean multilingual multi-vector
Embed server	`TEI` + `Infinity` for multi-model	Apache 2.0 / MIT	Supported stack
Vector DB	`Qdrant` (preferred) or `pgvector 0.8`	Apache 2.0 / PostgreSQL	Native hybrid + quantisation
Quantisation	`int8 SQ` + `binary + TurboQuant + rescoring`	Apache 2.0	Cuts the corpus 16×–32× with < 3% loss
Minimum hardware	RTX 4090 24 GB	—	For PoC and small sites
Production hardware	4×H100 80 GB cluster	—	For RAG with an SLO of p99 < 500 ms

The alternative stack, if explicit MMTEB in Spanish weighs more than the tri-modality of bge-m3: replace bge-m3 with Snowflake/snowflake-arctic-embed-l-v2.0 (Apache 2.0, MRL → 256) and explicitly add SPLADE-v3 or plain BM25 for the sparse layer. It loses the elegance of the single forward and gains 1-2 points on Spanish MIRACL.

Conclusion

The embedder is the easiest piece of a RAG to oversimplify and the one that decides real quality the most once everything else is in place. The three families (dense, sparse, multi-vector) are not three options to choose between but three trades that bge-m3 performs in a single pass and that hybrid retrieval consumes in parallel. The maths that matters is modest, InfoNCE with τ, MaxSim, FLOPS regularization, MRL, but the operational traps are many and quiet: forgotten normalisation, missing chat template, badly chosen Matryoshka dimension, tokenizer drift. For sovereign production in Spanish the list of viable models fits in fewer than ten, and the real decision comes down to “bge-m3 or Snowflake Arctic L 2.0”, with jina-colbert-v2 added as layer three when the fine-grained quality justifies the cost. The license-clean stack fits on an RTX 4090 for a PoC and scales to a 4×H100 cluster for real production.

References

Chen et al. M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. arXiv:2402.03216. https://arxiv.org/abs/2402.03216
Sturua et al. Jina Embeddings v3: Multilingual Embeddings With Task LoRA. arXiv:2409.10173. https://arxiv.org/abs/2409.10173
Günther et al. Jina Embeddings v4: Universal Embeddings for Multimodal Multilingual Retrieval. arXiv:2506.18902. https://arxiv.org/abs/2506.18902
Nussbaum et al. Nomic Embed v2: Multilingual Mixture of Experts. arXiv:2502.07972. https://arxiv.org/abs/2502.07972
Wang et al. Improving Text Embeddings with Large Language Models (E5-Mistral). arXiv:2401.00368.
Yu et al. Arctic-Embed 2.0: Multilingual Retrieval Without Compromise. Snowflake, 2024-12. https://www.snowflake.com/blog/arctic-embed-2-multilingual/
Lee et al. NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models. arXiv:2405.17428.
Khattab y Zaharia. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. SIGIR 2020.
Santhanam et al. ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction. arXiv:2112.01488.
Jha et al. Jina-ColBERT-v2: A General-Purpose Multilingual Late Interaction Retriever. arXiv:2408.16672.
Lassance et al. SPLADE-v3. arXiv:2403.06789.
Kusupati et al. Matryoshka Representation Learning. NeurIPS 2022, arXiv:2205.13147.
Enevoldsen et al. MMTEB: Massive Multilingual Text Embedding Benchmark. arXiv:2502.13595.
Hugging Face Text Embeddings Inference. https://github.com/huggingface/text-embeddings-inference
Michael Feil. Infinity. https://github.com/michaelfeil/infinity
Qdrant. TurboQuant 1.18 release notes. https://qdrant.tech/articles/turboquant-quantization/
pgvector. Release notes 0.7 / 0.8. https://github.com/pgvector/pgvector
Hugging Face. Embedding Quantization. https://huggingface.co/blog/embedding-quantization

Qdrant on lo0 — Tech Blog

End-to-end document ingestion: from PDF to indexed chunk

TL;DR

The analogy: an archive’s cataloguing chain

The six-stage pipeline

Stage 1 — Parsing: layout-aware vs plain text

OCR for scans

Stage 2 — Cleaning, normalising and deduplicating

Deduplication: why it matters

Stage 3 — Chunking

What the 2026 benchmarks say (and why semantic disappoints)

A worked numerical example of chunking

Stage 4 — Enriching with metadata

Stages 5 and 6 — Embedding and indexing

A cross-cutting stage — Incremental ingestion and idempotency

The arithmetic: sizing the corpus and the index

Corpus sizing and ingestion time

Vector index size

Applied to the generic 4×H100 cluster

Closing: quality is decided upstream

See also

References

Function calling and tool-augmented retrieval: the detective who knows which file to ask for

TL;DR

The analogy: the detective who knows which file to ask for

What function calling is: the anatomy of a tool call

Defining tools in the system prompt

The life cycle of a tool call

Support in OSS models

The ReAct pattern: Reasoning + Acting

A full ReAct trace: a real production query

The three sources and when to use each one

Tool routing: how the LLM picks the right tool

Descriptions that work versus those that do not

Parallel tool calling

Tool selection accuracy: the eval metric

Error handling in tool calls

SQL injection via prompt

Rate limits, timeouts and tool errors

Diagram of the ReAct loop with the three sources

Latency maths of the ReAct pipeline

Reference values: Llama-3.1-70B on 4×H100 SXM (320 GB, NVLink)

Total latency by number of iterations

Comparison with a naive RAG

On-premise hardware for ReAct agents

What we have not covered

See also

References

Semantic cache in RAG: the receptionist with a photographic memory

TL;DR

The analogy: the receptionist with a notebook

The problem in production

How the semantic cache works

Flow diagram

The threshold θ and its trade-off

The arithmetic of the saving

OSS stack 2026

GPTCache

MeanCache

Qdrant as a dual cache store

Langfuse for traceability

On-premise hardware: a reference configuration

Cases where the cache fails

1. Queries with a temporal context

2. Personalised queries over private data

3. TTL and a stale corpus

Integration into the pipeline as middleware

What we have not covered

See also

References

PostgreSQL + Qdrant in RAG ingestion: the postman who keeps two worlds in sync

The postman and the civil registry

The consistency gap problem

Outbox pattern: the outbox sheet

Mechanism

The outbox worker

CDC with Debezium: the direct phone line

Mechanism

Upsides and downsides

Comparison: outbox vs CDC

vLLM `--task embed`, for the 7B embedders