Rag on lo0 — Tech Blog

Serving embeddings and rerankers with TEI in production

Thu, 11 Jun 2026 09:40:00 +0000

Sixth piece in an operational series about squeezing a generic on-premise LLM cluster of 4×H100 SXM 80 GB. If RAG on the CPU argued where the data plane runs and the embedder zoo decided which model you serve, this post looks at the engine that serves them: the embeddings and rerankers server. The sisters in this batch, document ingestion from PDF to indexed chunk, GitOps for the inference stack with Flux and hardening and secrets for the sovereign stack, build the rest of the system around this piece. The closer of the series, the end-to-end sovereign assistant with LibreChat + LiteLLM + RAG, is still in draft.

TL;DR

Serving embeddings is not model.encode(text) scattered through the ingestion code. Just as you do not put a transformers.generate() inside your API and call it vLLM, you do not put the embedder inline either: you put it behind a dedicated server that does batching, controls concurrency, emits metrics and exposes a single HTTP contract that both ingestion and query-time reuse. In the 2026 open source ecosystem that server is TEI, Text Embeddings Inference from Hugging Face: a Rust binary that serves embedding, reranker (cross-encoder) and sequence classification models, with token-based dynamic batching, and that exposes OpenAI-compatible endpoints (/v1/embeddings) on top of the native /embed, /embed_sparse, /rerank and /predict (TEI repo, docs). It runs on CPU (ONNX Runtime, Intel MKL) and on CUDA (with FlashAttention and cuda graphs), with the same contract on both sides, so the rest of the system does not know, or care, what silicon sits behind it. The technical piece that gives almost all the performance is dynamic batching: TEI groups requests that arrive separately into a batch that shares the fixed cost of the forward, controlled by --max-batch-tokens (how many tokens fit per batch) and --max-concurrent-requests (backpressure). Grouping raises throughput, going from serving 1 sequence to 32 multiplies tokens/s at almost constant cost until the hardware saturates, in exchange for some latency while the tray fills. The numbers: a 568M encoder (bge-m3, bge-reranker-v2-m3) takes ~1.1 GB in fp16 and ~0.57 GB in int8, so several replicas fit even in a small H100 slice; embedding a short query online costs tens of ms, but reranking the top-50 costs linearly in candidates because a cross-encoder evaluates each (query, doc) pair separately. The split across the 4×H100 cluster: TEI-CPU on the fleet for batch ingestion (no SLA, throughput/€), TEI-GPU in a MIG slice of one H100 for high-QPS online embeddings and rerank, talking to ingestion and to the LiteLLM gateway through the OpenAI contract.

The analogy: the stamping press that only stamps vectors

Picture a workshop with a specialised stamping press. It does not make varied parts; it does one single thing: it takes flat material and, in one stroke, stamps a shape into it. In our case it takes text and stamps a vector. And like every industrial press, it has one property that defines all its behaviour: the cost of the stroke is almost fixed, no matter how much material you put on the tray.

Lowering the press, heating the die, aligning, pressing and lifting costs, say, one second. If you put a single sheet in, you spend that whole second stamping one part: a waste. If you fill the tray with thirty sheets and lower the press once, you spend nearly the same second and come out with thirty stamped parts. The cost per part collapses. The press performs far better when it fills the tray before pressing.

This is exactly the dynamic batching of an embeddings server. The “stroke” is the model’s forward pass on the GPU or the CPU: an operation that loads the weights, multiplies them by the activations and emits the vectors, with a large fixed cost that does not scale with how many sequences you process at once, until you saturate memory bandwidth or the compute units. Serving requests one at a time is stamping one sheet per stroke. Grouping them into a batch is filling the tray.

But the press has an operational dilemma, and here is the heart of the post. If you wait until the tray is completely full before pressing, the first sheets that arrived sit waiting for the rest: latency. If you press as soon as one sheet drops so nobody has to wait, you go back to the waste of one stroke per part: low throughput. The embeddings server solves this with a bounded wait window: it gathers whatever arrives within a window of microseconds to milliseconds, or until the batch token budget is full, and then it presses. That is the policy that turns a pile of independent requests into a few efficient strokes without anyone waiting too long.

The rest of the post is, at bottom, about configuring that press properly: how many sheets fit on the tray (max-batch-tokens), how much queue is tolerated before work is rejected (max-concurrent-requests), whether the press lives on the cheap CPU fleet (nightly ingestion, pure throughput) or in a slice of the expensive GPU (online, bounded latency), and why the reranker is a different press whose cost does grow with the number of parts.

What TEI is, exactly

Text Embeddings Inference (TEI) is Hugging Face’s inference server for the family of models that does not generate tokens: embedders, cross-encoder rerankers and sequence classifiers. It is to encoders what vLLM is to generative LLMs: a high-performance server, written in Rust, that handles tokenisation, batching, concurrency, metrics and the HTTP contract, leaving the application code to only make calls (TEI repo).

Three model types, three jobs:

Embedding (bi-encoder): turns a text into a reusable vector. bge-m3, Snowflake Arctic Embed, multilingual-e5. The vector is stored in the index and reused on every search.
Reranker (cross-encoder): takes a pair (query, document) and emits a relevance score, not a vector. bge-reranker-v2-m3. It produces nothing reusable: every pair is evaluated again.
Sequence classification: emits labels/probabilities over a text. Useful for lightweight guardrails, routing, language or toxicity detection.

The endpoints

TEI exposes both an OpenAI-compatible endpoint and its native endpoints, verified against the current documentation (Quick Tour, DeepWiki):

Endpoint	What for	Model type
`/v1/embeddings`	Embeddings with the OpenAI contract (drop-in for clients that already speak OpenAI)	embedder
`/embed`	Dense embeddings, TEI’s native API	embedder
`/embed_sparse`	Sparse embeddings (lexical head of models like `bge-m3`/SPLADE)	embedder
`/rerank`	Orders a list of texts by relevance against a query	reranker (cross-encoder)
`/predict`	Sequence classification (labels/scores)	classifier
`/embed_all`	Per-token embeddings (no pooling)	embedder
`/similarity`	Direct similarity between texts	embedder
`/metrics`, `/health`, `/info`, `/tokenize`, `/decode`	Operation, observability and utilities	all

The operational key is in the first row: TEI speaking the OpenAI contract on /v1/embeddings means the embedder is swappable without touching client code. The same code that called OpenAI’s text-embedding-3-large points at your on-prem TEI by changing the base_url, and the gateway (LiteLLM) routes it as one more provider. /rerank, by contrast, is not part of the OpenAI standard, no such endpoint exists in their API, so the gateway treats it as a reranking-specific endpoint (LiteLLM TEI rerank).

The backends

TEI runs on CPU and on GPU with the same HTTP contract, which is exactly the property the CPU/GPU split of RAG needs (TEI docs, hardware):

CPU: cpu-* image. ONNX Runtime (recommended) or Intel MKL backend. It exploits the integer compute paths (AVX-512+VNNI, AMX on 4th-gen Xeon) when the model is in int8. It is the batch ingestion engine.
CUDA: architecture-specific images (cuda-*, with variants for different compute capabilities). It uses FlashAttention and cuda graphs to squeeze the GPU. It requires compute capability ≥ 7.5 (Volta is out) and CUDA 12.2+ compatible drivers. It is the high-QPS online engine.
There is also Metal support (Apple Silicon) and experimental ROCm (AMD Instinct), less relevant to the on-prem case of this post.

The golden rule: the same bge-m3 served by TEI-CPU and by TEI-GPU exposes an identical /v1/embeddings. The client does not notice the silicon. That is what lets you put ingestion on CPU and online on GPU without rewriting anything.

Why a dedicated server and not calling the model inline

The temptation, especially in the prototype, is to load the embedder into the application process and call model.encode() directly. It works in the demo and breaks in production for five reasons, all solved by the dedicated server:

Batching. Your ingestion code processes documents; your query code embeds one query at a time. Inline, each encode() is an independent forward, one press stroke per sheet. A dedicated server joins requests from different origins (ingestion + several concurrent queries) into a single batch and amortises the fixed cost. This is what really multiplies throughput, and you cannot do it if the model lives inside each isolated process.
Concurrency and backpressure. Under load, what happens when 10,000 requests arrive at once? Inline, you run out of memory or queue without control. TEI has --max-concurrent-requests: above that limit it rejects instead of degrading everyone, which is the correct way to manage overload (TEI CLI args).
Metrics. TEI exposes /metrics in Prometheus format: latencies, batch size, tokens/s, queued requests. Inline you have none of this without hand-instrumenting every encode(). To observe the data plane you need that endpoint.
A reusable HTTP contract. The same server is consumed by batch ingestion, by online query-time and by any other service that needs vectors. One copy of the model in memory, one version point, one place to change the model. Inline, each service loads its own copy and the version drifts on its own.
Lifecycle separation. The embedder is updated, restarted or scaled without touching the application. If tomorrow you swap bge-m3 for Snowflake Arctic, you change the server container, not the code of N services. (Remember the embedder drift trap: changing embedder forces a reindex; having a single server makes that change atomic and auditable.)

It is the same argument for why nobody serves an LLM with transformers.generate() in production: you serve it with a dedicated engine. The embedder is no different.

Dynamic batching: the mechanics of the press

Here is the performance engine. TEI does token-based dynamic batching: instead of processing requests one at a time or grouping them by a fixed number of requests, it groups them by token budget (TEI README, Discussion #367).

Two knobs rule:

--max-batch-tokens (the default depends on the build, typically 16384): the total tokens that fit in a batch. With max-batch-tokens=1000 you get 10 requests of 100 tokens, or one of 1000. The docs say it clearly: this number should be as large as possible until the model becomes compute-bound. It is the size of the press tray, measured in tokens, not in parts, because the real cost scales with tokens, not with number of texts.
--max-concurrent-requests (default 512): how many requests it admits in flight before rejecting. It is the backpressure control, not the batching one: it protects against surges by returning an error instead of queuing indefinitely.

The policy: TEI keeps an internal queue. When there is work, it fills a batch up to the max-batch-tokens budget (or until there are no more requests waiting) and sends it to the model in one stroke. Requests that arrive while the previous batch is being processed wait in the queue and go into the next one. Watch out for a real operational detail: if max-batch-tokens is smaller than the model’s maximum input length, you can trigger a loop where a long request never fits into a batch (Issue #723); the token budget has to be at least as large as the longest sequence you accept.

The number: throughput with batching vs without batching

Let us model the press. Let $C$ be the fixed cost of a forward pass (loading weights, launching kernels) and $c$ the marginal cost per sequence inside the batch. For a batch of size $B$, the stroke time is roughly:

$$t(B) \approx C + c \cdot B$$

as long as the batch fits in the hardware (memory-bandwidth-bound, which is the normal regime of a small encoder on GPU). The throughput, sequences per second, is:

$$\text{throughput}(B) = \frac{B}{t(B)} = \frac{B}{C + c \cdot B}$$

Let us put in concrete numbers, indicative but of the right order for bge-m3 (568M) on a GPU serving ~256-token sequences. Say $C = 4$ ms of fixed cost per stroke and $c = 0.5$ ms per marginal sequence.

Without batching ($B=1$):

$$t(1) = 4 + 0.5 = 4.5 \text{ ms} \quad\Rightarrow\quad \text{throughput} = \frac{1}{4.5\text{ms}} \approx 222 \text{ sec/s}$$

With batching ($B=32$):

$$t(32) = 4 + 0.5 \times 32 = 20 \text{ ms} \quad\Rightarrow\quad \text{throughput} = \frac{32}{20\text{ms}} = 1600 \text{ sec/s}$$

The batch of 32 multiplies throughput by ~7.2× compared with serving one at a time, because the fixed cost $C$, which at $B=1$ was 89% of the time, is spread across 32. And the asymptotic limit, as $B \to \infty$, is $1/c = 2000$ sec/s: the press cannot go faster than its marginal cost, and we are already at 80% of that ceiling with $B=32$. Going up to $B=128$ would give ~1969 sec/s, a marginal improvement in exchange for a good deal more latency and memory. Batching has diminishing returns: most of the gain is in going from 1 to a few dozen.

The latency trade-off is the other face. The request that arrived first in the batch of 32 does not see its response until the whole stroke finishes: it waits for the tray to fill plus the 20 ms of the forward. If the 32 requests arrived within a window of, say, 5 ms, the first request suffers ~5 ms of batching wait + 20 ms of compute = 25 ms of latency, against the 4.5 ms it would have had being served alone. The p99 latency gets worse, that is the price of the full tray, but in exchange the system absorbs 7× more load with the same silicon. That is why the correct configuration depends on the use case: in batch ingestion (no user waiting) you maximise max-batch-tokens and latency does not matter; online (user waiting) you bound the wait window to keep p99 under control even at the cost of some throughput.

Deployment: compose for embedder and reranker

TEI is deployed as a container. The key arguments: --model-id (the model from the Hub), --pooling (how it aggregates tokens into the vector: cls, mean, splade, last-token), --dtype (precision), and the batching knobs already seen. Two separate services, since the embedder and the reranker are different models and different endpoints, each with its own container.

Embedder (`bge-m3`) on GPU

# docker-compose: TEI serving bge-m3 as an embedder, on GPU
services:
 tei-embed:
 image: ghcr.io/huggingface/text-embeddings-inference:cuda-1.9
 command:
 - --model-id=BAAI/bge-m3
 - --pooling=cls
 - --dtype=float16
 - --max-batch-tokens=16384 # big tray: the press performs
 - --max-concurrent-requests=512 # backpressure: reject above this
 ports: ["8081:80"]
 volumes: ["./hf-cache:/data"]
 deploy:
 resources:
 reservations:
 devices: [{driver: nvidia, count: 1, capabilities: [gpu]}]
 # exposes /v1/embeddings (OpenAI), /embed, /embed_sparse, /metrics

Reranker (`bge-reranker-v2-m3`) on GPU

 tei-rerank:
 image: ghcr.io/huggingface/text-embeddings-inference:cuda-1.9
 command:
 - --model-id=BAAI/bge-reranker-v2-m3  # cross-encoder, ~568M
 - --dtype=float16
 - --max-batch-tokens=16384
 - --max-concurrent-requests=256
 ports: ["8082:80"]
 volumes: ["./hf-cache:/data"]
 deploy:
 resources:
 reservations:
 devices: [{driver: nvidia, count: 1, capabilities: [gpu]}]
 # exposes /rerank — invoked ONLY over top-k (20/50), not over hundreds

bge-reranker-v2-m3 is a ~568M cross-encoder built on bge-m3, multilingual, that takes (query, document) and emits a relevance score directly, without producing a reusable vector (model card, BGE docs). It carries no --pooling because it emits no embedding: it emits a scalar.

CPU for batch ingestion

For the CPU fleet, it is enough to change the image and the dtype; the HTTP contract is identical:

 tei-embed-cpu:
 image: ghcr.io/huggingface/text-embeddings-inference:cpu-1.6
 command:
 - --model-id=BAAI/bge-m3
 - --pooling=cls
 - --dtype=float16  # on CPU, ONNX int8 if the export supports it
 - --max-batch-tokens=8192 # smaller tray: the CPU saturates sooner
 ports: ["8083:80"]
 volumes: ["./hf-cache:/data"]
 # ONNX/MKL backend: exploits AVX-512+VNNI / AMX on 4th-gen Xeon

Any OpenAI client embeds against the endpoint without knowing what is behind it:

from openai import OpenAI
client = OpenAI(base_url="http://tei-embed:80/v1", api_key="-") # api_key ignored
r = client.embeddings.create(model="BAAI/bge-m3", input=["user query"])
vec = r.data[0].embedding # same contract as OpenAI; the silicon is invisible

And the rerank, through its native (non-OpenAI) endpoint:

import requests
docs = ["retrieved chunk 1", "retrieved chunk 2", "...top-20 from retrieval..."]
r = requests.post("http://tei-rerank:80/rerank",
 json={"query": "user query", "texts": docs})
ranked = r.json() # [{index, score}, ...] ordered by relevance

Integration: how ingestion and the gateway consume it

The TEI server has two clients with opposite profiles, and that is precisely the reason to centralise it:

Ingestion (batch, no SLA) calls /v1/embeddings or /embed with large lots of chunks. Here throughput is what matters: ingestion pushes thousands of texts and TEI groups them into large batches against the high max-batch-tokens. It is the case of the press with a full tray. Ingestion then writes the vectors (dense + sparse) into the vector DB; the specific PDF-to-indexed-chunk pipeline is the sister piece of this series in preparation, and meanwhile the PostgreSQL + Qdrant schema in microservices covers the structure of the pipeline.

The gateway (query-time, online) consumes TEI through two different routes:

Embedding the query: a single call to /v1/embeddings with a short text. The gateway, LiteLLM or another L7 router, treats it as one more OpenAI embeddings provider, routed by model. That embedded query goes to the vector DB for the hybrid search.
Reranking the top-k: after retrieving 20-50 candidates from the retriever, the gateway calls /rerank on the reranker server with the query and the candidate list, and keeps the best ones. Since /rerank is not an OpenAI standard, LiteLLM exposes it through its own reranking contract (LiteLLM rerank with TEI).

The full query-time pattern: embed query (TEI dense) → search (vector DB, hybrid retrieval) → rerank top-k (TEI rerank) → send query + top-5 to the LLM. Two different TEI endpoints, one for dense and another for rerank, both behind the gateway, both swappable.

Sizing: memory, replicas and latency budget

Footprint of a 568M encoder

The weight footprint of bge-m3 (568M params) depends only on precision:

$$\text{fp16: } 568 \times 10^6 \times 2 \text{ B} \approx 1.14 \text{ GB} \qquad \text{int8: } 568 \times 10^6 \times 1 \text{ B} \approx 0.57 \text{ GB}$$

On top of this comes the space for the batch’s temporary activations (proportional to max-batch-tokens × hidden dimension) and, in TEI, the FlashAttention/cuda graphs structures. In practice, a bge-m3 in fp16 is served comfortably in ~6 GB of VRAM even at high batch, and in int8 on CPU the whole model (0.57 GB) fits in the cache and RAM of any server without blinking, activating the AMX/VNNI paths that make it viable, which is exactly the argument of RAG on the CPU.

How many replicas fit? If we reserve a 10 GB slice of an H100 for serving embeddings online, with an effective footprint of ~2.5 GB per replica (fp16 weights + activations + overhead), around 3-4 replicas of bge-m3 fit in that slice, splitting the QPS between them. In int8 the footprint drops and more fit, at the cost of a full point of quality. For the reranker, the weight footprint is the same (568M), but its batch pattern is different (query-doc pairs), so it is worth sizing it separately.

Latency budget: short query vs top-50 rerank

Here is the fundamental asymmetry between embedder and reranker, and it is what decides the sizing.

Embedding a short query (online): it is one forward over a text of tens of tokens. On GPU, inside a dynamic batch, it is tens of ms including the batching wait, the order of the earlier numerical example, ~5-25 ms. It is cheap and constant: a query is a query, no matter how many documents are in the corpus. That is why embedding online fits easily into an interactive latency budget.

Reranking the top-50 (online): a cross-encoder does not produce a reusable vector; it evaluates every pair (query, document) in a forward. The cost is linear in the number of candidates:

$$\text{cost}_{\text{rerank}} \propto k \times \text{forward}(\text{query} + \text{doc})$$

Reranking a top-50 is 50 forwards of (query + doc) pairs, where each pair is longer than the query alone (query + ~256-token chunk). TEI groups them into batches, and batching helps again there, but the total work is ~50× that of embedding one query. If embedding a query costs of the order of 10-20 ms, reranking 50 candidates costs of the order of hundreds of ms depending on the batch and the hardware. Hence the operational rule repeated throughout the series: broad, cheap recall in the retriever, precision rerank over FEW candidates. Reranking a top-20/50 fits in the online budget; reranking hundreds at high QPS does not, and that is where the reranker demands a dedicated GPU or a larger slice. The detail of why the cross-encoder is expensive but precise is in reranker and hybrid retrieval.

The query-time latency budget, added up:

Stage	Typical cost (order)	Scales with
Embed query (TEI dense)	~10-20 ms	constant (1 short text)
Hybrid search (vector DB, HNSW+sparse)	single-digit ms	index size (sublinear)
Rerank top-k (TEI cross-encoder)	~100-300 ms	linear in k
Generation (LLM, outside TEI)	seconds	output tokens

The reranker is the most expensive stage of the data plane, and the only one that scales badly with the width of retrieval. Size it as the critical component.

Applied to the generic 4×H100 cluster

Let us come down to the cluster of the series: 4×H100 SXM 80 GB plus a generic CPU fleet (Xeon with AMX, NUCs). TEI lives in two places depending on the load profile, and the identical OpenAI contract is what makes each place swappable.

TEI-CPU on the fleet → batch ingestion. Re-embedding the corpus is throughput-bound work with no latency SLA: its place is the CPU fleet with bge-m3 in int8 (ONNX/AMX). Here you maximise max-batch-tokens because nobody is waiting, and the press performs with the tray brimming. No H100 should spend a cycle reindexing a corpus that changes once a day, which is exactly the bad split that RAG on the CPU takes apart. The CPU fleet scales horizontally and cheaply; the corpus is chopped up and spread across nodes.

TEI-GPU in a MIG slice → high-QPS online embeddings and rerank. Query-time has a latency SLA and, if the system serves many requests per second, it needs the GPU’s throughput and low latency. But a 568M encoder does not deserve a whole H100: it takes ~2.5 GB effective, and leaving it 80 GB wastes 97% of the card. The right answer is a MIG slice: partition an H100 into hardware-isolated instances and give TEI one of the small slices (a 1g.10gb, for example), leaving the large slices for generation or for other tenants. The online embedder and reranker live there, with memory and compute isolation guaranteed by the partitioning. The how of the partitioning, MIG, MPS, time-slicing and when to use each, is in sharing a GPU; the reading for TEI is direct: MIG gives the hard isolation that an online service with bounded latency wants, while batch ingestion on CPU does not even touch the GPU.

The split, in one sentence: ingestion squeezes throughput/€ on CPU with a full tray; online squeezes latency/QPS in a MIG slice of an H100; both speak the same /v1/embeddings and /rerank, so moving load from one side to the other is changing a URL in the gateway.

If at some point you need a 7B embedder (gte-Qwen2, NV-Embed) whose quality bge-m3 cannot reach, that is no longer small-encoder work: it drags an LLM’s cost profile with it and is served where 7Bs are served, with vLLM --task embed in a large GPU slice, not with TEI-CPU. But it is the occasional exception, not the base load.

Conclusion

TEI is the stamping press of the RAG data plane: a specialised server that only stamps vectors and scores, and that performs far better when it fills the tray before pressing. The technical piece that gives the performance is token-based dynamic batching, controlled by max-batch-tokens and max-concurrent-requests, with a clear trade-off between throughput and p99 latency that is resolved differently in ingestion (tray brimming, no SLA) and online (bounded window, p99 protected). The architectural virtue is the OpenAI-compatible contract on /v1/embeddings, which makes the embedder swappable without touching code, plus the native /embed_sparse and /rerank endpoints for the sparse head of bge-m3 and for the bge-reranker-v2-m3 cross-encoder. The numbers drive the sizing: a 568M encoder fits in gigabytes, not tens of gigabytes, so several replicas live in a MIG slice; embedding a query is cheap and constant, but reranking is linear in candidates and is the critical stage of the latency budget. On the 4×H100 cluster, the right split is TEI-CPU for batch ingestion and TEI-GPU in a MIG slice for high-QPS online: two presses, two silicons, one contract.

References

Hugging Face — Text Embeddings Inference (TEI), repo y docs (Rust, backends CPU ONNX/MKL y CUDA, endpoints OpenAI-compatibles). https://github.com/huggingface/text-embeddings-inference · https://huggingface.co/docs/text-embeddings-inference
Hugging Face — TEI Quick Tour (endpoints /embed, /embed_sparse, /rerank, /predict, /v1/embeddings). https://huggingface.co/docs/text-embeddings-inference/quick_tour
Hugging Face — TEI CLI arguments (--model-id, --pooling, --dtype, --max-batch-tokens, --max-concurrent-requests). https://github.com/huggingface/text-embeddings-inference/blob/main/docs/source/en/cli_arguments.md
Hugging Face — TEI README / Discussion #367 (token-based dynamic batching; semántica de max-batch-tokens). https://github.com/huggingface/text-embeddings-inference/blob/main/README.md · https://github.com/huggingface/text-embeddings-inference/discussions/367
BAAI — BGE-M3 model card (568M, XLM-RoBERTa, dense+sparse+colbert, MIT). https://huggingface.co/BAAI/bge-m3
BAAI — bge-reranker-v2-m3 model card (cross-encoder ~568M sobre bge-m3, multilingüe). https://huggingface.co/BAAI/bge-reranker-v2-m3 · https://bge-model.com/bge/bge_reranker_v2.html
Chen, J., et al. BGE M3-Embedding. arXiv:2402.03216. https://arxiv.org/abs/2402.03216
BerriAI — LiteLLM: proveedor TEI para /rerank. https://github.com/BerriAI/litellm/issues/8372

End-to-end document ingestion: from PDF to indexed chunk

Thu, 11 Jun 2026 09:00:00 +0000

Fourth piece in an operational series about squeezing a generic on-premise LLM cluster of 4×H100 SXM 80 GB. Its siblings in this batch: serving embeddings and rerankers with TEI in production details the inference piece that this ingestion feeds; GitOps for the inference stack with Flux versions and deploys this whole pipeline; and hardening and secrets for the sovereign stack protects the corpus and the credentials this ingestion touches. The final consumer of what we build here, an end-to-end sovereign assistant, is assembled in a later instalment.

TL;DR

A RAG system inherits the quality of its corpus, and the corpus inherits the quality of the ingestion that manufactured it. This is the garbage-in/garbage-out of retrieval: a chunk badly extracted from a table, a page header repeated a thousand times, a scanned PDF from which you only got OCR noise; all of it gets embedded, indexed and reappears as poisoned context in the model’s answer. Document ingestion is not an afternoon’s script, it is a six-stage pipeline with engineering decisions at every one: (1) extract/parse, where almost everything is decided: layout-aware (Docling, which with Granite-Docling-258M preserves tables, formulas and structure, and which according to IBM avoids classic OCR and is up to ~30× faster) against plain text extraction (PyMuPDF), with OCR or a VLM for scans; (2) clean/normalise, removing boilerplate, normalising Unicode, rebuilding broken paragraphs; (3) chunk, and here 2026 has shifted the consensus: a February 2026 benchmark put recursive at 512 tokens in the lead (69% accuracy) ahead of semantic chunking (54%), and late chunking adds global context at no extra storage cost; (4) enrich metadata, with source, page, section, timestamp, ACL/tenant: for filtered retrieval, citation and auditability; (5) embed, through an embeddings server such as TEI; (6) index, in pgvector or Qdrant, with payload, dense + sparse. In between, exact deduplication (hash) and near-duplicate detection (MinHash/LSH or cosine with a threshold), because near-duplicates degrade retrieval recall and diversity. The numbers matter: a corpus of $N_{docs}\times$ pages $\times$ tokens/page gives the total tokens and, at a given CPU throughput, the ingestion time; and $N_{chunks}\times d\times\text{bytes}$ gives the index size, which in int8 falls 4× against fp32. The reading for the 4×H100: ingestion is CPU work, which links to the RAG-on-CPU piece; the GPU only comes in if you parse with a VLM (Granite-Docling) or use a 7B embedder.

The analogy: an archive’s cataloguing chain

Picture the cataloguing department of a large document archive. It is not one person putting papers into boxes; it is a chain of workstations, each with a different craft and its own quality standard. A document goes in raw at one end and comes out the other turned into a record you can find in seconds. If one station does its job badly, the ones downstream inherit the error and amplify it.

The first station is intake and reading. A heterogeneous box arrives: typed reports, crooked photocopies, folded tables, microfiche. An expert archivist really reads each item: distinguishes the body of the text from the margins, rebuilds a table that spans two pages, transcribes by hand what the scanner could not read. A novice archivist, by contrast, photocopies everything flat and hands over a slab of text where the table columns are interleaved into nonsense. That is exactly the difference between layout-aware parsing and plain text extraction, and it is where almost all of the quality is won or lost.

The second station is weeding out. Before filing anything, someone removes the duplicates (three copies of the same memo, two near-identical versions of a report) and cleans off the useless marks: “COPY” stamps, footers repeated on every sheet, coffee stains. If you do not weed, the archive fills up with copies that, when someone searches, return the same document six times and bury everything else. This is deduplication and cleaning.

The third station cuts it into records. A 400-page book is not catalogued as one giant record; it is broken down into manageable entries, by chapter, by section, of a size a reader can take in at a glance. Too big and the record mixes topics; too small and it loses context. This is chunking, and the size of the record is the decision that most shapes retrieval.

The fourth station labels. Each record carries a shelf mark, a date, its provenance, an access level (can anyone see this, or only the legal department?). Without those labels you cannot filter a search or say which document a statement came from. These are the metadata: source, page, section, timestamp, ACL.

The fifth and sixth stations place the record on the indexed shelf: they translate it into the catalogue’s language, a vector, and put it in the right drawer of the filing cabinet, so that a query finds the related records without walking through the whole thing. This is embedding and indexing in the vector store.

The moral runs through the whole post: RAG cannot retrieve better than what the cataloguing chain filed. If the first station chopped up a table badly, no reranker will fix it afterwards. Ingestion is the quality station for the entire system, and almost all of it, like the archive full of patient people working with nobody timing them, is background work that fits on CPU.

The six-stage pipeline

The temptation to treat ingestion as “read the PDF and chop it up” is the source of 80% of mediocre RAG systems. Each stage has a quality standard and a characteristic failure, and failures cascade downstream: if you parse badly, cleaning does not recover what was lost; if you chunk badly, embedding freezes the error into the vector; if you do not label, you will not be able to filter or cite. The rest of the post walks through the six stations and their decisions.

Stage 1 — Parsing: layout-aware vs plain text

Almost everything is won or lost here. A PDF is not text: it is a set of instructions for drawing glyphs on a canvas. “Extracting the text” from a PDF means reconstructing a reading order that the format does not guarantee, and tables, columns and figures break it systematically.

There are two philosophies, and the choice shapes the rest of the pipeline.

Plain text extraction. Tools such as PyMuPDF (fast, robust, no heavy dependencies) read the PDF’s text stream and dump it out. For single-column documents, running prose and no tables, it is perfect: extremely fast, faithful and cheap on CPU. Its limit shows up with structure: a two-column table comes out with the cells interleaved, a double-column document mixes the end of one with the start of the other, and a scanned invoice does not come out at all because there is no text layer. PyMuPDF can also segment by the table of contents (TOC) when the PDF carries one, which helps with chunking by section (Omdena, Document Parsing for RAG: A Complete Guide for 2026).

Layout-aware parsing. Tools such as Docling (an open source project driven by IBM Research) and unstructured.io first understand the layout, identifying titles, paragraphs, tables, figures, lists and formulas, and only then extract the content while respecting that structure. Docling captures table structure (rows, columns, multi-level headers) and, in its 2026 evolution, does so with a VLM: Granite-Docling-258M, released by IBM in January 2026 under Apache 2.0, a compact vision-language model (~258M parameters, Granite 3 backbone + SigLIP2 visual encoder) that converts pages, whether PDF, slides or scans, directly into a structured format called DocTags, preserving tables, code, inline and block mathematics, and the document’s hierarchy (IBM, Granite-Docling: End-to-end document understanding; model card on Hugging Face; Docling repo). IBM states that the VLM route avoids classic OCR and that this “reduces errors and speeds up the solution by up to 30×” against a traditional OCR pipeline. That is an IBM Research figure; I quote it and treat it as indicative, not as an independent benchmark reproduced here.

unstructured.io offers partitioning strategies graded by document complexity: fast (plain text, quick), hi_res (identifies the layout, recommended when classifying tables and elements properly matters), VLM and auto, balancing speed, cost and accuracy (Unstructured, PDF Parsing Strategies for RAG). The practical rule: fast for prose, hi_res or VLM for documents with tables and structure.

OCR for scans

When the document has no text layer (a scan, a photo, a digitised microfiche), you have to recognise the characters. Three routes:

Classic OCR (Tesseract, PaddleOCR, EasyOCR, which is what Docling integrates with when explicit OCR is needed). Mature, CPU-friendly, good with clean text; it struggles with tables, handwriting and complex layouts.
End-to-end VLM (Granite-Docling and similar). The model “looks” at the page and emits structure directly, with no separate OCR stage. Better with complex layouts; this is where the GPU does come in if the VLM is large or the volume is high.
Hybrid: OCR for character transcription, a layout model for the structure.

The honest criterion: for a corpus of native PDFs with text, PyMuPDF or unstructured fast handle it on CPU and cheaply. For a corpus with dense tables, forms or scans, layout-aware Docling/Granite-Docling earns its cost in chunk quality, and it is the only stage of the pipeline where the GPU can be justified.

Document case	Recommended tool	Silicon
Native PDF, single column, prose	PyMuPDF / unstructured `fast`	CPU
PDF with tables, double column, hierarchy	Docling / unstructured `hi_res`	CPU (layout models)
Scan, form, handwriting, complex layout	Granite-Docling (VLM) or OCR+layout	GPU if large VLM / high volume
HTML, DOCX, PPTX	Docling (multi-format) / native parsers	CPU

Stage 2 — Cleaning, normalising and deduplicating

Freshly parsed text comes in dirty. Cleaning means removing what adds nothing and normalising what is represented in a thousand ways:

Boilerplate: headers and footers repeated on every sheet, page numbers, watermarks, HTML navigation menus, cookie banners. If you do not remove them, they get embedded a thousand times over and contaminate both the index and the answers.
Unicode normalisation (NFC/NFKC), whitespace and dashes: the same character represented in several ways breaks exact matching and dirties the embeddings.
Paragraph reconstruction: undoing the hard line breaks the PDF inserted mid-sentence, without merging paragraphs that were meant to stay separate.

Deduplication: why it matters

RAG suffers two ills from duplicates. The exact kind, the same document uploaded three times, inflates the index and makes a search return the same answer repeatedly, wasting the top-k context slots. The near-duplicate kind, two near-identical versions of a report, a document and its draft, is worse: they look different to a hash but say the same thing, and they degrade retrieval recall and diversity. This is not theory: in the MS MARCO V2 collection a substantial overlap of near-duplicates has been documented which, left untreated, degrades retrieval precision and reduces the diversity of retrieved documents (Ragnarök / TREC RAG 2024).

Two levels of dedup, complementary:

Exact (hash). Compute a sha256 of the normalised content of each document (or chunk) and discard the matches. Cost $O(N)$, trivial. It catches byte-for-byte duplicates.
Near-dup (MinHash + LSH, or cosine of embeddings with a threshold). For those that differ slightly but mean the same thing. MinHash compresses each document into a compact signature such that the probability of two signatures matching at a position equals the Jaccard similarity of the original shingle sets; combined with Locality-Sensitive Hashing (LSH) it finds every near-duplicate pair without comparing all against all, reducing a quadratic problem to almost linear (Brenndoerfer, MinHash, Jaccard, LSH). It is the dominant technique in cleaning LLM training corpora (C4 and RefinedWeb use it) and applies just the same to a RAG corpus (Zilliz, Data Deduplication at Trillion Scale). The alternative, cosine of embeddings with a threshold (say > 0.97), catches semantic duplicates that MinHash cannot see (paraphrasing), but it requires you to already have the embeddings and is more expensive.

The practical rule: exact hash always (it is free); MinHash/LSH for large corpora with versions; cosine with a threshold if paraphrasing is a real problem. And deduplicate before embedding: re-embedding a duplicate means spending compute on rubbish you will then have to filter out.

Stage 3 — Chunking

Chunking is the decision that most shapes retrieval, and the one that carries the most myths. The trade-off is threefold: chunk size ↔ retrieval granularity ↔ context cost.

Large chunks: each one holds more context and there is less risk of cutting an idea in half, but the search is less precise (one vector represents too many topics) and, on retrieval, you put more tokens into the LLM’s prompt, which means more cost and more risk of diluting what is relevant.
Small chunks: very granular, precise retrieval, but each chunk loses context (a 43-token fragment may mean nothing outside its section) and you need to retrieve more of them to cover an answer.

The strategies, from least to most sophisticated:

Fixed size + overlap. Cut every $N$ tokens with an overlap of $k$ tokens between consecutive chunks so as not to slice a sentence dead. Simple, predictable, a reasonable baseline. The overlap is the insurance against cutting an idea right at the boundary.
Recursive (LangChain’s RecursiveCharacterTextSplitter). It tries to cut on separators in priority order (paragraph, then sentence, then word) to respect the natural structure as far as possible before falling back to a hard cut. It is the workhorse.
Semantic. Embed sentences and cut where the similarity between consecutive sentences falls below a threshold, grouping by coherence of meaning. It sounds better on paper; in 2026 practice it has disappointed (see below).
Structure/layout-aware (by headings). It exploits the hierarchy the layout-aware parser has already extracted: one chunk per section or subsection. unstructured offers the by_title strategy, which opens a new chunk when a title element appears, avoiding mixing text from different sections (Unstructured docs). If you parsed with Docling/hi_res, this strategy is almost free and is usually the best one for well-structured documents.
Late chunking. The twist of 2024–2026: instead of chunking and then embedding each chunk separately, embed the whole document first (with a long-context encoder) and then apply the chunk boundaries by mean-pooling the token embeddings inside each span. The result: every chunk keeps the document’s global context (a pronoun or a reference that only makes sense given the previous paragraph is encoded in the vector) and all of it with no extra storage cost, because you end up with one vector per chunk just as before (Jina AI, Late Chunking; arXiv:2409.04701).

What the 2026 benchmarks say (and why semantic disappoints)

It is worth being sceptical about fashion. A Vecta benchmark from February 2026 covering 7 strategies over 50 academic papers put recursive at 512 tokens in first place with 69% accuracy, while semantic chunking came in at 54%, partly because it produced tiny fragments, averaging 43 tokens, too small to mean anything (Firecrawl, Best Chunking Strategies for RAG in 2026). A systematic analysis from January 2026 also identified a “context cliff” around 2,500 tokens, where answer quality drops once you feed in contexts that are too long, an extra argument against giant chunks ([ibid.]). The honest reading: recursive at a moderate size with overlap remains the baseline that is hard to beat; late chunking is the improvement with the best cost/benefit ratio when the model supports it; semantic promises more than it delivers.

A worked numerical example of chunking

Take a technical document of 30 pages, with ~500 tokens of useful prose per page after cleaning (tables and figures are chunked separately). That is $30 \times 500 = 15{,}000$ tokens of text. We chunk with recursive at 512 tokens and 20% overlap ($0.20 \times 512 \approx 102$ tokens). The effective “step” between the start of one chunk and the next is:

$$\text{step} = \text{size} - \text{overlap} = 512 - 102 = 410 \text{ tokens}$$

The number of chunks in the document is then, approximately:

$$N_{chunks} \approx \left\lceil \frac{T_{doc} - \text{overlap}}{\text{step}} \right\rceil = \left\lceil \frac{15{,}000 - 102}{410} \right\rceil \approx \lceil 36.3 \rceil = 37 \text{ chunks}$$

Without overlap it would have been $\lceil 15{,}000 / 512 \rceil = 30$ chunks; the 20% overlap costs us 7 extra chunks (~23% more) in exchange for not splitting ideas at the boundaries. That is the concrete price of overlap: more vectors to embed, index and store, in exchange for robustness in retrieval. We will use this factor when sizing the full corpus.

Stage 4 — Enriching with metadata

A chunk with no metadata is a record with no shelf mark: it exists but it is no use. Each chunk gets a payload attached with, at a minimum:

Source and location: document_id, document name/URI, page number, section/heading (which the layout-aware parser already gave you). Essential for citation: being able to say “this comes from document X, page 12, section 3.2” is what separates an auditable RAG from one that hallucinates with no traceability.
Timestamp: when it was ingested and the document’s date. It lets you filter by freshness and detect stale content.
ACL / tenant: who is allowed to see this chunk. It is critical and it is applied as a filter at retrieval time: a user from department A must not retrieve chunks marked for B only. Without this, the RAG system is a data leak waiting to happen.
Embedding model version (model_version): so you know which embedder generated each vector and can migrate without mixing incompatible spaces.

These metadata are not decoration: they enable filtered retrieval (searching only in what the user can see, or only in documents after a given date), citation (reconstructing the origin of every statement) and auditability (knowing what was retrieved, from where and when). All of it lives in the point’s payload in the vector store.

Stages 5 and 6 — Embedding and indexing

The last two stations translate the chunk into a vector and place it on the shelf.

Embedding. The chunks are sent in batches to an embeddings server, typically Hugging Face’s TEI (Text Embeddings Inference), which exposes the OpenAI /v1/embeddings contract and runs on CPU or GPU. It is throughput-bound work with no latency SLA: its natural home is the CPU with a small encoder in int8 (the sibling piece in this batch, serving embeddings and rerankers with TEI, details the how). It is worth emitting dense + sparse at the same time: the dense vector captures semantics, the sparse one (SPLADE/BM25-like) exact lexical overlap, and together they make hybrid retrieval more robust.

Indexing. The vectors, with their payload, are upserted into the vector store. Two reference options, both current in 2026:

pgvector (a PostgreSQL extension). Its great virtue is living inside Postgres: ACID transactions, joins with the relational metadata, a single database to operate. Version 0.8 added halfvec (half-precision vectors, 2× less storage) and 0.9 (early 2026) added support for sparse vectors and speed improvements. Its known limit: it has no native int8 quantisation, so high-dimensional embeddings consume RAM linearly (Encore, pgvector vs Qdrant 2026; Katz, Scalar and binary quantization for pgvector).
Qdrant (a dedicated vector engine). It supports int8 scalar quantisation (float32 → int8, 4× less memory) and product quantisation, native sparse vectors and RRF fusion for hybrid search. It is more efficient in memory and in quantisation; the cost is operating one more system alongside Postgres (Markaicode, pgvector vs Qdrant 2026).

The practical rule: pgvector if you already have Postgres and the corpus fits comfortably in RAM (a single database to operate and back up); Qdrant if memory efficiency and int8 quantisation are critical because of the corpus size. Synchronisation between the relational truth (Postgres) and the index (Qdrant) when both are used is detailed in PostgreSQL + Qdrant in microservices.

A cross-cutting stage — Incremental ingestion and idempotency

A living corpus changes: documents are added, edited, deleted. Re-indexing everything each night is expensive and causes windows of unavailability. The alternative is incremental ingestion with two pillars:

Idempotency by doc-id + hash. Each chunk is identified deterministically ({doc_id}_{chunk_index}) and each document carries a content hash. On reprocessing, if the hash has not changed, nothing is re-embedded: the compute is saved. If it changed, the old chunks for that doc_id are deleted and the new ones are upserted. An upsert with a deterministic id is idempotent: reprocessing an event twice does not create duplicates.
CDC (Change Data Capture). Instead of polling, Debezium reads the PostgreSQL WAL and propagates additions, edits and deletions to the index in near real time. A deletion in Postgres triggers the deletion of that document’s chunks in the vector store, avoiding the “ghost documents” that contaminate retrieval. The deep dive is in PostgreSQL + Qdrant in microservices and in the one on Debezium and CDC.

The arithmetic: sizing the corpus and the index

Two calculations you need to be able to do before provisioning anything.

Corpus sizing and ingestion time

Suppose a corporate corpus of $N_{docs} = 50{,}000$ documents, averaging 20 pages and 500 useful tokens per page after cleaning. The total tokens in the corpus:

$$T_{corpus} = N_{docs} \times \text{pages} \times \text{tokens/page} = 50{,}000 \times 20 \times 500 = 5 \times 10^{8} \text{ tokens}$$

Five hundred million tokens. Applying the overlap factor from the chunking example (~1.23×, the 23% extra chunks from the 20% overlap) and an effective size of 512 tokens/chunk, the number of chunks is:

$$N_{chunks} \approx \frac{T_{corpus}}{\text{step}} = \frac{5 \times 10^{8}}{410} \approx 1.22 \times 10^{6} \text{ chunks}$$

That is, ~1.22 million chunks to embed. At a conservative CPU embedding throughput of 3,000 tok/s per Xeon server in int8, the same figure we use in the RAG-on-CPU piece, the ingestion time for the first full corpus on one box is:

$$t_{ingesta} = \frac{T_{corpus}}{\text{throughput}} = \frac{5 \times 10^{8}}{3{,}000} \approx 1.67 \times 10^{5} \text{ s} \approx 46 \text{ hours}$$

46 hours on a single box sounds bad, but ingestion is embarrassingly parallel: the corpus is shared out. With 8 CPU servers it drops to ~6 hours, comfortably within a weekend window for the initial load; and subsequent incremental ingestions (only what changed) take minutes. Layout-aware parsing adds its own cost, since Docling with a VLM is slower than PyMuPDF, but it is also batch and parallelises the same way.

Vector index size

Each vector has dimension $d = 1024$ (that of bge-m3). In float32 (4 bytes/dimension), each vector takes:

$$\text{bytes}_{fp32} = d \times 4 = 1024 \times 4 = 4096 \text{ B} = 4 \text{ KB}$$

For the 1.22 M chunks, the dense vectors alone in fp32:

$$\text{size}_{fp32} = N_{chunks} \times d \times 4 = 1.22 \times 10^{6} \times 4096 \text{ B} \approx 5.0 \text{ GB}$$

In int8 (1 byte/dimension), each vector takes 1 KB and the total falls 4×:

$$\text{size}_{int8} = N_{chunks} \times d \times 1 = 1.22 \times 10^{6} \times 1024 \text{ B} \approx 1.25 \text{ GB}$$

To this you must add the HNSW index (~1.2× the size of the vectors for $m=16$) and the payload (metadata + chunk text, ~500 B/chunk → ~0.6 GB). In round numbers:

Configuration	Vectors	HNSW (~1.2×)	Payload	Total
fp32	~5.0 GB	~6.0 GB	~0.6 GB	~11.6 GB
int8	~1.25 GB	~1.5 GB	~0.6 GB	~3.4 GB

The reading: a corpus of 50,000 documents fits in the RAM of a single node even in fp32, and in int8 (Qdrant) it fits with room to spare, which keeps search latency in single-digit milliseconds. int8 quantisation is almost always the balance point, saving 4× with a recall loss typically below 1%. (These numbers are order-of-magnitude, with the constants and assumptions stated; they are for sizing, not for pinning down an invoice.)

Applied to the generic 4×H100 cluster

Let us bring this down to the series’ cluster: 4×H100 SXM 80 GB plus a generic CPU fleet (Xeon with AMX, NUCs). The correct split for ingestion is almost all CPU, with the GPU as an occasional exception:

Cleaning, dedup, chunking, metadata, ingestion embedding and indexing → the CPU fleet. All of this is batch work, throughput-bound, with no latency SLA. It is exactly the “data plane” the RAG on CPU piece talks about: no H100 should spend a cycle chunking documents or building an HNSW index (which was always CPU by design). The nightly incremental re-indexing of a corpus that changes on the scale of hours is the textbook case of “CPU work in no hurry”.
The GPU only comes in at two points. First, in VLM parsing: if the corpus has scans, dense tables or forms and you choose Granite-Docling-258M or another vision-language model, that parsing can be accelerated on GPU, although at 258M parameters it is light and, at moderate volumes, runs on CPU without drama. Second, in the large embedder: if retrieval quality demands a 7B embedder (gte-Qwen2, NV-Embed) instead of bge-m3 (568M), that embedder is once again an LLM-class model and lives where the 7B models live, on the GPU.
The 4×H100 are reserved for generation. As throughout the series, the expensive and scarce silicon is kept for what is latency-bound, the LLM producing the answer, and at most for the peaks of VLM parsing or 7B embedding the CPU cannot absorb. To give a sense of the ceiling: a 4×H100 node serving bge-m3 through TEI runs at around ~2,000 chunks/s, against the thousands of tok/s of a Xeon in int8; but using the H100s for daily ingestion means spending the resource the whole organisation fights over on a job the CPU fleet does overnight without anyone missing it.

The sentence that sums up the split: ingestion is the archive’s cataloguing chain, and almost all of it is done by patient, cheap staff (CPU); the star writer (GPU) is only disturbed when there is a page no classic OCR can decipher.

Closing: quality is decided upstream

The recurring error in mediocre RAG is not in the reranker or the prompt: it is in an ingestion that parsed a table badly, did not weed out the duplicates, or chunked at a size that destroys context. Garbage-in, garbage-out: no downstream component fixes what ingestion spoiled upstream. Investing in the cataloguing chain, with layout-aware parsing where it is needed, real dedup, measured chunking, complete metadata, is what moves the needle on system quality the most, and almost all of it fits on the CPU fleet. The GPU, like the star writer, should only touch the corpus when its brain is genuinely required.

References

IBM — Granite-Docling: End-to-end document understanding (Granite-Docling-258M, Apache 2.0, DocTags, enero 2026). https://www.ibm.com/new/announcements/granite-docling-end-to-end-document-conversion
ibm-granite — granite-docling-258M model card (VLM ~258M, Granite 3 + SigLIP2). https://huggingface.co/ibm-granite/granite-docling-258M
Docling project — Docling: Get your documents ready for gen AI (layout, tablas, multi-formato). https://github.com/docling-project/docling
Unstructured — PDF Parsing Strategies for RAG (fast / hi_res / VLM / auto; chunking by_title). https://unstructured.io/blog/mastering-pdf-transformation-strategies-with-unstructured-part-2 · https://docs.unstructured.io/
Omdena — Document Parsing for RAG: A Complete Guide for 2026 (PyMuPDF, TOC, pipeline). https://www.omdena.com/blog/document-parsing-for-rag
Firecrawl — Best Chunking Strategies for RAG (and LLMs) in 2026 (benchmark Vecta feb-2026: recursive 512 → 69%, semántico 54%; context cliff ~2.500 tok). https://www.firecrawl.dev/blog/best-chunking-strategies-rag
Günther, M., et al. — Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models. arXiv 2409.04701. https://arxiv.org/abs/2409.04701 · Jina AI: https://jina.ai/news/late-chunking-in-long-context-embedding-models/
Brenndoerfer, M. — MinHash: Jaccard Similarity, LSH, and Near-Duplicate Detection. https://mbrenndoerfer.com/writing/minhash-algorithm-jaccard-similarity-lsh-deduplication
Zilliz — Data Deduplication at Trillion Scale (MinHash LSH dominante en limpieza de corpus). https://zilliz.com/blog/data-deduplication-at-trillion-scale-solve-the-biggest-bottleneck-of-llm-training
Ragnarök / TREC RAG 2024 — near-duplicados en MS MARCO V2 degradan recuperación y diversidad. https://arxiv.org/pdf/2406.16828
Encore — pgvector vs Qdrant in 2026 (halfvec, sparse, límites de cuantización de pgvector). https://encore.dev/articles/pgvector-vs-qdrant
Markaicode — pgvector vs Qdrant (2026 Benchmarks) (Qdrant int8 scalar quantization 4×). https://markaicode.com/vs/pgvector-vs-qdrant/
Katz, J. — Scalar and binary quantization for pgvector. https://jkatz05.com/post/postgres/pgvector-scalar-binary-quantization/
Hugging Face — Text Embeddings Inference (TEI), backends CPU/GPU, endpoints OpenAI-compatibles. https://github.com/huggingface/text-embeddings-inference

Taking RAG to the CPU: separating the data plane from the generation plane

Thu, 11 Jun 2026 03:20:00 +0000

Third piece in an operational series about squeezing a generic on-premise LLM cluster of 4×H100 SXM 80 GB. Its siblings: sharing one GPU between workloads (time-slicing, MPS, MIG) and serving several models on one GPU (swap + sleep) attack the split inside the GPU. This one attacks the split outside: which parts of RAG never need to touch the GPU at all. The closing piece of the series, the end-to-end sovereign assistant (fourth instalment, in preparation), assembles the complete system where these pieces fit together.

TL;DR

A RAG system is not one thing, it is three phases with opposing compute profiles, and putting them all on the GPU “because it is AI” is an allocation mistake. (1) Build/ingest, embedding the corpus and building the index, is batch, throughput-bound work with no latency SLA: its natural home is the CPU. (2) Query-time retrieval, embedding the query, HNSW search, RRF fusion, lightweight rerank, is mostly CPU, with caveats only for heavy reranking; vector search always ran on CPU, even in stacks sold as “GPU”. (3) Generation, the LLM producing the answer, is latency-bound, and there the GPU is irreplaceable: a 7B on CPU gives a time to first token of seconds, unacceptable for chat. The technical key to why (1) and (2) fit on CPU: the embedder is not an LLM. bge-m3 is around 568M parameters (an XLM-RoBERTa encoder), not 7B+; in int8 it occupies about 580 MB and activates fast integer compute paths (Intel AVX-512 + VNNI + AMX on 4th-gen Xeon and later; NEON SDOT/UDOT on ARM). The runtimes are ready: TEI with a CPU backend (same OpenAI API /v1/embeddings and /rerank), Qdrant’s fastembed (ONNX-CPU), bge-m3 in ONNX int8 with its three heads (dense/sparse/ColBERT). The Intel + Hugging Face blog post with Optimum Intel and fastRAG reports up to around 10× on indexing for BGE-large int8 on a 4th-gen Xeon (their benchmark figure, encoding-only; I quote it and qualify it below). The operational conclusion: separate the data plane (CPU) from the generation plane (GPU). In the 4×H100 cluster, no H100 should be spent re-indexing a corpus that changes once a day. That goes to the generic CPU fleet (Xeon AMX, NUCs), and the H100s are reserved for generation and, at most, for rerank peaks or large 7B embedders. What does not go down to CPU: interactive generation, massive reranking at high QPS, re-indexing with a strict real-time SLA, and 7B embedders (gte-Qwen2, NV-Embed).

The analogy: the library and the librarian

Picture a serious research library. There are three distinct jobs, done by different people, on different clocks.

The first is cataloguing. Boxes of new books arrive; someone opens them, classifies them, assigns a shelf mark, indexes them in the catalogue and puts them on the right shelf. It is patient background work, done at night or between other tasks. Nobody is standing over you with a stopwatch waiting for today’s batch to be catalogued: what matters is that tomorrow it is done and done well. It is pure throughput: how many books you catalogue per hour, not how long you take on one specific book. This is ingestion.

The second is handling an enquiry at the desk. A reader turns up and asks about a topic. The librarian goes to the catalogue, which already exists, finds half a dozen relevant shelf marks, fetches them and puts the books on the desk. It is fast, light, and consists of searching an index that already exists, not building it. This is retrieval.

The third is writing a reasoned report from those books. The reader, or an expert you commission, reads the six books, compares them, synthesises, and writes an argued answer with citations. This is slow, demands a very well-trained mind, and the reader is waiting: here there is a human stopwatch. This is generation, the LLM.

The moral is about staff allocation. You do not put your star writer, expensive, scarce, with a queue of people waiting for reports, to catalogue boxes of books at dawn. Cataloguing is done by a large, cheap team that works through the night without rushing. The star writer touches only what genuinely needs their mind: writing. In our system, the star writer is the H100, and cataloguing at dawn is corpus ingestion. Spending the H100 on re-indexing is exactly the error of putting the writer to work labelling boxes.

The rest of the post is, essentially, which parts of the library work the cheap CPU team can do (almost all of them) and which is irreducibly the GPU writer’s job (only the last one).

The three phases and their compute profiles

The confusion that GPU over-provisioning feeds on is treating “RAG” as a monolithic block that “uses AI, therefore goes on the GPU”. No. RAG is a data pipeline with a generative model plugged in at the end. The correct architectural boundary does not separate “what uses models” from “what does not”, because both sides use models, but throughput-bound from latency-bound, which is the same as separating the data plane from the generation plane.

Why ingestion fits on CPU: the embedder is not an LLM

The whole argument rests on a size asymmetry that gets overlooked. People hear “embeddings” and “generation” and put them in the same bag of “big models that need a GPU”. But the embedding encoder and the generative LLM are two orders of magnitude apart in parameters.

bge-m3, the reference multilingual embedder, is an XLM-RoBERTa of around 568M parameters (model card, paper arXiv:2402.03216). Its sibling reranker, bge-reranker-v2-m3, is built on the same base and lands at roughly the same 568M parameters (model card). Compare that with an entry-level generative LLM: a Llama 3.1 8B has around 14× more parameters, and the big production models sit at 70B+. A 568M encoder is, in compute budget terms, a different animal.

Two structural differences make that encoder comfortable on CPU:

It is an encoder, not an autoregressive decoder. It processes the whole sequence in a single forward pass and emits the vector. There is no token-by-token decode, no growing KV cache, none of the memory-bound generation phase that kills the CPU. It is a dense matrix pass and that is it.
It quantises to int8 with almost no loss. In int8, bge-m3 occupies on the order of 580 MB and, above all, activates the integer compute paths that a modern CPU executes quickly: matrix instructions such as Intel AMX (Advanced Matrix Extensions, 4th-generation Xeon and later), AVX-512 with VNNI (Vector Neural Network Instructions) on earlier Xeons, and NEON SDOT/UDOT on ARM. The quality loss from going FP32 to int8 on these models usually stays below 1% of retrieval recall, practically invisible (Intel + Hugging Face, CPU Optimized Embeddings).

Let us quantify the int8 size. For $P = 568 \times 10^6$ parameters at 1 byte each:

$$\text{size}_{\text{int8}} \approx 568 \times 10^6 \text{ params} \times 1 \text{ byte/param} \approx 568 \text{ MB}$$

In other words, the model fits in the cache and RAM of any server or NUC without blinking, and the bottleneck is integer compute, exactly what AMX/VNNI accelerate. There is nothing in this profile that asks for a GPU.

Runtimes that already do this effortlessly

Nothing needs inventing. The CPU ecosystem for the data plane is mature:

Text Embeddings Inference (TEI) from Hugging Face: a Rust server with CPU backends via ONNX Runtime (recommended) or Intel MKL, and OpenAI-compatible endpoints (/v1/embeddings) plus /rerank (TEI repo). That is, the data plane on CPU exposes exactly the same HTTP contract as a GPU server; the rest of the system never learns which silicon is behind it.
fastembed from Qdrant: a lightweight library that loads embedders on ONNX-CPU and generates dense, sparse and ColBERT vectors (fastembed repo). Designed from the start to run without a GPU.
bge-m3 in ONNX int8 with its three heads (dense / sparse-lexical / ColBERT multivector) exported and quantised, ready for ONNX Runtime CPU.

The Intel and Hugging Face figure that anchors viability: in their benchmark with Optimum Intel + fastRAG on a 4th-generation Xeon (8480+, 56 cores, 1 socket), the int8 variant of BGE-large reaches up to around 10× indexing throughput against FP32 (HF blog, Haystack/deepset). The small print needs reading, so I read it: that ~10× is encoding-only (tokenisation excluded), at sequence length 256, comparing int8 against FP32 on the same CPU. It is not “CPU 10× faster than GPU”, it is “int8 10× faster than FP32 on CPU”. It is still the relevant figure: it tells you that with quantisation the CPU goes from unworkable to perfectly useful for batch ingestion.

Viability table: CPU for each component?

This is the operational table. The column that matters is the caveat, because a bare “yes” or “no” lies.

Component	CPU viable?	Caveat
Chunking (slicing the corpus)	Yes, always	It is regex, parsing and windows; it never had anything to do with the GPU.
Ingest embedding `bge-m3` dense	Yes, its best case	Overnight batch, int8 + AMX/VNNI. This is exactly what the CPU shines at.
Sparse / SPLADE / BM25 head	Yes, CPU-native	Lexical work is pure inverted index; the GPU adds nothing here.
Building the HNSW index (Qdrant, pgvector)	Yes, always CPU	The HNSW graph build is CPU by design in these engines.
Query embedding (online)	Yes	A single short text; tens of ms on CPU, more than enough for chat.
Dense + sparse search + RRF	Yes	Vector search always ran on CPU, even in “GPU” stacks. RRF is sorting lists.
Cross-encoder reranker `bge-reranker-v2-m3` top-20/50	Yes, with care	A cross-encoder evaluates $k$ query-doc pairs: cost $\propto k$. Over 20-50 candidates it works; over hundreds at high QPS, no.
ColBERT late interaction	Marginal on CPU	The token-by-token matrix product of late interaction is heavy; viable at low volumes, suffers with QPS.
LLM generation	No, in practice	A 7B on CPU gives TTFT in seconds. Interactive latency = GPU.

Two rows deserve underlining because they dismantle myths.

“Vector search needs a GPU.” False from the start. The HNSW index, the navigable small-world graph used by Qdrant, pgvector with vector/halfvec, Milvus in CPU mode and almost everything else, was always built and traversed on CPU. Even the stacks advertised as “GPU-accelerated RAG” do the embedding on GPU but the ANN search stays on CPU in the vast majority of deployments; the GPU variants of the index (CAGRA and similar) are the expensive exception, not the norm, and are justified only with billions of vectors and extreme QPS. For a corporate corpus of millions of chunks, HNSW on CPU resolves in single-digit milliseconds.

“The reranker is a model, therefore GPU.” The bge-reranker-v2-m3 reranker is a cross-encoder of around 568M: it runs on CPU. The caveat is the number of pairs. A cross-encoder does not produce a reusable vector; it evaluates the (query, document) pair jointly, so its cost grows linearly with the number of candidates $k$:

$$\text{cost}_{\text{rerank}} \propto k \times \text{forward}(\text{query} + \text{doc})$$

Reranking the top-20 or top-50 coming out of hybrid retrieval is perfectly affordable on CPU. Reranking hundreds of candidates at high QPS is not: there the linear cost explodes and the GPU wins. The rule of thumb: broad cheap recall in the retriever, precision rerank over few candidates. (The detail of hybrid retrieval and reranking is in the fundamentals piece linked below.)

The numbers, with honest methodology

Here comes the part where a lot of people lie by omission. I am going to give throughput ranges, but they are literature ranges and orders of magnitude, not my own measurements on this hardware. Take them as such: the right decision does not depend on nailing the number, it depends on understanding the allocation.

For bge-m3 dense, sequence ≈256 tokens, embedding throughput moves roughly like this:

Platform	Dense throughput (order of magnitude)	Reading
High-end GPU (5090 fp16, TEI)	~12k tok/s+ (indicative)	The ceiling; expensive and scarce.
Large server CPU (Xeon ~56 cores, int8 ONNX)	low band of thousands tok/s	~1/5–1/10 of the GPU, but horizontally scalable and cheap.
Edge CPU / NUC (4-8 cores, int8)	tens to low hundreds tok/s	Enough for overnight ingestion of a local corpus.

The temptation is to read the second row as “CPU is 5-10× slower, discarded”. That is the wrong reading for ingestion. For batch work with no SLA, what rules is not absolute tok/s but throughput per euro and throughput per watt, and there the arithmetic changes sign.

Take a numerical example of allocation. Suppose a corpus of 2 million chunks of around 256 tokens that has to be re-indexed once a day (the corpus changes, the embeddings have to be redone). That is:

$$2 \times 10^6 \text{ chunks} \times 256 \text{ tok/chunk} \approx 5.1 \times 10^8 \text{ tokens}$$

At a conservative CPU throughput of, say, 3,000 tok/s per int8 Xeon server:

$$t_{\text{ingest}} \approx \frac{5.1 \times 10^8 \text{ tok}}{3000 \text{ tok/s}} \approx 1.7 \times 10^5 \text{ s} \approx 47 \text{ hours on a single server}$$

47 hours on one box sounds bad until you remember two things. First, this is embarrassingly parallel: the corpus is sliced and distributed; with 8 CPU servers it drops to around 6 hours, with 16 to around 3 hours, comfortably inside the overnight window. Second, and more importantly: that same work on the GPU blocks the GPU. If the H100 does 12k tok/s, it takes around 12 hours, but they are 12 hours of the H100, the resource the whole organisation fights over in order to generate. Spending the scarce, expensive resource on re-indexing a corpus that changes once a day is bad allocation, even if it is “faster”: you are optimising the wrong tok/s.

The mental rule: for ingestion, optimise throughput/€ and throughput/W; absolute tok/s belongs to the generation plane, where the human stopwatch really is running.

Decision tree: CPU or GPU for this piece?

Reference architecture (a): CPU-only

The first case is a node with no GPU: a NUC, an office Xeon, a sovereign edge server in a branch or in an air-gapped environment. The whole data plane lives there; generation is delegated to a remote GPU endpoint or done in batch with an SLM when latency is not pressing.

The stack:

TEI-CPU serving bge-m3 int8 with dense + sparse (same OpenAI contract /v1/embeddings, plus /rerank for the reranker).
Qdrant with a dense HNSW index plus sparse vectors, native RRF fusion.
Reranker bge-reranker-v2-m3 over the top-k (via TEI’s /rerank).
Gateway that orchestrates and, for generation, calls an external endpoint.

# docker-compose: full RAG data plane on CPU (no GPU)
services:
 tei-embed:
 image: ghcr.io/huggingface/text-embeddings-inference:cpu-latest
 command: ["--model-id", "BAAI/bge-m3", "--pooling", "cls", "--dtype", "int8"]
 ports: ["8081:80"]
 # ONNX/MKL backend: exploits AVX-512+VNNI / AMX if the Xeon supports it

 tei-rerank:
 image: ghcr.io/huggingface/text-embeddings-inference:cpu-latest
 command: ["--model-id", "BAAI/bge-reranker-v2-m3", "--dtype", "int8"]
 ports: ["8082:80"]
 # exposes /rerank — invoked ONLY over top-20/50, never over hundreds

 qdrant:
 image: qdrant/qdrant:latest
 ports: ["6333:6333"]
 volumes: ["./qdrant_storage:/qdrant/storage"]
 # HNSW dense + sparse vectors + RRF, all CPU

Hybrid search with RRF fusion in Qdrant (dense + sparse in a single query):

from qdrant_client import QdrantClient, models

client = QdrantClient(url="http://qdrant:6333")

# query embedding: dense and sparse from TEI-CPU (HTTP wiring omitted)
hits = client.query_points(
 collection_name="corpus",
 prefetch=[
 models.Prefetch(query=dense_vec, using="dense", limit=50),
 models.Prefetch(query=sparse_vec, using="sparse", limit=50),
 ],
 query=models.FusionQuery(fusion=models.Fusion.RRF), # native RRF
 limit=20,
).points
# -> then: POST /rerank (TEI) over these 20, keep the top-5
# -> then: the gateway sends query + top-5 to the GENERATION endpoint (GPU)

Generation, on this CPU-only node, leaves the node: the gateway builds the augmented prompt and sends it to a vLLM endpoint on the GPU cluster (or, if there is no interactive SLA, to an SLM on CPU in batch mode, accepting a TTFT of seconds). The entire data plane, everything above, runs without a single GPU.

Reference architecture (b): the recommended hybrid

This is the one I recommend for the general case with a GPU cluster available: data plane on CPU, generation plane on GPU, connected by OpenAI-compatible HTTP contracts so that each side is replaceable.

Generation server, minimal, on GPU:

# vLLM on the GPU cluster — generation ONLY
services:
 vllm-gen:
 image: vllm/vllm-openai:latest
 command: >
 --model meta-llama/Llama-3.1-8B-Instruct
 --dtype bfloat16 --max-model-len 8192
 --gpu-memory-utilization 0.85
 # exposes /v1/chat/completions — the gateway sends it query + the top-5 already retrieved
 deploy:
 resources:
 reservations:
 devices: [{driver: nvidia, count: 1, capabilities: [gpu]}]

The virtue of the design: because both sides speak the OpenAI contract over HTTP, the CPU data plane and the GPU generation plane scale separately and are replaceable. If tomorrow you want to move the rerank to GPU because QPS went up, you change one URL. If you want to add more CPU ingest nodes, you add them without touching generation. The whole stack is OSS and licence-clean: bge-m3 and bge-reranker-v2-m3 are MIT (bge-m3, reranker), Qdrant is Apache-2.0, TEI and vLLM are OSS.

Applied to the generic 4×H100 cluster

Let us bring this down to the cluster in the series: 4×H100 SXM 80 GB plus a generic CPU fleet (Xeon with AMX, NUCs). The correct allocation:

Build and indexing → CPU fleet. No H100 should spend a cycle re-embedding the corpus. That goes to the Xeon AMX machines (large servers, throughput in the thousands of tok/s in int8) or, for small local corpora, to the NUCs overnight. Overnight re-indexing of a corpus that changes once a day is the textbook case of “unhurried CPU work”.
The H100s → generation. All four cards are reserved for what only they do well: producing tokens at interactive latency. This is what the sibling pieces in the series, sharing a GPU and several models on one GPU, help to squeeze: once ingestion stops competing for the GPU, all the expensive silicon is free to generate and is shared better between models and tenants.
The H100s, at most, → rerank peaks or large embedders. If at some point you need a 7B embedder (gte-Qwen2, NV-Embed) for a domain where bge-m3 does not reach, or a massive rerank at a QPS the CPU cannot absorb, those peaks can indeed visit the GPU. But they are the occasional exception, not the base load.

The auditability angle: ENS / NIS2

There is a compliance argument that rarely gets mentioned and that the CPU/GPU split gives you almost for free.

A CPU-only node with no proprietary driver is easier to audit. There is no closed NVIDIA kernel stack, no CUDA and firmware versions to reconcile with the supply chain, no proprietary driver surface to document for an ENS or a NIS2 assessment. The whole data plane, chunking, embeddings, index, search, runs on OSS software on generic CPU with standard instructions. For a sovereign or classified environment, being able to say “the plane that touches the corpus depends on no proprietary binary” is a real argument, not marketing.

And there is a second auditability angle intrinsic to RAG done well: source traceability. A RAG system that retrieves identifiable chunks and cites them is auditable, since you can reconstruct which document each claim came from, as opposed to context-stuffing or the model’s opaque parametric knowledge, where there is no way to know where a fact came from. That traceability lives in the data plane (what was retrieved, from which source, with what score), precisely the plane we are putting on auditable CPU. The two arguments reinforce each other: the auditable silicon and the auditable chain of evidence are the same plane.

When NOT to take it to CPU

For honesty, and to avoid falling into the mirror image of the hype, the cases where CPU is not the answer:

Generation at interactive latency. The obvious case. A 7B on CPU gives TTFT in seconds: unacceptable for chat. If the user is waiting, generation goes to GPU. With no practical exceptions today.
Massive reranking at high QPS. A cross-encoder or ColBERT over hundreds of candidates, multiplied by many requests per second, saturates the CPU. The cost $\propto k \times \text{QPS}$ crosses the threshold where the GPU pays off. Keep CPU rerank bounded to top-20/50; if you need more breadth at higher QPS, move up to GPU.
Real-time re-indexing with a strict SLA. If the corpus changes continuously and freshness is measured in seconds (not hours), CPU throughput may not fit the window. There, ingest embedding may need a GPU, but note that this is rare: most corporate corpora change at the pace of hours or days, not seconds.
Large embedders (7B). bge-m3 (568M) is comfortable on CPU; a gte-Qwen2 or NV-Embed at 7B is LLM-class again and drags the same cost profile as generation. If your retrieval quality demands a 7B embedder, that embedder lives where 7Bs live: on the GPU.

The sentence that sums it all up: the CPU is the default home of the data plane; the GPU is the justified exception for what is latency-bound and massively online. Start by putting everything on CPU and move up to GPU only what proves it does not fit, not the other way round.

References

Chen, J., et al. BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings. arXiv 2402.03216. https://arxiv.org/abs/2402.03216
BAAI — BGE-M3 model card (568M params, XLM-RoBERTa, 8192 tokens, MIT). https://huggingface.co/BAAI/bge-m3
BAAI — bge-reranker-v2-m3 model card (cross-encoder on top of bge-m3, ~568M). https://huggingface.co/BAAI/bge-reranker-v2-m3
Intel + Hugging Face — CPU Optimized Embeddings with Optimum Intel and fastRAG (~10× indexing for BGE-large int8, 4th-gen Xeon). https://huggingface.co/blog/intel-fast-embedding
deepset / Haystack — CPU-Optimized Embedding Models with fastRAG and Haystack. https://haystack.deepset.ai/blog/cpu-optimized-models-with-fastrag
Hugging Face — Text Embeddings Inference (TEI), CPU ONNX/MKL backends, OpenAI-compatible endpoints. https://github.com/huggingface/text-embeddings-inference
Qdrant — fastembed (ONNX-CPU, dense/sparse/ColBERT) and Hybrid Search with RRF. https://github.com/qdrant/fastembed · https://qdrant.tech/documentation/beginner-tutorials/hybrid-search-fastembed/

Aggressive RAG in small models: trading parameters for retrieval

Tue, 09 Jun 2026 02:20:00 +0000

This post belongs to the series on inference performance in small models. Its sibling piece, The roofline inverts in small models, explains why the compute-bound prefill is the bottleneck that shapes the whole discussion here. It is worth reading first: here we assume that adding more context is not free.

TL;DR

An SLM (say 1B–8B parameters) knows fewer facts than a 70B–700B model, simply because it has fewer weights in which to memorise them. But its ability to reason over text in front of it, following instructions, extracting, synthesising, comparing, degrades far less with size than its encyclopaedic knowledge does. The operational consequence is direct: use the SLM as a reasoning engine over curated context, not as a database. Move knowledge from the weights into the context via retrieval. The problem is that “aggressive retrieval” is often read as “put in lots of chunks”, and that collides head-on with three facts about SLMs: shorter context windows, worse use of long context (the lost in the middle effect is more severe the smaller the model) and a compute-bound prefill whose cost grows with the context length $C$, linearly in the projections and quadratically in attention. You cannot simply add tokens. Each retrieved token is paid for twice, in degraded quality and in TTFT, and the SLM is the worst placed to absorb either cost. The way out is not to retrieve less, but to retrieve better: reranking for precision over recall, compressing the context before injecting it, prefix caching the stable documents, semantic caching of answers, and structured output with external tools that stand in for internal knowledge. This post works through the maths and gives a TTFT number before and after compressing a context from 4,000 to 1,000 tokens on an RTX 4090.

The analogy: the open-book exam

Two students sit the same exam. The first has a prodigious memory: they have memorised the entire syllabus, paragraph by paragraph. The second has an ordinary memory, forgetting dates, confusing names, but is allowed to bring in a crib sheet.

If the second student’s crib sheet is a chaos of piled-up photocopies, they lose: they take too long to find what they are after, get distracted by irrelevant pages and run out of time. But if their crib sheet is excellent, trimmed to the essentials, reordered by relevance, with the important things at the top and no filler, then they not only do not lose: they often win, because they reason just as well as the first student and, on top of that, work from verified material instead of from blurry recollections they may be inventing.

The moral has three layers, and each maps to an engineering decision:

Memorising everything is expensive. The first student invested months. A large model invests parameters, and VRAM, and inference FLOPs, in memorising facts.
The crib sheet matters more than its size. A well-made one-page crib sheet beats ten badly organised pages. More retrieved context is not better context: the material’s precision beats volume.
Knowing how to search and synthesise is a different skill from knowing. It is the one the SLM retains. The whole strategy consists of leaning on that skill and subcontracting the memory.

The rest of the post is, essentially, how to build an excellent crib sheet under the constraint that the student (the SLM) reads slowly and gets tired with long texts.

The capacity argument: how many facts fit in the weights

Let us start by justifying the thesis with an order of magnitude, not with faith. How much factual knowledge really fits in a model’s weights?

There is a recurring empirical estimate in the interpretability and memorisation literature: a dense model can store on the order of 2 bits of memorised information per parameter before saturating (the exact figure varies by study and training regime; take it as an order of magnitude, not as a law). An 8B-parameter model then has an information storage ceiling on the order of:

$$8 \times 10^9 \text{ params} \times 2 \text{ bits/param} = 1.6 \times 10^{10} \text{ bits} \approx 2 \text{ GB of information}$$

And that budget is not only for facts: the vast majority goes on grammar, syntax, reasoning ability, code, formatting, and only a fraction is left for encyclopaedic knowledge. Compare it with the other side: a retrievable corpus of several million documents, a corporate wiki, a document repository, a technical knowledge base, easily takes up hundreds of GB to terabytes of text, indexed and queryable with millisecond latency. The asymmetry is two or three orders of magnitude in favour of the external corpus.

The conclusion is not that the weights are useless, they are where reasoning lives, which is the expensive thing to replicate, but that competing with an external index on fact capacity is losing by construction. A 70B model has ~9× more memorisation budget than an 8B one, but it is still negligible against the corpus. That is why the large model also does RAG in production. The difference is that the SLM needs it: without retrieval, its factual knowledge is too sparse and, worse, prone to hallucinating exactly in the gaps it did not memorise.

The central tension: retrieving more is not putting in more

This is where most naive designs break. “Aggressive retrieval” sounds like a large top-k: if retrieval helps, retrieve 20 chunks instead of 5. But in an SLM that fails for two independent reasons, one of quality and one of cost.

(a) SLMs make worse use of long context

The lost in the middle effect (Liu et al., 2023) is well known: LLMs retrieve information placed at the beginning and end of the context better, and the information in the middle worse. What gets less emphasis is that the effect is more severe the smaller the model. An SLM has fewer attention heads, fewer layers and poorer internal representations for “tracking” a relevant fact buried at position 14 out of 20 chunks. Its nominal context window also tends to be shorter (4K–32K against the 128K+ of the large ones), and the effective window, the length beyond which quality collapses, is smaller still. Putting in 20 chunks does not mean the model reads all 20: it means it will probably ignore or misread the ones in the middle, while paying the cost of all of them.

(b) Prefill grows with the context and is compute-bound

This is the blow people underestimate. Prefill, processing the full prompt before emitting the first token, is the compute-bound phase of inference (unlike decode, which is memory-bound; the detail lives in The roofline inverts). Its cost grows with the context length $C$, and it determines TTFT (time to first token). More chunks → more prefill tokens → more TTFT and more compute cost per request. In an SLM, where prefill is proportionally more expensive relative to the model, this hurts especially.

The operational conclusion is uncomfortable but clear: you cannot compensate for fewer parameters simply by putting in more context. Each retrieved token is paid for twice, in degraded quality and in TTFT, and the SLM is the worst placed to absorb both costs. The way out is to retrieve less but better, and to compress what you retrieve.

The maths of prefill

Let us put numbers on “prefill grows with the context”. For a context of $C$ tokens, a transformer layer does two classes of work:

Linear projections (QKV, attention output, FFN): each token is multiplied by fixed-size weight matrices. The cost is $O(C)$ in FLOPs, linear in the number of tokens.
Attention ($QK^\top$ and the multiplication by $V$): each token attends to all the others. The cost is $O(C^2)$, quadratic in the number of tokens.

The total prefill cost per layer has the form:

$$\text{FLOPs}_{\text{prefill}} \approx \underbrace{a \cdot C}_{\text{projections}} + \underbrace{b \cdot C^2}_{\text{attention}}$$

with $a$ and $b$ constants that depend on the model dimension. For moderate contexts (a few thousand tokens) in an SLM, the linear term still dominates or is comparable to the quadratic one; the quadratic term becomes dominant at long contexts. What matters: if you compress the context $C \to C/k$, the linear term falls $\times k$ and the quadratic one falls $\times k^2$. Compression is the only lever that attacks both terms at once, and it attacks the worse one disproportionately.

Numerical example: TTFT before and after compressing, RTX 4090

Let us model TTFT as the time to process the prefill tokens at a given prefill throughput. Take an RTX 4090 (24 GB, Ada Lovelace) serving a quantised SLM, with a prefill throughput of ~5,000 tok/s (an illustrative figure; the real value depends on the model, the quantisation and the batch, so measure it, do not assume it).

Let there be a retrieved context of 4,000 tokens (8 chunks of ~500 tokens). Approximating TTFT as dominated by the context prefill:

$$\text{TTFT}_{\text{before}} \approx \frac{4000 \text{ tok}}{5000 \text{ tok/s}} = 0.80 \text{ s}$$

Now we compress that context to 1,000 tokens ($k = 4$). Prefill throughput is not constant with $C$, it drops a little at long contexts because of the quadratic term, but taking the conservative linear tokens/throughput approximation:

$$\text{TTFT}_{\text{after}} \approx \frac{1000 \text{ tok}}{5000 \text{ tok/s}} = 0.20 \text{ s}$$

TTFT falls from 0.80 s to 0.20 s, a $4\times$ reduction in the linear part. But the FLOPs arithmetic is even more favourable in the attention component: that part of the work falls $\sim k^2 = 16\times$. In practice total TTFT does not fall 16× because the cost is not purely quadratic at this scale, but the real reduction lies between 4× and something larger depending on how much attention weighed, and the aggregate compute saving (what pays the electricity bill and frees the GPU for another request) is substantially bigger than the plain 4× of the token count.

The argument generalises: compressing the context by a factor $k$ reduces TTFT by at least $\sim k\times$ and the attention cost by $\sim k^2\times$. For an SLM, where TTFT is often the SLA that matters, this is the difference between an assistant that answers instantly and one that feels slow.

The five levers to resolve the tension

The strategy is not “retrieve less and settle”. It is retrieve aggressively from the index and then distil aggressively what you retrieved before it reaches the SLM. Five levers, in order of application within the pipeline.

1. Aggressive reranking: precision over recall

The initial retriever (dense, sparse or hybrid) optimises recall: it brings back 50–100 candidates so as not to leave anything out. The reranker, a cross-encoder that sees the query and the document together, optimises precision: it reorders those candidates and you keep the 3–5 best. For an SLM this is not a luxury, it is structural: since the model uses long context badly, every chunk that goes in must earn its place. Better 4 chunks of very high relevance than 15 mediocre ones. The detail of hybrid retrieval and reranking is in Reranking and hybrid retrieval; here the rule is enough: maximise recall in the retriever, maximise precision in the reranker, and inject few.

2. Context compression: distilling the crib sheet

Once you have the best chunks, they still contain filler: padding sentences, redundancy, context irrelevant to the specific query. Context compression trims them before injection:

Extractive compression (LLMLingua / LongLLMLingua style, Jiang et al. 2023): a small model scores the perplexity or relevance of each token/sentence with respect to the query and removes the low-information ones, keeping the densest extractive subset. It reduces tokens without a second large generative model in the loop. LongLLMLingua adds position-aware reordering to mitigate lost in the middle.
Abstractive compression: a model summarises the retrieved chunks into shorter text. More aggressive in token reduction, but it introduces a generative step (cost and possible loss of fidelity).
Soft prompts / context distillation: compressing the retrieved context not into text, but into a handful of embeddings/soft tokens that the model consumes directly. It reduces the number of prefill tokens to a minimum, at the price of a trained, model-specific component.

The key point connects with the maths above: compressing what you retrieved by a factor $k$ reduces prefill tokens $\times k$, and therefore TTFT $\sim\times k$ and the attention cost $\sim\times k^2$. It is the lever with the best return when long context is the bottleneck.

3. Prefix caching of stable context

Not all the context changes between requests. System instructions, definitions, recurring reference documents, schemas: these are stable prefixes. Prefix caching stores the already-computed KV cache of those prefixes and reuses it, so that prefill only processes the new part (the query and the specific chunks). If 60 % of your context is stable, you save 60 % of that segment’s prefill on every hit. For it to work, the stable context must go at the start of the prompt (the KV cache is prefix-dependent) and it pays to maximise the hit rate; the hit-rate engineering detail is in Prefix cache hit rate. It combines especially well with RAG: retrieved documents that repeat across sessions get cached once.

4. Semantic caching of answers

A layer in front of the model: if a query is semantically equivalent to one answered before (embedding similarity above a threshold), return the cached answer and skip the model entirely, retrieval, prefill and decode included. In real workloads with long tails of repeated or near-repeated questions (FAQ, support), the saving is enormous because it eliminates the full cost, not just the prefill one. The trap is the threshold: too lax and you serve wrong answers to similar-but-different questions. The design is in Semantic caching for RAG.

5. Structured output and function calling: lean on tools, not on memory

The last lever changes what the SLM depends on. Instead of asking it to know a fact (its weak point), make it call a tool that knows it: a database query, an API, a calculator, a validator. Structured output (forcing JSON conforming to a schema) and function calling turn the SLM into an orchestrator that extracts arguments from the context and delegates the computation or the query. A reasonably capable SLM emits a well-formed tool call far more reliably than it recalls a specific fact. This reduces the pressure on parametric knowledge and on retrieval: for structured, fresh data (prices, inventory, states), querying beats retrieving text and beats memorising. The fundamentals are in Structured output and Function calling.

The complete pipeline

The five levers are not alternatives: they chain together. The flow, with the token counter falling at each step:

The order matters. Retrieving aggressively (high recall) before filtering guarantees that the right material is among the candidates; reranking and compressing afterwards guarantees that only the dense and relevant material pays the prefill toll; caching wraps everything so as not to repeat work. The SLM only sees the final crib sheet, short and ordered.

Implications for on-premise inference

The mental trap to avoid: treating the SLM as a large model with less quality. It is not. It is a different cost profile that rewards a different design. Three practical consequences:

The token budget is a first-class resource. With a large model with a 128K window, “putting in a bit more” is cheap relative to the model. With an SLM, every context token shows up in TTFT and in quality. Treat context size as a quantity to minimise subject to covering the answer, not to maximise.
The investment is worth it precisely because the model is cheap. Reranker, compressor and caches add complexity, but the model they serve is economical enough to run many replicas. The bottleneck shifts from the model to the data pipeline, which is exactly where you want it.
Retrieving does not replace adapting; they combine. For deep, recurring domain knowledge, adapting the SLM with LoRA (see the sibling QLoRA and aggressive multi-LoRA) can put part of the knowledge “into the weights” cheaply, reducing what has to be retrieved. Aggressive RAG and aggressive adaptation do not compete: the first gives freshness and citability, the second gives fluency and domain formatting. A good design uses both.

On the RTX 4090 (24 GB, Ada Lovelace)

The canonical scenario: a quantised SLM (4B–8B in INT4/FP8) fits with room to spare, leaving VRAM for a generous KV cache, essential for prefix caching, and for the reranker (a cross-encoder of a few hundred MB). The LLMLingua-style extractive compressor runs on a separate small model or on CPU. The TTFT calculation above (0.80 s → 0.20 s compressing 4× at ~5,000 tok/s) is representative of this card. The rule of thumb: if TTFT goes above your SLA, the first adjustment is to compress the context, not to change the model.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

With 320 GB and native FP8 prefill is much faster, so the temptation is to relax the token discipline. That is not quite advisable: the lever changes from TTFT to aggregate throughput. Compressing the context not only speeds up each request but frees prefill compute to serve more requests per GPU; compute-bound prefill is exactly the resource that saturates first under load. Here prefix caching and semantic caching, shared across replicas, are what pay off most: at high QPS, the prefill work you avoid by caching is pure throughput you gain. The SLM is still the cheap reasoning engine; the difference is that you now run many in parallel and the data pipeline is what decides how many requests fit.

What we have not covered

Evaluating compression: how to measure that compressing $k=4$ does not throw away correct answers (faithfulness, answer recall over a set of questions with ground truth).
Query-aware versus query-agnostic compression: compressing before or after knowing the question changes what can be cached and what can be discarded.
Chunking and granularity: chunk size interacts with reranking and compression; that is left for the corpus curation post.
Multi-hop and agents: when a question requires several retrieval rounds, the token budget is split across hops and compression discipline becomes critical.

References

Lewis, P., et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS 2020. https://arxiv.org/abs/2005.11401
Liu, N.F., et al. Lost in the Middle: How Language Models Use Long Contexts. TACL 2024. https://arxiv.org/abs/2307.03172
Jiang, H., et al. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models. EMNLP 2023. https://arxiv.org/abs/2310.05736
Jiang, H., et al. LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression. ACL 2024. https://arxiv.org/abs/2310.06839

Debezium and CDC: the notary who hears changes before anyone asks

Thu, 04 Jun 2026 11:00:00 +0200

TL;DR

Change Data Capture (CDC) with Debezium listens to the PostgreSQL Write-Ahead Log and turns every INSERT, UPDATE and DELETE into a structured Kafka event. Unlike traditional polling (SELECT ... WHERE updated_at > ?), it detects deletes, has latency in the tens of milliseconds and adds no extra load to the database. In RAG pipelines, this means that when a document is deleted from Postgres, the Qdrant chunks disappear too, automatically and in real time. The supporting infrastructure is modest: the connector uses 2-4 cores and 4-8 GB RAM to process thousands of events per second.

The master analogy: the land registry notary

Picture the land registry. Every time a flat is sold, mortgaged, or a mortgage is cancelled, the registrar records the operation in the registry ledger, a chronological and immutable journal. If you want to know what has changed in the registry, you have two options.

Option A (polling): you send someone every 5 minutes with a list of properties to ask “has anything changed?”. Problems: if a title was cancelled (DELETE), the property no longer exists by the time your envoy arrives, and there is no trace of it. If 20 different departments are doing the same thing, that is 20 people pestering the registrar every 5 minutes. And the minimum latency is the interval: 5 minutes.

Option B (Debezium): you hire a notary who sits right at the registrar’s desk. Every time the registrar signs an operation into the ledger, the notary notes it down immediately and notifies whoever needs to know. Title cancellations included, because the notary sees them as clearly as any other operation: he was there when it was signed.

In this analogy:

The registry ledger is the PostgreSQL WAL (Write-Ahead Log).
The notary is the Debezium connector.
The notary’s bookmark, which guarantees he loses no page even if he steps out for a moment, is the logical replication slot.
The messenger who carries the notifications to interested parties is Kafka (or Redpanda, or NATS JetStream).

We will pick this thread up in every section. When a technical detail is unclear, go back to the image of the notary.

1. The problem CDC solves

The most common synchronisation pattern between services sharing PostgreSQL is periodic polling:

SELECT id, content, updated_at
FROM documents
WHERE updated_at > $1
ORDER BY updated_at
LIMIT 1000;

This pattern has three structural problems.

DELETEs are invisible. When you delete a row, updated_at is not updated: the row is gone. The next time the poller queries, the row does not exist and there is no way to know it ever did. In a RAG pipeline, this means orphan chunks in Qdrant: the document no longer exists in Postgres, but its vectors keep polluting search results.

The minimum latency is the interval. If the poller runs every 5 seconds, average latency is 2.5 seconds. For near-real-time synchronisation (dashboards, alerts, RAG with frequently changing documents), that is too much.

Load scales with the number of consumers. If 10 services poll the same table every 5 seconds, that is 10 × 12 = 120 queries/minute producing no useful work; they only check whether there is anything new. On large tables with complex indexes, that is real load on the database.

CDC inverts the model: the database notifies, the consumers listen. Zero polling, zero extra load, DELETEs included, latency in the tens of milliseconds.

2. What the PostgreSQL WAL is

The operations journal

The Write-Ahead Log (WAL) is the chronological, immutable record of every operation Postgres performs. Before modifying any data page on disk, Postgres writes the operation to the WAL. That sequence, log first then data, is what guarantees durability (the D in ACID) and enables crash recovery: if Postgres dies mid-transaction, on restart it replays the WAL to return the database to a consistent state.

The WAL is the registry ledger of our analogy: chronological, immutable, complete.

Physical vs logical replication

PostgreSQL supports two WAL-based replication modes:

Physical replication: it replicates disk blocks verbatim. The standby receives the same bytes as the primary. It serves high availability and failover, but the target must be an exact copy of Postgres; you cannot send the changes to an external application.
Logical replication: instead of disk blocks, it replicates semantic operations: “the row with id=42 was inserted into table documents with these values”. The target can be anything that understands the protocol: another Postgres, Debezium, or any custom consumer.

CDC uses logical replication. It is what lets Debezium understand “what changed and in which table” rather than “which disk block changed at which offset”.

The replication slot: the notary’s bookmark

A logical replication slot is a persistent cursor in the WAL. Postgres keeps a record of how far into the WAL each slot has consumed. While a slot exists, Postgres guarantees it will not discard the WAL segments the slot has not yet read.

This is exactly the notary’s bookmark: even if the notary goes out for lunch, the ledger stays open at the last page he read. When he comes back, he carries on from exactly where he left off, having lost nothing.

The risk is the inverse: if the notary never comes back, the bookmark stops the registrar from archiving the old pages. If the Debezium connector goes down and does not recover for hours, the WAL grows indefinitely on disk until the slot is deleted manually or the consumer starts consuming again. This is called WAL disk blowup and is the most important operational risk of Debezium.

Mandatory monitoring:

SELECT slot_name,
 confirmed_flush_lsn,
 pg_current_wal_lsn(),
 pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn) AS lag_bytes
FROM pg_replication_slots
WHERE slot_type = 'logical';

The pgoutput plugin

The WAL stores operations in an internal binary format. For Debezium to understand them, Postgres needs to decode them into a readable format. The pgoutput decoding plugin, included in the Postgres core since version 10, does exactly that: it translates the binary WAL events into messages with the before/after structure of each row.

Debezium uses pgoutput by default. It requires no external extensions (unlike the wal2json plugin that was popular before Postgres 10).

3. Debezium architecture

The connector as a Kafka Connect plugin

Debezium is not a standalone service; it is a plugin for the Kafka Connect framework. Kafka Connect manages the connector’s lifecycle (start, stop, reconnection, offset tracking) and provides the parallelism and fault tolerance infrastructure.

The connector talks to Postgres through the logical replication protocol (not over JDBC), using the credentials of a user with the REPLICATION role.

PostgreSQL (WAL + pgoutput)
│
│ logical replication protocol
▼
Debezium Connector (Kafka Connect worker)
│
│ Kafka Producer API
▼
Kafka topic: rag.public.documents
│
▼
Consumer (Qdrant sync, audit log, fine-tuning pipeline...)

Structure of a Debezium event

Each change in the table becomes a JSON message with this structure:

INSERT ("op": "c" — create):

{
 "before": null,
 "after": {
 "id": 42,
 "content": "Contrato de arrendamiento...",
 "tenant_id": "acme",
 "updated_at": 1748934000000
 },
 "op": "c",
 "source": {
 "version": "2.7.0.Final",
 "connector": "postgresql",
 "db": "rag_db",
 "schema": "public",
 "table": "documents",
 "lsn": 29823948,
 "txId": 1047,
 "ts_ms": 1748934000123
 }
}

DELETE ("op": "d"):

{
 "before": {
 "id": 42,
 "content": "Contrato de arrendamiento...",
 "tenant_id": "acme",
 "updated_at": 1748934000000
 },
 "after": null,
 "op": "d",
 "source": { "lsn": 29824102, "txId": 1051, "ts_ms": 1748934060200 }
}

The before field holds the row’s previous state, available because Postgres can be configured with REPLICA IDENTITY FULL to include the complete row in the WAL on delete or update. Without that setting, before contains only the primary key.

This is the key for the RAG pipeline: the DELETE event carries the document id. The consumer uses it to delete every associated chunk in Qdrant with a doc_id = 42 filter. Without CDC, those chunks would never have been deleted.

Initial snapshot

When the connector starts for the first time (or after a reset), it cannot begin consuming the WAL from “the beginning of time”, only from the moment the slot was created. How does it guarantee the consistency of the initial state?

Through a transactional snapshot: the connector opens a transaction in REPEATABLE READ mode, exports the snapshot ID (pg_export_snapshot()), and does a full SELECT of the configured tables inside that transaction. It then starts consuming the WAL from the snapshot’s LSN. That way there is no gap: the snapshot covers the state up to an instant, and the WAL covers from that instant onwards.

SMT transformations (Single Message Transforms)

Before emitting the event to the Kafka topic, the connector can apply inline transformations called SMT. Common use cases:

Filtering sensitive columns (ReplaceField with blacklist): strip password_hash, phone_number before they reach the topic.
Adding metadata (InsertField): enrich the event with a tenant_id extracted from the original HTTP header (if it is in the row).
Conditional routing (Filter): drop events from rows with status = 'draft' before emitting them.

SMTs are pure configuration, requiring no code, and they are applied inside the connector process with no perceptible extra latency.

4. Debezium vs the Outbox pattern

The Outbox pattern is the most common alternative to pure CDC. Instead of emitting events directly to Kafka, the application writes to an outbox table in Postgres within the same transaction that modifies the data. A separate worker reads that table and publishes the events.

Criterion	Debezium (pure CDC)	Outbox pattern
Event latency	~50-200 ms from commit	Depends on the worker interval (typically 1-5 s)
Consistency	At-least-once	At-least-once
DELETE detection	Native (the DELETE event includes `before`)	Only if the app writes to the outbox on delete
Setup complexity	High (Kafka Connect, replication slot, permissions)	Low (an extra table plus a simple worker)
Infrastructure dependency	Requires Kafka/Redpanda/NATS JetStream	Postgres plus a worker; Kafka optional
WAL disk blowup risk	Yes, if the slot stops consuming	No
Schema visibility	Reads the table’s real schema	The event schema is defined by the app
Schema migration	Needs care (events reflect DDL changes)	More flexible (the event is whatever the app puts there)
When to use it	When you need DELETEs, low latency, or you cannot modify the app	When the app controls the event domain and infrastructure is limited

Rule of thumb: if you control the application code and do not need native DELETEs, the Outbox is simpler. If you do not control the code (legacy database, third-party application) or DELETEs are critical (a RAG pipeline with document deletion), Debezium is the right choice.

5. The maths

Throughput

Debezium on a connector with 4 workers can process between 10,000 and 50,000 events/second on modest hardware (4 cores, 8 GB RAM). The real bottleneck is not the connector but the Kafka broker: with 3 brokers and suitable partitions, Kafka can comfortably sustain 500,000 messages/second with 1 KB messages (source: public Confluent benchmarks, 2023).

For a typical RAG pipeline with 100 documents modified per minute:

$$\text{events/s} = \frac{100}{60} \approx 1{.}7 \text{ events/s}$$

That is 0.0034% of the connector’s capacity. Debezium will not be the bottleneck in any realistic RAG scenario.

End-to-end latency

The path from a commit in Postgres to an upsert in Qdrant has these stages:

Stage	Typical latency
Commit in Postgres → WAL written	< 1 ms (synchronous with the commit)
WAL written → Debezium reads it (WAL lag)	10-50 ms
Debezium → Kafka produce (ack)	5-20 ms
Kafka → Consumer (poll interval)	0-100 ms (configurable)
Consumer → Qdrant upsert/delete	5-15 ms
Typical total	30-200 ms

With fetch.min.bytes=1 and fetch.max.wait.ms=10 on the consumer, the Kafka poll latency drops to ~10 ms. The realistic range for an optimised pipeline is 30-100 ms.

WAL disk blowup risk

If the connector stops consuming, Postgres retains the WAL from the slot’s confirmed_flush_lsn onwards. The retained volume grows linearly with time and the write rate:

$$\text{retained WAL} = \text{write rate} \times \text{average WAL event size} \times \text{time without consuming}$$

Example with a moderate load (50,000 writes/hour, 500 bytes average per WAL event):

$$50{,}000 \times 500 \text{ B} \times 1 \text{ h} = 25 \text{ MB/h}$$

With a high load (1,000,000 writes/hour):

$$1{,}000{,}000 \times 500 \text{ B} \times 1 \text{ h} = 500 \text{ MB/h}$$

If the connector is down for 48 hours under high load: 24 GB of retained WAL. That can fill the disk and block Postgres completely.

Recommended alert: set an alert when lag_bytes > 1 GB or when confirmed_flush_lsn has not advanced for more than 15 minutes. See the monitoring query in section 2.

6. Use cases in LLMOps / RAG

RAG synchronisation with real deletion

This is the use case that most clearly justifies Debezium over polling. The flow:

A user deletes document id=42 from the document management interface.
Postgres runs DELETE FROM documents WHERE id = 42.
Debezium detects the DELETE in the WAL and emits the event with "op": "d" and "before": {"id": 42, ...}.

The consumer receives the event and runs:

qdrant_client.delete(
 collection_name="documents",
 points_selector=Filter(must=[FieldCondition(key="doc_id", match=MatchValue(value=42))])
)

Every chunk with doc_id=42 disappears from Qdrant in ~100 ms.

Without Debezium, those chunks would stay indefinitely, polluting retrieval results with fragments of documents that no longer exist in the source of truth.

Event sourcing for fine-tuning datasets

Every time a human annotator updates a row in the annotations table (correcting an LLM output), Debezium emits the UPDATE with before and after. The consumer writes the pair (original_output, correction) into the dataset curation pipeline, with no need for the annotator to do anything beyond saving in the interface. The fine-tuning pipeline knows exactly what changed and when, with no polling and no risk of duplicates from overlapping time windows.

Immutable audit log

WAL events are, by definition, the most faithful record of what happened in the database: they are the same data Postgres uses for crash recovery. Kafka with long retention (90 days, or size-based retention) serves as an immutable audit log without modifying the application schema or adding triggers. This is particularly useful in regulated environments where traceability of data modifications is required.

7. Architecture diagram

8. Minimum configuration

PostgreSQL: enable logical replication

-- Requires restarting Postgres
ALTER SYSTEM SET wal_level = logical;
ALTER SYSTEM SET max_replication_slots = 10;
ALTER SYSTEM SET max_wal_senders = 10;

-- Reload configuration (wal_level requires a full restart)
SELECT pg_reload_conf();

-- Dedicated user for Debezium
CREATE USER debezium WITH REPLICATION LOGIN PASSWORD 'cambiar_esto';
GRANT SELECT ON TABLE public.documents TO debezium;

-- REPLICA IDENTITY FULL to get a complete 'before' on DELETEs and UPDATEs
ALTER TABLE public.documents REPLICA IDENTITY FULL;

Debezium connector (Kafka Connect REST API)

{
 "name": "postgres-debezium",
 "config": {
 "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
 "database.hostname": "postgres",
 "database.port": "5432",
 "database.user": "debezium",
 "database.password": "cambiar_esto",
 "database.dbname": "rag_db",
 "topic.prefix": "rag",
 "table.include.list": "public.documents",
 "plugin.name": "pgoutput",
 "slot.name": "debezium_rag",
 "publication.autocreate.mode": "filtered",
 "snapshot.mode": "initial",
 "tombstones.on.delete": "true",
 "transforms": "unwrap",
 "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
 "transforms.unwrap.drop.tombstones": "false",
 "transforms.unwrap.delete.handling.mode": "rewrite"
 }
}

curl -X POST http://kafka-connect:8083/connectors \
 -H 'Content-Type: application/json' \
 -d @connector-config.json

Check its status:

curl http://kafka-connect:8083/connectors/postgres-debezium/status

Minimal consumer in Python

from confluent_kafka import Consumer
from qdrant_client import QdrantClient
from qdrant_client.models import Filter, FieldCondition, MatchValue
import json

consumer = Consumer({
 "bootstrap.servers": "kafka:9092",
 "group.id": "qdrant-sync",
 "auto.offset.reset": "earliest",
 "enable.auto.commit": False,
})
consumer.subscribe(["rag.public.documents"])
qdrant = QdrantClient("qdrant", port=6333)

while True:
 msg = consumer.poll(timeout=0.1)
 if msg is None:
 continue
 event = json.loads(msg.value())
 op = event.get("op")
 if op in ("c", "u"): # INSERT or UPDATE
 doc = event["after"]
 # ... vectorise and upsert into Qdrant
 elif op == "d": # DELETE
 doc_id = event["before"]["id"]
 qdrant.delete(
 collection_name="documents",
 points_selector=Filter(
 must=[FieldCondition(key="doc_id", match=MatchValue(value=doc_id))]
 )
 )
 consumer.commit()

9. On-premise deployment

The Debezium stack does not compete for GPU. On a node with 4×H100 SXM (320 GB, NVLink) serving the inference LLM, the CDC pipeline runs entirely on general-purpose (CPU-only) nodes:

Component	Recommended resources	Role
Kafka Connect + Debezium	2-4 cores, 4-8 GB RAM	Read the WAL, emit events
Kafka brokers (×3)	4 cores, 32 GB RAM each	High availability, retention
Qdrant-sync consumer	2 cores, 4 GB RAM	Vectorise plus upsert/delete
Qdrant	8 cores, 64 GB RAM	Vector store

The Debezium connector is notably light: in production at 10,000 events/second, the connector typically uses less than 1 core and 2 GB of RAM. JVM memory (Kafka Connect runs on the JVM) must be capped explicitly with -Xmx4g to stop the GC from causing pauses.

For high availability, Kafka Connect supports distributed mode with multiple workers. If a worker dies, the connector is reassigned automatically to another worker within seconds, and the replication slot guarantees no events are lost during the switchover.

What we have not covered

Debezium with MySQL, MongoDB and Oracle: each connector uses the native log mechanism (binlog on MySQL, oplog on MongoDB, LogMiner on Oracle). The resulting event API is similar, but the configuration details and limitations differ.
Debezium Server: standalone mode without Kafka Connect, with direct sinks to HTTP, S3, Redis Streams or NATS. Useful when the Kafka infrastructure is too complex for the use case.
Schema Registry: how Avro with Confluent Schema Registry or Apicurio manages event schema evolution (adding columns, changing types) without breaking existing consumers.
Exactly-once semantics: why at-least-once is enough for most RAG cases (an idempotent upsert into Qdrant with the same vector does no harm) and when exactly-once is needed (financial counters, inventory deductions).
Outbox pattern plus Debezium combined: Debezium reading the outbox table instead of the business table’s WAL directly, the Transactional Outbox + CDC pattern that combines the best of both worlds.

References

Debezium Documentation — PostgreSQL Connector. debezium.io/documentation/reference/stable/connectors/postgresql.html
PostgreSQL Documentation — Logical Replication. postgresql.org/docs/current/logical-replication.html
PostgreSQL Documentation — Write-Ahead Logging. postgresql.org/docs/current/wal-intro.html
PostgreSQL Documentation — Replication Slots. postgresql.org/docs/current/logicaldecoding-explanation.html
Confluent — Kafka Performance Benchmarks (2023). confluent.io/blog/kafka-fastest-messaging-system
Gunnar Morling — Outbox Pattern. morling.dev/blog/sending-messages-as-part-of-database-transactions
Debezium — SMT documentation. debezium.io/documentation/reference/stable/transformations
Qdrant Documentation — Filtering. qdrant.tech/documentation/concepts/filtering

Function calling and tool-augmented retrieval: the detective who knows which file to ask for

Thu, 04 Jun 2026 10:00:00 +0200

This post goes deeper into the agentic reasoning mechanism that extends the RAG described in RAG with reranker and hybrid retrieval. The retriever invoked when the LLM chooses vector_search is exactly the pipeline in that article. The JSON Schema that defines each tool call is structured output applied to the tool interface. And the agent’s requests pass through the L7 inference gateway before reaching the model.

TL;DR

A naive RAG always queries the same source. Function calling gives the LLM the ability to decide which tool to invoke (vector store, SQL, web search) according to what the query actually needs. The ReAct pattern chains those invocations in a reasoned loop until there is enough evidence. A 3-iteration pipeline with Llama-3.1-70B on on-premise hardware takes ≈ 1.1 s against the ≈ 300 ms of a single-step RAG; the gain is not in speed but in queries that a naive RAG simply cannot answer. The critical eval metric is tool selection accuracy: the percentage of turns in which the model picks the right tool, measured over a synthetic eval set.

The analogy: the detective who knows which file to ask for

A detective in a novel does not go to the same filing cabinet regardless of the case that lands on his desk. When he takes a case, he reasons first: what kind of evidence do I need? If there are witnesses, he asks for the statements (vector search over unstructured documents). If there are financial transactions, he asks the bank for the accounting records (SQL over the structured database). If the suspect has recent activity that the company cannot possibly have indexed, he goes to the press archive (web search). He does not consult all three sources at once on every case: he picks the one the evidence requires, receives the result, reasons again if he needs more, and only writes the report when he has enough evidence.

A bad detective always goes to the same filing cabinet. A naive RAG is that bad detective: it vectorises the query, goes to the vector store, and returns whatever it finds even if the question was “how many contracts?”, something no PDF chunk can answer better than a COUNT(*) in SQL.

Function calling is giving the LLM the ability to reason about which source to ask for, and to invoke it in a structured way. The analogy has three edges worth keeping in mind:

The detective does not improvise the file he asks for: there is a catalogue of available sources with a description of what each one contains. The tool description in the system prompt does that job.
The detective can ask for several pieces of evidence at once if they are independent (parallel tool calling).
The detective knows when to stop: if after N rounds he reaches no conclusion, he declares that he does not have enough evidence. The agent has an iteration limit for the same reason.

What function calling is: the anatomy of a tool call

Function calling, also called tool use, is a mechanism by which the LLM, instead of generating free text as its answer, generates a structured JSON object representing a tool invocation. The system intercepts that JSON, executes the real tool, and returns the result as a message with the tool role in the conversation.

Defining tools in the system prompt

Each tool is defined through a JSON Schema that specifies name, description and parameters. This JSON Schema is exactly the same mechanism described in structured output, applied here to the tool interface:

{
 "tools": [
 {
 "name": "vector_search",
 "description": "Search internal company documents about policies, contracts and procedures. Use when the query requires unstructured text, document context or semantic similarity.",
 "parameters": {
 "type": "object",
 "properties": {
 "query": {"type": "string", "description": "Natural language search query"},
 "top_k": {"type": "integer", "default": 5}
 },
 "required": ["query"]
 }
 },
 {
 "name": "sql_query",
 "description": "Query the SQL database for structured metrics, counts, aggregations and financial data. Use when the query requires exact numbers, filters, sums or joins over structured records.",
 "parameters": {
 "type": "object",
 "properties": {
 "query": {"type": "string", "description": "Parameterized SQL query with $1, $2 placeholders"},
 "params": {"type": "array", "items": {}, "description": "Parameter values for the placeholders"}
 },
 "required": ["query"]
 }
 },
 {
 "name": "web_search",
 "description": "Search public web for real-time information, recent news or current prices. Use only when data is public and not covered by internal sources.",
 "parameters": {
 "type": "object",
 "properties": {
 "query": {"type": "string"}
 },
 "required": ["query"]
 }
 }
 ]
}

The life cycle of a tool call

When the LLM decides to invoke a tool, the message it generates in place of free text has this structure (OpenAI-compatible format, the same one vLLM supports):

{
 "role": "assistant",
 "tool_calls": [
 {
 "id": "call_01",
 "type": "function",
 "function": {
 "name": "sql_query",
 "arguments": "{\"query\": \"SELECT COUNT(*), SUM(amount) FROM contracts WHERE amount > $1 AND year = $2 AND provider_region = $3\", \"params\": [100000, 2025, \"EU\"]}"
 }
 }
 ]
}

The system executes the tool and returns:

{"role": "tool", "tool_call_id": "call_01", "content": "{\"count\": 47, \"total\": 8300000}"}

The LLM receives that message as a continuation of the conversation and decides whether it needs more information or can generate the final answer.

Support in OSS models

In 2026, native function calling support (not emulated via system prompt) is available in:

Llama 3.1 / 3.3+: native tool call format, supported in vLLM with --enable-auto-tool-choice --tool-call-parser llama3_json
Qwen 2.5+: native support with --tool-call-parser hermes
Mistral NeMo / Mistral 7B Instruct v0.3+: native format with --tool-call-parser mistral
Qwen3 (2025+): extended native support with parallel tool calling

Without --enable-auto-tool-choice, vLLM can force the tool call format via guided decoding (JSON Schema), but the ability to decide when to invoke a tool versus generating free text requires specific model training.

The ReAct pattern: Reasoning + Acting

ReAct (Yao et al., 2022) is the loop that turns an LLM with tools into an agent with reasoning. The central idea is that the model interleaves thoughts (explicit reasoning about what to do) with actions (tool invocations) and observations (results received). This loop repeats until the model reckons it has enough information to answer, or until the iteration limit is reached.

Thought → Action → Observation → Thought → Action → Observation → ... → Response

A full ReAct trace: a real production query

Query: “How many contracts above 100k€ did we sign in 2025 with EU providers, and which special conditions recur?”

This query requires two different sources: the exact numbers live in SQL, and the context of special conditions lives in the contract PDFs indexed in the vector store.

Iteration 1

Thought: The query asks for aggregated data (count and sum) with exact filters over structured fields (amount, year, region). This is a SQL query, not a semantic search.

Action: sql_query

SELECT COUNT(*) AS num_contratos, SUM(amount) AS total_eur
FROM contracts
WHERE amount > $1 AND year = $2 AND provider_region = $3

params: [100000, 2025, "EU"]

Observation: {"num_contratos": 47, "total_eur": 8300000}

Iteration 2

Thought: I have the numbers. Now I need the special conditions that recur in these contracts. That kind of information is in the free text of the PDFs, not in the structured table. I go to the vector store.

Action: vector_search

"EU provider contracts 2025 amount above 100k special conditions recurring clauses"

top_k: 5

Observation:

[chunk_1] Provider A contract — clause 8.3: late-delivery penalty of 2% per month on the invoiced amount...
[chunk_2] Provider C contract — clause 7.1: price revision linked to the euro area CPI capped at 4% per year...
[chunk_3] Provider B contract — clause 8.3: late-delivery penalty of 2% per month...

Thought: I have enough data from both sources. Clause 8.3 on late-delivery penalties appears in at least two of the retrieved contracts. I can synthesise the answer.

Response: “In 2025 you signed 47 contracts with EU providers above 100,000 €, for a cumulative total of 8.3 M€. The special condition that appears most often in the higher-value contracts is the late-delivery penalty clause (8.3 in the contract templates of Provider A and B): 2% per month on the invoiced amount. The price revision linked to the euro area CPI, capped at 4% per year (clause 7.1), also recurs.”

This trace has 2 iterations. A more complex query (comparing with the previous year, cross-referencing with payment status) would add further iterations. Common practice is to set a limit of 5–10 iterations maximum to avoid loops.

The three sources and when to use each one

The choice of source is neither arbitrary nor left solely to the LLM: the architecture defines which tools exist and how they are described. The table below summarises the selection criteria:

Characteristic	Vector store (Qdrant)	SQL / structured (PostgreSQL)	Web search
Data type	Free text, documents, PDFs	Tables with a fixed schema	Public pages, news
Structure	Unstructured	Highly structured	Semi-structured
Temporality	Static index (periodic refresh)	Real time (transactional)	Real time (crawl)
Typical latency	5–50 ms	10–200 ms	200–2,000 ms
Privacy	Internal data, full sovereignty	Internal data, full sovereignty	Public data only
Natural query	Yes (natural language → embedding)	No (parameterised SQL)	Yes
Exact aggregations	No	Yes (`COUNT`, `SUM`, `GROUP BY`)	No
When to use	Document context, semantics, PDFs	Metrics, counts, exact filters, joins	Data that does not exist internally and is public

The most important rule of thumb: if the question contains words like “how many”, “sum”, “total”, “more than X”, “in year Y” and the data is in a structured table, the right answer is sql_query. If the question asks for context, explanations, clauses, procedures or examples from documents, the answer is vector_search. If it asks for the current price of something or recent news about a third party, web_search, but only if no data sovereignty is involved.

Tool routing: how the LLM picks the right tool

The description of each tool in the system prompt is the most critical factor for routing accuracy. An LLM with good function calling capabilities can still choose badly if the descriptions are ambiguous or overlap.

Descriptions that work versus those that do not

Weak description (leads the LLM to use the wrong tool):

"search_docs" — Searches for information in the available sources.
"query_data" — Gets data from the system.

Strong description (delimits precisely when to use each one):

"vector_search" — Search internal company documents about policies, contracts and procedures.
Use when the query requires unstructured text, document context or semantic
similarity. NOT for counts, sums or exact filters.
"sql_query" — Query the SQL database for structured metrics, counts, aggregations and
financial data. Use when the query requires exact numbers, filters, sums or
joins over structured records. NOT for finding document context.

The difference lies in two elements: (1) positive use-case examples, and (2) explicit exclusions with NOT for. Both reduce the semantic overlap between tools and improve tool selection accuracy.

Parallel tool calling

When two tools are independent of each other, that is, when the result of one does not affect the query of the other, the LLM can invoke them simultaneously in the same turn:

{
 "tool_calls": [
 {"id": "call_01", "function": {"name": "sql_query", "arguments": "..."}},
 {"id": "call_02", "function": {"name": "vector_search", "arguments": "..."}}
 ]
}

The system executes both in parallel and returns both observations before the LLM’s next turn. This cuts total latency when the queries are independent: instead of 2 sequential iterations (2 × tool_latency), you pay 1 × max(sql_latency, vector_latency). For the detective example: if he needs both the accounting records and the witness statements to answer, he can ask for them at the same time.

Tool selection accuracy: the eval metric

Tool selection accuracy is the percentage of turns in which the LLM picks the right tool given a set of evaluated queries:

\[ \text{TSA} = \frac{\text{turns with correct tool chosen}}{\text{total turns with expected tool call}} \]

It is measured over a synthetic eval set built with (query, expected_tool, expected_args) triples. An example of a minimal eval set:

Query	Expected tool	Failure indicator
“How many orders in March?”	`sql_query`	LLM uses `vector_search`
“What does the returns policy say?”	`vector_search`	LLM uses `sql_query`
“What is the price of copper today?”	`web_search`	LLM uses `vector_search`
“Sum the Q3 contracts”	`sql_query`	LLM uses `vector_search`

A TSA < 0.85 in a production agent is a sign that the tool descriptions need revising before the model does. For more detail on how to build these evals, see LLM evals.

Error handling in tool calls

SQL injection via prompt

The most serious risk in tool-augmented retrieval is that the LLM generates malicious SQL, either because a user induced it via prompt injection, or because the model hallucinated a destructive query. This attack vector is covered in detail in LLM Guard, but the minimum rules on the tool side are:

Rule 1: parameterised queries always, never direct interpolation.

# NEVER this:
cursor.execute(f"SELECT * FROM contracts WHERE provider = '{llm_output}'")

# Always this:
cursor.execute("SELECT * FROM contracts WHERE provider = $1", [llm_output])

Rule 2: DB user with minimum permissions. The user the agent runs SQL as must have SELECT on the necessary tables and nothing else. No DROP, INSERT, UPDATE or DELETE.

Rule 3: table allowlist. The system validates that the query generated by the LLM only references tables on an allowlist before executing it.

Rule 4: per-query timeout. Queries that block for too long, potentially induced to DoS the database, are cancelled with a configured timeout.

Rate limits, timeouts and tool errors

When a tool fails, the error is returned to the LLM as an observation:

{"role": "tool", "tool_call_id": "call_01", "content": "{\"error\": \"timeout after 5s\", \"tool\": \"web_search\"}"}

The system prompt must instruct the LLM on what to do in this case:

If a tool returns an error or is unavailable, acknowledge the limitation in your response.
Do not retry more than once. If web_search is unavailable, state that real-time data
is not accessible at this moment and answer with available internal sources only.

This stops the agent from entering retry loops and handles graceful degradation: if web_search is unavailable, it answers with what it has in the internal sources.

Diagram of the ReAct loop with the three sources

Latency maths of the ReAct pipeline

Each iteration of the ReAct loop has three latency components:

\[ T_{\text{iter}} = \text{TTFT}_{\text{LLM}} + T_{\text{tool}} + \Delta_{\text{context}} \]

where:

TTFT_LLM: time to the LLM’s first token (dominated by the prefill of the accumulated context)
T_tool: tool execution time
Δ_context: overhead of the growing context window (each iteration adds the previous output to the context)

Reference values: Llama-3.1-70B on 4×H100 SXM (320 GB, NVLink)

With Llama-3.1-70B in FP8 on a node with 4×H100 SXM (320 GB HBM3, NVLink 900 GB/s), the typical values in production are:

Component	Value
TTFT with context < 4k tokens	≈ 150 ms
TTFT with context 8k tokens	≈ 220 ms
`sql_query` (simple query, indexed)	≈ 50 ms
`vector_search` (top-5, Qdrant in RAM)	≈ 20 ms
`web_search` (external API)	≈ 600 ms

Total latency by number of iterations

1-iteration pipeline (simple query, a single tool):

\[ T_1 = 150 + 50 = 200 \text{ ms} + \text{final synthesis} \approx 200 + 300 = 500 \text{ ms} \]

2-iteration pipeline (SQL + vector_search in sequence):

\[ T_2 = (150 + 50) + (180 + 20) + 400 = 800 \text{ ms} \]

The context in the second iteration already includes the result of the first, so TTFT rises slightly to ≈ 180 ms.

3-iteration pipeline (the most common case in complex queries):

\[ T_3 = (150 + 50) + (180 + 20) + (200 + 50) + 450 \approx 1,100 \text{ ms} \]

Parallel tool calling (SQL + vector_search in parallel, a single iteration):

\[ T_{\text{parallel}} = 150 + \max(50, 20) + 400 = 600 \text{ ms} \]

When the two queries are independent, parallel tool calling trims latency from ≈ 800 ms to ≈ 600 ms: a 25% improvement over the 2 sequential iterations case.

Comparison with a naive RAG

Configuration	Latency	Queries it can answer
Naive RAG (1 retriever, 1 step)	≈ 300 ms	Document context queries
ReAct 1 iteration (SQL)	≈ 500 ms	Structured aggregation queries
ReAct 2 iterations (SQL + vector)	≈ 800 ms	Hybrid numeric + context queries
ReAct 3 iterations	≈ 1,100 ms	Complex multi-source queries
ReAct with web_search	≈ 1,500 ms	Queries requiring real-time data

The practical conclusion: multi-hop ReAct is 3–5× slower than a single-step naive RAG. The gain is not in speed but in query coverage: a naive RAG cannot answer “how many contracts?” because that answer is in no text chunk at all. For applications with a strict latency SLO (< 500 ms), you have to work out whether the use case really needs ReAct or whether a well-configured RAG with hybrid retrieval covers 90% of the queries.

On-premise hardware for ReAct agents

A ReAct agent with Llama-3.1-70B in production has different requirements from a naive RAG, because the context grows with each iteration and prefill throughput is more critical.

Recommended configuration: 4×H100 SXM (320 GB HBM3, NVLink 900 GB/s)

Llama-3.1-70B in FP8: fits in 2×H100 (70B params × 1 byte/param ≈ 70 GB + KV cache). With 4×H100 you can serve it with tensor parallelism TP=4, cutting prefill TTFT by ≈ 2×.
Qdrant instance: it can sit on the same node (if the collection fits in RAM) or on a dedicated node. For collections < 50M vectors of 768 dims: ≈ 150 GB, which fits in the RAM of a dual-socket server.
PostgreSQL: separate node or managed instance. The agent adds no unusual load to SQL; the queries are simple and bounded by timeout.
vLLM with --enable-auto-tool-choice --tool-call-parser llama3_json --max-model-len 16384: a 16k token context comfortably covers the 5–10 turns of a ReAct pipeline.

Minimum viable configuration: 2×H100 SXM (160 GB)

Llama-3.1-70B in FP8 with TP=2. TTFT ≈ 250–300 ms for 4k token contexts (a 60–100% increase over TP=4).
Good enough for internal workloads with < 20 concurrent requests.
Not advisable for an SLO < 1 s with more than 5 concurrent users and a long context.

What we have not covered

Agentic retrieval loops with planning. ReAct is the simplest agent pattern. When a query requires decomposition into sub-tasks with dependencies, you need orchestration frameworks such as LangGraph (state graphs), smolagents (Hugging Face, agents with Python code as actions) or llama-index Agents (planning + retrieval pipeline). These introduce a planning step before the execution loop.

MCP (Model Context Protocol). Anthropic’s emerging standard, with OSS implementations, for defining tools portably across frameworks and hosts. Instead of defining each tool’s JSON Schema separately in every application, MCP centralises those definitions in an MCP server that any compatible client can discover and invoke. Adoption in 2025–2026 is fast among OSS frameworks (LangChain, smolagents, OpenWebUI).

Tool caching. If the same tool call (same arguments, same tool) is going to be invoked several times within the same context or in very similar contexts, the result can be cached. The mechanism is analogous to the semantic cache described for RAG: before executing the tool, the hash of the arguments (or its embedding, for semantic matching) is compared against a cache with a TTL. Especially valuable for sql_query with frequent queries and data that changes little.

Multi-agent. When an orchestrating agent delegates sub-tasks to specialised agents, one for SQL, one for document retrieval, one for code generation, you enter multi-agent system territory. Each sub-agent can have its own tool set and its own LLM (possibly smaller and more specialised). Coordination between agents introduces additional tracing and observability complexity.

References

Yao, S. et al. (2022). ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629. https://arxiv.org/abs/2210.03629
vLLM documentation. Tool calling. https://docs.vllm.ai/en/stable/features/tool_calling.html
Qdrant documentation. Search. https://qdrant.tech/documentation/concepts/search/
OpenAI. Function calling. https://platform.openai.com/docs/guides/function-calling
Meta AI. Llama 3.1 Model Card. https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md
Qwen Team (Alibaba). Qwen2.5 Technical Report. arXiv:2412.15115. https://arxiv.org/abs/2412.15115
Anthropic. Model Context Protocol. https://modelcontextprotocol.io
OWASP. LLM Top 10 for Large Language Model Applications. LLM01: Prompt Injection. https://owasp.org/www-project-top-10-for-large-language-model-applications/

Evaluating a RAG system without fooling yourself: RAGAS, the golden dataset and the four metrics that matter

Thu, 04 Jun 2026 09:00:00 +0200

TL;DR

A RAG pipeline fails in ways that user satisfaction cannot distinguish: the LLM can hallucinate even with good chunks, or retrieval can ignore key documents even though the LLM synthesises well what it receives. RAGAS breaks evaluation down into four orthogonal metrics, faithfulness, answer relevance, context precision and context recall, each pointing at a different sub-component. The golden dataset is the reference calibrator; without it the metrics have no anchor. The complete stack runs 100 % on-premise with vLLM as the judge and Langfuse for traceability.

The master analogy: the quality inspector at a furniture factory

Imagine you manufacture chairs. You could ask customers “is it comfortable?” and leave it there. But that question does not tell you what to fix when the answer is “no”. The quality inspector does not ask that: they measure the board with a Shore hardness test, check that each leg is exactly 45 cm, verify that the assembly manual includes the twelve screws from the BOM, and detect whether a low-density board slipped past the incoming filter.

RAGAS is that inspector applied to RAG:

Faithfulness → does the board have the specified hardness? The LLM can only use the material (chunks) that retrieval hands it.
Context Precision → is the leg exactly the right length? Of the K retrieved chunks, how many are genuinely useful and how many are filler that confuses the assembler?
Context Recall → does the manual include all the screws? Of all the facts the correct answer should contain, how many appear in the retrieved chunks?
Noise Sensitivity → if the operator uses a medium-low density board, does it show in the final product? If you introduce irrelevant chunks, does the LLM start hallucinating?

Without measuring each dimension separately, the diagnosis is opaque: “the RAG is not working well” does not tell you whether to repair the embedder, the reranker, the prompt or the corpus.

The problem of evaluating RAG

Classification has one uncomfortable virtue: if you predict 87 out of 100 labels correctly, accuracy = 0.87. There is no ambiguity. RAG does not enjoy that luxury.

A RAG system can fail in at least three independent ways:

Retrieval correct, LLM hallucinates: the chunks contain the correct answer, but the LLM generates claims that are not in those chunks. Faithfulness low; context recall high.
LLM correct, retrieval fails: retrieval returns irrelevant chunks (low context precision) or incomplete ones (low context recall). If the LLM has enough parametric knowledge, it may look as though it is answering well, but in reality it is ignoring the context, which is a time bomb once the parametric knowledge goes stale.
Retrieval and LLM both correct, answer does not answer the question: the answer is faithful to the context and the chunks are relevant, but the question was a different one. Answer relevance low.

Each failure mode requires a different metric and a different corrective action. Using a single metric (BLEU, ROUGE, user satisfaction) mixes the signals and makes it impossible to prioritise improvement work.

The four RAGAS metrics

1. Faithfulness — fidelity to the context

Question: how many claims in the generated answer are supported by the retrieved chunks?

Calculation:

$$\text{Faithfulness} = \frac{|\text{claims supported by the context}|}{|\text{total claims in the answer}|}$$

The process uses an LLM-as-judge (see https://blog.lo0.es/en/posts/llm-as-judge-exam-marker-not-oracle/): first the atomic claims are extracted from the answer (“the model was released in 2023”, “it supports contexts of 128k tokens”, and so on), then the judge classifies each claim as supported or not supported by the chunks.

Example: The generated answer has 5 claims. The judge determines that 4 are in the chunks and 1 is an unsupported extrapolation.

$$\text{Faithfulness} = \frac{4}{5} = 0{.}80$$

Warning sign: faithfulness < 0.85 indicates that the LLM is generating content that goes beyond the context, which is to say it is hallucinating with superficial backing.

2. Answer Relevance — relevance of the answer

Question: does the answer actually answer the question as posed?

Intuition: An answer that answers the question well “implies” that question. If you generate N hypothetical questions from the answer and measure their semantic similarity to the original question, you get a relevance signal.

Calculation:

$$\text{AnswerRelevance} = \frac{1}{N} \sum_{i=1}^{N} \cos(\vec{q}_{\text{original}}, \vec{q}_{i}^{\text{generated}})$$

where $\vec{q}$ are question embeddings.

Example: For the question “Which Python versions does FastAPI support?” and an answer about web frameworks in general, the hypothetical questions generated will be along the lines of “what are the best web frameworks?”, with low cosine against the original question → low answer relevance.

3. Context Precision — retrieval precision

Question: of the K retrieved chunks, what proportion are genuinely relevant?

Calculation (weighted version):

$$\text{ContextPrecision@K} = \frac{\sum_{k=1}^{K} \text{Precision@}k \cdot \mathbb{1}[\text{chunk}_k \text{ is relevant}]}{|\text{relevant chunks in top-K}|}$$

The most direct form: the LLM judge classifies each chunk as relevant or not for answering the question. Precision is the relevant fraction.

Example: 5 chunks are retrieved. The judge considers 3 relevant and 2 noise.

$$\text{ContextPrecision} = \frac{3}{5} = 0{.}60$$

Warning sign: precision < 0.6 indicates that retrieval is contaminating the context with information that can contradict or dilute the correct answer.

4. Context Recall — retrieval recall

Question: of all the facts needed to build the correct answer (ground truth), what proportion are covered by the retrieved chunks?

Calculation:

$$\text{ContextRecall} = \frac{|\text{ground-truth claims attributable to some chunk}|}{|\text{total claims in ground truth}|}$$

This metric requires ground truth, which is to say you need the golden dataset.

Example: The ground truth has 6 claims. The retrieved chunks cover 5 of them; the sixth is in a document that retrieval did not find.

$$\text{ContextRecall} = \frac{5}{6} \approx 0{.}83$$

5. Noise Sensitivity — sensitivity to noise

Question: if you introduce irrelevant chunks into the context, how far does faithfulness fall?

It is computed as the difference in faithfulness between the clean condition and the condition with controlled noise:

$$\Delta\text{Faithfulness} = \text{Faithfulness}_{\text{clean}} - \text{Faithfulness}_{\text{noisy}}$$

A $\Delta > 0{.}15$ indicates that the prompt or the LLM are fragile against contaminated contexts, a problem that low context precision turns into a routine occurrence.

A complete worked example

Scenario

Question: “What latency guarantees does Redis offer for in-memory read operations with a 10 GB dataset?”

Answer generated by the RAG:

“Redis guarantees sub-millisecond latencies for GET operations on datasets that fit in RAM. With a 10 GB dataset and clients on the same local network, P99 is typically below 1 ms. Redis also supports horizontal clustering to scale beyond the RAM of a single node. In addition, Redis has included native JSON support since version 7.2.”

Ground truth (reference answer):

“Redis operates entirely in memory, which guarantees sub-millisecond latencies for GET. On local networks with 10 GB datasets in RAM, P99 sits below 1 ms. Clustering allows scaling beyond the RAM of a single node.”

Retrieved chunks (5 chunks, summarised fragments):

#	Summarised content	Relevant
C1	“Redis operates in memory; GET has latencies < 1 ms on LAN”	Yes
C2	“Redis Cluster allows sharding to scale total RAM”	Yes
C3	“Redis Sentinel manages high availability through automatic failover”	No
C4	“Redis benchmarks: P50 = 0.3 ms, P99 = 0.9 ms on a 10 GB dataset”	Yes
C5	“Redis Stack adds modules: RedisJSON, RediSearch, RedisTimeSeries”	No

Step-by-step calculation

Faithfulness:

Claims in the generated answer:

“Redis guarantees sub-millisecond latencies for GET on in-RAM datasets” → supported by C1, C4
“With 10 GB on LAN, P99 < 1 ms” → supported by C4
“Redis supports horizontal clustering to scale RAM” → supported by C2
“Redis has included native JSON support since version 7.2” → NOT supported by any chunk (C5 mentions RedisJSON as a Redis Stack module, not as native to Redis core)

$$\text{Faithfulness} = \frac{3}{4} = 0{.}75$$

Claim 4 is an extrapolation that mixes information from C5 imprecisely, a partial hallucination.

Context Precision:

Relevant chunks: C1, C2, C4 (3 out of 5).

$$\text{ContextPrecision} = \frac{3}{5} = 0{.}60$$

C3 and C5 are noise. C5 in particular contributed to the partial hallucination about JSON.

Context Recall:

Ground-truth claims:

“Redis operates in memory, GET < 1 ms” → attributable to C1 ✓
“P99 < 1 ms on LAN with 10 GB” → attributable to C4 ✓
“Clustering scales beyond the RAM of a single node” → attributable to C2 ✓

$$\text{ContextRecall} = \frac{3}{3} = 1{.}00$$

Retrieval found every chunk needed for the ground truth. The problem is not recall but precision (C3 and C5 contaminated the context).

Answer Relevance:

The judge generates 3 hypothetical questions from the answer:

“What latencies does Redis offer for in-memory reads?” — cos = 0.91
“How does Redis scale horizontally?” — cos = 0.74
“Which JSON modules does Redis include?” — cos = 0.52 (a drift caused by the hallucination)

$$\text{AnswerRelevance} = \frac{0{.}91 + 0{.}74 + 0{.}52}{3} = 0{.}72$$

The drift towards JSON reduced relevance. A tighter answer would have scored around 0.90.

Summary of the example

Metric	Value	Diagnosis
Faithfulness	0.75	LLM extrapolated beyond the context
Context Precision	0.60	Retrieval returned 2 irrelevant chunks
Context Recall	1.00	Retrieval captured everything needed
Answer Relevance	0.72	Answer drifts off topic

Main corrective action: improve the reranker so that C3 and C5 are filtered before they reach the LLM. The faithfulness and relevance problem is a direct consequence of the low precision, not of the LLM itself.

Building the golden dataset

What it is and why it matters

The golden dataset is a set of (question, relevant chunks, correct answer) tuples that acts as a reference calibrator. Without it, context recall cannot be computed (there is no ground truth) and the other metrics lack any interpretive anchor: is 0.75 faithfulness good or bad for this corpus and this domain?

A well-built golden dataset lets you:

Compare pipeline versions (embedder v1 vs v2, chunk size 512 vs 1024)
Detect regressions in CI before deploying
Stratify the analysis by question type

LLM-assisted construction pipeline

Purely manual construction is expensive. The standard pattern in 2026 is LLM assistance with human review of a sample:

Step 1 — Seed chunk selection. From the full corpus, select representative chunks through stratified sampling (by section, date, document type). For a technical corpus of 10,000 chunks, 500-1,000 seeds is a reasonable starting point.

Step 2 — Question generation. A powerful LLM (Llama-3.1-70B or similar) generates 2-3 questions per seed chunk using a prompt along these lines:

Given the following documentation fragment, generate specific questions
that can only be answered correctly using THIS fragment and not
general knowledge. The questions should be the ones an engineer looking
for operational information would ask.
Fragment: {chunk}

Step 3 — Reference answer generation. The same LLM, with access to the seed chunk (and to adjacent chunks if the question requires them), generates the reference answer.

Step 4 — Human review of a sample. Manually review 10-20 % of the generated dataset. The most common rejection criteria: trivial questions that any LLM answers without the corpus, answers the LLM padded with parametric knowledge instead of the chunks, and badly formulated or ambiguous questions.

Minimum size

Use case	Minimum pairs	Notes
Prototype / initial validation	50-100	Enough to spot coarse problems
Technical corpus in production	200-500	Allows basic stratification
Robust production with full stratification	500-1,000+	Needed to detect subtle regressions

Stratifying the dataset

A flat golden dataset measures the average but hides the extreme cases. The minimum recommended stratification includes three question types:

Easy (single-hop): A single chunk contains all the information needed. The baseline that any decent RAG must beat.
Hard (multi-hop): The correct answer requires combining information from 2-4 different chunks. This is where the limits of the reranker and the synthesis prompt show up.
Adversarial: The question has a false premise, or the corpus does not contain the answer. A correct RAG must answer “I do not have enough information”, whereas a fragile RAG hallucinates confidently. This question type directly measures the risk of high-impact hallucination.

The Goodhart trap

“When a measure becomes a target, it ceases to be a good measure.” — Charles Goodhart

If you optimise the embedder or the reranker using the golden dataset as a loss function, the dataset is corrupted as a metric: the system learns to perform well on those specific questions without improving on the general domain.

The solution is the same as in supervised ML: separate the dev set (for optimisation and iteration) from the test set (for final evaluation, frozen and audited). The test set must never be used to make design decisions; only to report the state of the system at release time.

Correlation with real satisfaction

Field studies published by the Databricks teams (2024) and the RAGAS adoption analyses (2025) point to interpretable operational thresholds:

Metric range	Observable symptom	Corrective action
Faithfulness < 0.75	Users frequently report “made-up answers”	Review the LLM prompt; increase citation instructions; lower the temperature
Faithfulness 0.75-0.85	Occasional hallucinations on peripheral topics	Improve context precision to eliminate contaminating chunks
Faithfulness ≥ 0.85	Correlates with positive NPS in field studies	Hold steady; monitor for drift
Context Precision < 0.60	LLM includes contradictory information; inconsistent answers	Tune the reranker; reduce K; review similarity thresholds
Context Recall < 0.70	Multi-hop questions fail; key information missing	Review the chunking strategy; add larger chunks; enrich metadata
Answer Relevance < 0.70	Answers that are “correct but do not answer”	Review the synthesis prompt; add an explicit instruction to stick to the question

Low context precision is especially pernicious: irrelevant chunks are not neutral. They raise the probability that the LLM uses incorrect information as though it were relevant, degrading faithfulness in a chain reaction. It is the transmission belt by which a retrieval problem turns into an LLM problem.

Diagram: the continuous evaluation loop

The continuous evaluation loop: corpus → retrieval → LLM → RAGAS judge → metrics → alert → corrective action → corpus.

The 2026 OSS stack for running RAGAS on-premise

ragas (Apache 2.0)

The ragas library supports asynchronous evaluation and multiple LLM backends. Integrating vLLM as the judge removes any need to send data to external APIs, which is critical in environments with sensitive data.

from ragas import evaluate
from ragas.metrics import (
 faithfulness,
 answer_relevancy,
 context_precision,
 context_recall,
)
from langchain_openai import ChatOpenAI, OpenAIEmbeddings

# Judge LLM pointing at on-premise vLLM
judge_llm = ChatOpenAI(
 model="meta-llama/Llama-3.1-70B-Instruct",
 base_url="http://vllm-service:8000/v1",
 api_key="sk-local", # vLLM ignores the value but requires the field
)

embeddings = OpenAIEmbeddings(
 model="BAAI/bge-m3",
 base_url="http://embedding-service:8001/v1",
 api_key="sk-local",
)

result = evaluate(
 dataset=golden_dataset, # HuggingFace Dataset with standard columns
 metrics=[faithfulness, answer_relevancy, context_precision, context_recall],
 llm=judge_llm,
 embeddings=embeddings,
)

The dataset RAGAS expects has four columns: question, answer, contexts (a list of strings), ground_truth.

Langfuse for eval traceability

Each RAGAS evaluation is recorded in Langfuse as a dataset experiment, linking the scores to the production spans (see https://blog.lo0.es/en/posts/llm-tracing-opentelemetry-genai/). This makes it possible to correlate a drop in faithfulness with the specific request that caused it. Without that link, the metrics are numbers with no actionable context.

from langfuse import Langfuse

lf = Langfuse()

# Create or retrieve the dataset in Langfuse
dataset = lf.get_or_create_dataset("rag-golden-v3")

# Record the experiment scores
for idx, row in result.to_pandas().iterrows():
 lf.score(
 name="ragas-faithfulness",
 value=row["faithfulness"],
 trace_id=row["trace_id"], # linked to the production span
 )

Prometheus + Grafana for operational alerts

RAGAS metrics are exposed as Prometheus gauges. A Grafana dashboard with thresholds configures alerts when faithfulness falls sustainedly below 0.80:

# Prometheus alert rule
- alert: RAGFaithfulnessLow
 expr: avg_over_time(rag_faithfulness_score[30m]) < 0.80
 for: 10m
 labels:
 severity: warning
 annotations:
 summary: "RAG faithfulness below threshold ({{ $value | humanize }})"
 description: "Review context precision and the reranker. Possible corpus drift."

Running RAGAS against on-premise vLLM — practical considerations

Judge size: Llama-3.1-70B as the judge produces results comparable to GPT-4 on faithfulness and context evaluation, according to the RAGAS 0.2 benchmarks (2025). Smaller models (8B-13B) degrade judge quality on multi-hop questions.
Throughput: On on-premise hardware with 4×H100 SXM (320 GB, NVLink), a run of 200 evaluations with Llama-3.1-70B takes roughly 8-12 minutes with batch_size=8 and vLLM in continuous batching mode.
Cost per evaluation: With no external API, the marginal cost is electricity plus GPU amortisation. With 4×H100 at around 3 kW sustained, a run of 200 evaluations costs < 0.10 € in energy at a typical industrial tariff.
Recommended frequency: weekly offline eval over the full golden dataset plus sampled online eval (5-10 % of production requests) with a subset of metrics that do not require ground truth (faithfulness, answer relevance).

What we have not covered

Alternatives to RAGAS: TruLens (evaluation with modular feedbacks), DeepEval (programmatic assertions, pytest integration), ARES (Stanford framework with trained classifiers instead of LLM-as-judge), and the OpenAI evals framework. Each has different trade-offs in judge cost, reliability and ease of integration.
Continuous eval in production: automatically sampling real requests, anonymising them, running a subset of metrics without ground truth and using the result to detect system drift before users report it. It requires a data pipeline separate from the inference pipeline.
Multilingual eval: RAGAS with a judge in Spanish or Catalan over a non-English corpus has documented biases when the judge is a model trained fundamentally in English. The semantic similarity embeddings for answer relevance are especially sensitive to corpus language vs judge language.
A/B testing of RAG configurations: using RAGAS metrics as the success criterion in controlled experiments, chunk size 512 vs 1024, pure BM25 vs hybrid, cross-encoder reranker vs biencoder, with statistical significance computed over the golden dataset.

References

Es Shahul, et al. RAGAS: Automated Evaluation of Retrieval Augmented Generation. arXiv:2309.15217 (2023). https://arxiv.org/abs/2309.15217
RAGAS Documentation v0.2. Metrics Reference. https://docs.ragas.io/en/stable/concepts/metrics/ (accessed June 2026)
Langfuse. Dataset Experiments. https://langfuse.com/docs/datasets/overview (accessed June 2026)
Databricks. LLM Quality Evaluation: From Lab to Production. Databricks Engineering Blog (2024).
Saad-Falcon, J. et al. ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems. arXiv:2311.09476 (2023).
Goodhart, C.A.E. Problems of Monetary Management: The U.K. Experience. Papers in Monetary Economics. Reserve Bank of Australia (1975). Modern formulation of the law that bears his name.
vLLM Project. OpenAI-Compatible Server. https://docs.vllm.ai/en/stable/serving/openai_compatible_server.html (accessed June 2026)

Semantic cache in RAG: the receptionist with a photographic memory

Thu, 04 Jun 2026 08:00:00 +0200

TL;DR

In a RAG system with real traffic, 30–70% of queries are semantically equivalent to some earlier one even when the text differs. The semantic cache intercepts those queries ahead of the retriever and the LLM, returning the answer already computed if the cosine similarity with a previous query exceeds a threshold θ (typically 0.92–0.95). At 10,000 requests/day and a 45% hit rate, that amounts to not running 4,500 LLM generations: roughly 0.62 GPU hours saved every day on a cluster running Llama-3.1-70B. The fundamental trade-off is that a high θ gives more accurate answers but less saving; a low θ maximises saving but can return incorrect answers for subtly different queries.

The analogy: the receptionist with a notebook

Picture the front desk of a 400-room hotel. Over the course of the day, the receptionist takes hundreds of questions. But if you look through the log book, you will see that 60% of those questions are variants of the same ten:

“Where is the gym?”
“What time is breakfast?”
“Do you have parking?”
“How do I connect to the WiFi?”

By the third day, the receptionist has built a mental notebook of answers. When someone asks “where can I go to exercise?”, they do not call the concierge (retrieval) or open the hotel’s 300-page internal manual (LLM): they look at the notebook, work out that the question is the same as “where is the gym?”, and answer in two seconds.

But when someone asks “what time does the gym close today?”, the receptionist knows the notebook cannot be trusted: the schedule may have changed because of a private event. They have to call the concierge.

That is exactly the mechanism of a semantic cache:

The notebook is the cache store (Redis with a vector index, or a Qdrant collection).
Working out that “exercise” ≈ “gym” is the cosine similarity search with threshold θ.
Calling the concierge is retrieval over the corpus.
Consulting the manual is LLM generation.
“Today” is the temporal query signal that invalidates the cache.

The threshold θ is exactly what separates “where is it” (semantically the same) from “what time is it open today” (semantically different). It is not magic: it is vector arithmetic over learned representations.

The problem in production

A typical RAG pipeline has three layers of latency and compute: embedding the query, vector search over the corpus, and generation with the LLM. In development, that cost is irrelevant. In production with 50 concurrent users, each of those layers scales linearly with the number of requests.

The problem is that users ask the same questions over and over, with slightly different wording:

Original query	Equivalent query
“How do I configure the agent?”	“What is the process for configuring the agent?”
“error installing the dependency”	“the dependency installation fails”
“what is an embedding?”	“explain to me what embeddings are”

Running the full pipeline for each of these variants is pure waste. Empirical studies in technical support and corporate Q&A systems report that between 30% and 70% of a day’s queries are semantically redundant with respect to earlier queries from the same week.

The query distribution in real systems follows a power law similar to the Zipf distribution: the 100 most frequent topics account for roughly 60% of total traffic. A well-calibrated cache captures exactly that concentration.

How the semantic cache works

The full flow is shown in the diagram below. Let us describe it in prose first.

When a new query $q$ arrives:

Embedding the query: $q$ is embedded with the same model used to index the corpus. This is critical: if the corpus was indexed with text-embedding-3-large and the cache uses a different embedder, the vector spaces are not comparable.
Search in the cache store: an ANN (Approximate Nearest Neighbor) search is run over the vectors of previously cached queries. The most similar query $q^*$ and its cosine similarity $s$ are retrieved.
Threshold decision:
$$ \text{answer} = \begin{cases} r^* & \text{if } s(q, q^*) \geq \theta \\ \text{pipeline}(q) & \text{if } s(q, q^*) < \theta \end{cases} $$
where $r^*$ is the cached answer associated with $q^*$.
On a miss: the full pipeline runs (retrieval + LLM). The generated answer is stored in the cache with a configurable TTL for future similar queries.

The cache store is not an ordinary key-value database. It is a vector index over the query embeddings, with the values being the generated answers. Each entry has the structure:

{vector: embed(q), response: r, ttl: T, metadata: {...}}

Flow diagram

The semantic cache flow as middleware between the gateway and the retriever. Cache hits skip the vector search over the corpus and the LLM generation entirely.

The threshold θ and its trade-off

The threshold θ is the most sensitive parameter in the system. It works exactly like the receptionist’s recognition threshold: if it is too demanding, it will only identify textually identical questions and the notebook will not be much use. If it is too lax, it will return the answer to “where is the gym?” to someone who asked “what time does the gym close?”.

The cosine similarity between two vectors $\mathbf{a}$ and $\mathbf{b}$ is:

$$ s(\mathbf{a}, \mathbf{b}) = \frac{\mathbf{a} \cdot \mathbf{b}}{\|\mathbf{a}\| \cdot \|\mathbf{b}\|} $$

For prose text (Spanish or English), modern embedders such as text-embedding-3-large or nomic-embed-text assign cosine similarities of around 0.90–0.96 to semantically equivalent paraphrases and similarities of 0.75–0.88 to related but non-equivalent queries.

The cache quality metric is not just hit rate: it is precision@cache, defined as the fraction of cached answers that remain correct for the new query. A cached answer is “correct” if an evaluator (another LLM, or metrics such as BERTScore) judges it equivalent to the one the full pipeline would have generated for that specific query.

θ	Estimated hit rate	Estimated precision@cache	Effective saving
0.85	~65%	~72%	~47%
0.90	~55%	~85%	~47%
0.92	~48%	~91%	~44%
0.93	~45%	~94%	~42%
0.95	~35%	~98%	~34%
0.97	~18%	~99.5%	~18%

The effective saving is defined as $\text{hit rate} \times \text{precision@cache}$, since a hit with an incorrect answer is not a saving: it is an error that can cost more in lost trust than what was saved in GPU.

The empirical sweet spot for most corporate Q&A applications in Spanish or English lies between θ = 0.92 and θ = 0.95. In highly specialised domains where small nuances change the answer (medicine, law, finance), θ ≥ 0.95 is advisable.

The arithmetic of the saving

Let us put concrete numbers on a real system.

Base configuration:

10,000 requests/day
Technical corpus of 1 million chunks in a Qdrant index
LLM: Llama-3.1-70B on 4×H100 SXM (320 GB, NVLink)
Average answer: 200 output tokens
LLM throughput on this hardware: ~400 tokens/s/GPU with batching (continuous batching enabled, see continuous-batching-fundamentos)

Cost of a request without cache:

Embedding the query takes ~2 ms on a GPU. The ANN vector search over 1 M chunks in Qdrant takes ~5 ms (measured empirically with HNSW, ef=128). Generating 200 tokens at 400 tok/s in total (4 GPUs) comes to:

$$ t_{\text{LLM}} = \frac{200 \text{ tokens}}{400 \text{ tok/s}} = 0.5 \text{ s per request} $$

If 10,000 requests reach the LLM in a day, the total GPU time spent on generation is:

$$ T_{\text{GPU}} = 10{,}000 \times 0.5 \text{ s} = 5{,}000 \text{ s} \approx 1.38 \text{ GPU hours per day} $$

With a semantic cache at θ = 0.93 and a hit rate of ~45%:

Only 5,500 requests (55%) reach the LLM:

$$ T_{\text{GPU,cache}} = 5{,}500 \times 0.5 \text{ s} = 2{,}750 \text{ s} \approx 0.76 \text{ GPU hours per day} $$

Saving:

$$ \Delta T_{\text{GPU}} = 1.38 - 0.76 = 0.62 \text{ GPU hours/day} $$

In inference compute, this amounts roughly to being able to serve 45% more users without adding hardware, or cutting inference costs by 45% if you work with external APIs billed per token.

The cost of the semantic cache itself (embedding the query + ANN search over the cache store) is ~7 ms per request, negligible against the 500 ms of generation avoided on hits.

Zipf distribution of topics:

The reason it works is the Zipf distribution of traffic. If we number topics by frequency (topic 1 = most frequent), the frequency of topic $k$ is proportional to $1/k$. With 1,000 distinct topics:

$$ \text{fraction of traffic covered by top-}N = \frac{\sum_{k=1}^{N} 1/k}{\sum_{k=1}^{1000} 1/k} \approx \frac{\ln N}{\ln 1000} = \frac{\ln N}{6.9} $$

For the top-100 topics: $\ln(100)/6.9 \approx 4.6/6.9 \approx 67\%$ of traffic. The cache does not need to cover every topic: it captures 67% of traffic while covering only 10% of the topics.

OSS stack 2026

GPTCache

GPTCache is the reference library for a standalone semantic cache. Its architecture is modular:

Embedder: ONNX Runtime with converted models (onnx/all-MiniLM-L6-v2 by default), with no GPU dependency for the cache layer.
Vector store: Faiss (local), Milvus, or Qdrant.
Scalar store: SQLite (development) or Redis (production) for metadata, TTL, and answers.
Similarity evaluation: cosine by default, configurable.

Minimal configuration in Python:

from gptcache import cache
from gptcache.adapter import openai
from gptcache.embedding import Onnx
from gptcache.manager import CacheBase, VectorBase, get_data_manager
from gptcache.similarity_evaluation.distance import SearchDistanceEvaluation

onnx = Onnx()
data_manager = get_data_manager(
 CacheBase("redis", url="redis://localhost:6379"),
 VectorBase("qdrant", host="localhost", collection_name="query_cache")
)
cache.init(
 embedding_func=onnx.to_embeddings,
 data_manager=data_manager,
 similarity_evaluation=SearchDistanceEvaluation(),
)
cache.set_openai_key()

GPTCache intercepts calls to the OpenAI API (or to compatible proxies) transparently. The TTL is configured at the data_manager level.

MeanCache

MeanCache (2024) extends GPTCache to multi-turn conversations. The problem with standard GPTCache is that in dialogue the “relevant” query is not just the last message but the whole context window. MeanCache computes the query embedding as the weighted mean of the embeddings of the last $k$ turns:

$$ \mathbf{e}_{\text{query}} = \frac{\sum_{i=1}^{k} w_i \cdot \mathbf{e}_{q_i}}{\sum_{i=1}^{k} w_i} $$

where $w_i$ decreases with the age of the turn. This reduces false positives in dialogues where the topic keeps shifting.

Qdrant as a dual cache store

If the RAG corpus is already in Qdrant, you can use the same instance with a separate collection for the cache. The advantages are operational: a single service to manage, the same backup and monitoring infrastructure.

The cache collection uses payload filters to implement TTL:

from qdrant_client import QdrantClient
from qdrant_client.models import Filter, FieldCondition, Range
import time

client = QdrantClient("localhost", port=6333)

# Search the cache with a TTL filter
hits = client.search(
 collection_name="semantic_cache",
 query_vector=query_embedding,
 query_filter=Filter(
 must=[FieldCondition(
 key="expires_at",
 range=Range(gt=time.time())
 )]
 ),
 limit=1,
 score_threshold=0.93
)

You have to run a periodic cleanup of expired entries, since Qdrant has no native TTL (unlike Redis).

Langfuse for traceability

Langfuse is the OSS standard for LLM pipeline observability (see tracing-llm-otel-genai). Every request should be tagged with whether it was a cache hit or a miss:

from langfuse import Langfuse
from langfuse.decorators import observe

langfuse = Langfuse()

@observe()
def process_query(query: str) -> dict:
 cache_result = semantic_cache.lookup(query)
 if cache_result:
 langfuse.update_current_observation(
 metadata={"cache_hit": True, "cache_score": cache_result.score}
 )
 return cache_result.response
 # full pipeline...
 langfuse.update_current_observation(
 metadata={"cache_hit": False}
 )

With this metadata, Langfuse lets you compute the real hit rate, the distribution of similarity scores, and detect whether the threshold θ needs adjusting.

On-premise hardware: a reference configuration

For an on-premise deployment with this stack, a suitable configuration for RAG with a semantic cache is:

Inference node: 4×H100 SXM (320 GB NVLink in total) for Llama-3.1-70B in FP8. Throughput ~400 tok/s in generation with continuous batching (vLLM or TGI).

Vector services node: CPU with 256 GB RAM. Qdrant for the corpus (1–10 M chunks) and for the cache store (up to 500K entries in memory). Redis 7.x for metadata and an exact-match cache as a first layer.

Embedding node: CPU or a mid-range GPU (A10G). The cache embedder can run on ONNX Runtime on CPU with no perceptible impact on latency (~2 ms per embedding).

Separating the cache store from the corpus matters: the corpus holds millions of chunks with large HNSW indexes; the cache store holds at most tens of thousands of queries with a much smaller index and search times of 1–2 ms.

Cases where the cache fails

The receptionist with a notebook fails in three well-defined scenarios:

1. Queries with a temporal context

“What is the current status of the incident?” or “What changed in the latest release?” are questions whose correct answer changes over time. A cache with a 24-hour TTL could return stale information.

The solution is to detect temporal markers in the query (regular expressions over “today”, “now”, “current”, “latest”, “yesterday”, and their Spanish equivalents) and force a cache miss for these queries, whatever the similarity score.

2. Personalised queries over private data

If the RAG has access to user data (account history, private documents), two different users asking the same question must get different answers. A shared cache that ignores user context is a privacy risk.

The solution is a cache partitioned by user_id or tenant_id. This lowers the hit rate (each user’s cache is smaller) but it is the only safe option in multi-tenant architectures.

3. TTL and a stale corpus

When the corpus is updated (new documents are ingested, errors are corrected), cached answers can go out of date. A fixed TTL (24–48 hours) mitigates the problem but does not remove it.

For corpora with frequent updates, the solution is an active invalidation mechanism: when the corpus is updated in Qdrant, a job runs that identifies which cache entries might be affected (by semantic overlap with the updated chunks) and deletes them. This is the “selective cache invalidation” mentioned in the section on topics not covered.

Integration into the pipeline as middleware

The semantic cache is implemented as middleware between the API gateway and the retriever. It does not change the API contract: the client still sends queries and receives answers in the same format.

class SemanticCacheMiddleware:
 def __init__(self, cache_store, retriever, llm, threshold=0.93):
 self.cache = cache_store
 self.retriever = retriever
 self.llm = llm
 self.threshold = threshold

 async def process(self, query: str, context: dict) -> dict:
 # First layer: exact-match cache (Redis GET, O(1))
 exact = await self.cache.exact_lookup(query)
 if exact:
 return {**exact, "cache_type": "exact"}

 # Second layer: semantic cache (ANN search)
 query_embedding = await self.embed(query)
 semantic = await self.cache.semantic_lookup(
 query_embedding, threshold=self.threshold
 )
 if semantic:
 return {**semantic, "cache_type": "semantic"}

 # Miss: full pipeline
 chunks = await self.retriever.retrieve(query_embedding)
 response = await self.llm.generate(query, chunks)

 # Store for future queries
 await self.cache.store(
 embedding=query_embedding,
 query=query,
 response=response,
 ttl=context.get("ttl", 86400)
 )
 return {**response, "cache_type": "miss"}

The first exact-match layer (Redis GET) is an extra optimisation: for textually identical queries, the embedding is not even computed. The cost is a Redis operation taking microseconds. Only if there is no exact match does it fall through to the semantic lookup.

What we have not covered

Selective cache invalidation: when a subset of the corpus is updated (for example, the documents for a specific product are reindexed), you would need to identify which cache entries overlap semantically with the updated chunks and mark them stale. The mechanism involves computing similarity between the embeddings of the updated chunks and the embeddings of the cached queries, which is expensive at scale.
Multi-tenant cache: isolation vs sharing: in a SaaS product with multiple customers, a shared cache maximises the hit rate but can expose one tenant’s answers to another if filtering is not done correctly. A cache partitioned by tenant is safe but has much lower hit rates. The middle ground is a shared cache with ACL filtering applied over Qdrant’s payload filters.
Semantic cache for streaming responses: when the LLM emits tokens in streaming mode (SSE), the cache cannot easily intercept the complete answer. The options are: cache on the first miss and return the whole answer at once on hits (breaking the streaming experience), or implement a “fake streaming” that emits the tokens of the cached answer at a controlled rate.
Exact-match cache as a first layer: ahead of the semantic cache, an O(1) lookup in Redis with the query as the key can catch textually identical queries at negligible cost. The code in the previous section already shows this two-layer architecture.

References

Bang, J. et al. (2024). MeanCache: User-Centric Semantic Cache for Large Language Model Based Web Applications. arXiv:2403.02694.
Zilliz. (2023). GPTCache: A Library for Creating Semantic Cache for LLM Queries. GitHub: zilliztech/GPTCache.
Qdrant Team. (2024). Qdrant Documentation: Filtering with payload. qdrant.tech/documentation.
Manning, C. D., Raghavan, P., Schütze, H. (2008). Introduction to Information Retrieval. Cambridge University Press. Cap. 19: Web search (distribución Zipf).
Langfuse. (2024). Observability for LLM Applications. langfuse.com/docs.
Meta AI. (2024). Llama 3.1 Model Card. ai.meta.com.
Guo, Y. et al. (2023). Evaluating the Factual Consistency of Large Language Models Through Summarization. Referencia para BERTScore como métrica de evaluación de respuestas cacheadas.

PostgreSQL + Qdrant in RAG ingestion: the postman who keeps two worlds in sync

Thu, 04 Jun 2026 07:00:00 +0200

TL;DR — In a production RAG system, PostgreSQL holds the official truth about documents and Qdrant holds the vectors for search. Keeping them in sync is not trivial: if you delete a document from Postgres and do not invalidate its chunks in Qdrant, the system returns answers from ghost documents. There are two patterns to avoid this: the outbox pattern (atomic transaction plus asynchronous worker, at-least-once) and CDC with Debezium (reading the Postgres WAL directly, low latency, higher complexity). This article explains when to use each one, how to orchestrate them as microservices and what numbers to expect with bge-m3 on on-premise hardware.

The postman and the civil registry

Picture a town with two complementary offices.

The first is the civil registry: it keeps the official census. Every time someone is born, dies or moves house, the registry is the first to know. It is slow, structured, transactional. If the registry says someone exists, they exist. If it says they died, they are dead. PostgreSQL is the civil registry of your documents.

The second is the postman’s notebook: a copy optimised for finding any resident in seconds, organised by area, phonetic names and habitual routes. The postman cannot update the registry, but can search at speeds the registry would never reach. Qdrant is the postman’s notebook.

The problem is synchronisation. If the registry records a death but nobody tells the postman, he will keep trying to deliver letters to an address that no longer exists. In RAG, that translates into indexed chunks of documents that have already been deleted, edited or replaced: ghost documents that pollute the results.

How does the registry tell the postman?

Outbox pattern: every time the registry updates its ledger, it notes the change on an outbox sheet. A messenger clerk reads that sheet periodically and updates the postman’s notebook. Guaranteed, asynchronous, fault tolerant.
CDC with Debezium: the postman has a direct phone line to the registry. Every time the clerk writes something new, the phone rings and the postman updates his notebook in near real time.

The consistency gap problem

In a naive RAG architecture, the flow is:

The user uploads a document → it is inserted into Postgres with metadata.
A worker splits it into chunks, generates embeddings and upserts into Qdrant.
Retrieval uses Qdrant to find relevant chunks and Postgres to hydrate metadata.

So far so good. The problem shows up in updates and deletes:

The user edits a document → Postgres updates the row, but the vectors of the old chunks are still in Qdrant. Retrieval returns stale context.
The user deletes a document → Postgres removes the row, but the chunks remain in Qdrant. Retrieval returns chunks from a document that should no longer exist.
The permission system revokes a tenant’s access → Qdrant has no way of knowing unless there is explicit synchronisation.

This is not a theoretical problem. In living corpora (corporate wikis, knowledge bases updated daily), the consistency gap accumulates noise progressively. An internal study on production pipelines shows that without active reconciliation, 3-8% of indexed chunks correspond to documents that no longer exist in the source of truth after 30 days of operation.

The answer is not “reindex everything every night”. With 10M chunks and a non-trivial embedding model, that costs hours of compute and causes unavailability windows. The answer is change propagation with guarantees.

Outbox pattern: the outbox sheet

Mechanism

The outbox pattern solves the problem of “writing to two systems in the same operation” without distributed transactions (which are expensive and fragile).

The idea is simple: PostgreSQL is the single coordinator. When the ingestion microservice processes a document, it performs two writes in the same local transaction:

It inserts or updates the document in the documents table.
It inserts an event into the outbox_events table.

If the transaction fails, both writes are rolled back. If it succeeds, both are committed atomically. There is no inconsistent intermediate state.

-- Relevant tables
CREATE TABLE documents (
 id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
 tenant_id TEXT NOT NULL,
 title TEXT NOT NULL,
 content TEXT NOT NULL,
 checksum TEXT NOT NULL,
 updated_at TIMESTAMPTZ DEFAULT now()
);

CREATE TABLE outbox_events (
 id BIGSERIAL PRIMARY KEY,
 aggregate_id UUID NOT NULL, -- document id
 event_type TEXT NOT NULL, -- 'document.created' | 'document.updated' | 'document.deleted'
 payload JSONB NOT NULL,
 created_at TIMESTAMPTZ DEFAULT now(),
 processed_at TIMESTAMPTZ -- NULL = pending
);

-- Example of an atomic insert
BEGIN;
 INSERT INTO documents (tenant_id, title, content, checksum)
 VALUES ($1, $2, $3, $4)
 RETURNING id INTO _doc_id;

 INSERT INTO outbox_events (aggregate_id, event_type, payload)
 VALUES (_doc_id, 'document.created', jsonb_build_object(
 'tenant_id', $1,
 'title', $2,
 'checksum', $4
 ));
COMMIT;

The outbox worker

A separate process (the outbox worker) polls outbox_events where processed_at IS NULL, processes each event (chunking, embedding, upsert into Qdrant) and marks the row as processed:

UPDATE outbox_events
 SET processed_at = now()
 WHERE id = $1;

Guarantee: at-least-once. If the worker fails between the Qdrant upsert and the UPDATE, the event will be reprocessed. Qdrant tolerates idempotent upserts (same point id = overwrite), so reprocessing does not create duplicates.

Latency: it depends on the polling interval. With polling every 500ms, p50 latency is ~250ms; p99, ~500ms. Acceptable for most RAG cases, where the user does not expect to see a document indexed in under a second.

CDC with Debezium: the direct phone line

Mechanism

Change Data Capture (CDC) reads PostgreSQL’s Write-Ahead Log (WAL) directly. Postgres writes every change to the WAL before applying it to the tables; it is the mechanism it uses for replication and recovery. Debezium subscribes to a logical replication slot and turns those events into structured messages.

-- Enable logical replication in postgresql.conf
-- wal_level = logical

-- Create a replication slot for Debezium
SELECT pg_create_logical_replication_slot('debezium_slot', 'pgoutput');

The full flow:

Postgres WAL → Debezium connector → Kafka/NATS → Indexer consumer → Qdrant

Debezium emits events with the record’s before/after structure:

{
 "op": "d",
 "before": {
 "id": "550e8400-e29b-41d4-a716-446655440000",
 "tenant_id": "acme",
 "checksum": "sha256:abc123"
 },
 "after": null
}

With "op": "d" (delete), the consumer knows it must delete every point in Qdrant whose payload contains that document_id.

# Consumer: delete by payload filter
qdrant_client.delete(
 collection_name="corpus",
 points_selector=FilterSelector(
 filter=Filter(
 must=[
 FieldCondition(
 key="document_id",
 match=MatchValue(value=event["before"]["id"])
 )
 ]
 )
 )
)

Upsides and downsides

CDC removes polling and cuts latency to tens of milliseconds (WAL propagation time plus consumer processing). But it adds operational complexity: you need to manage the replication slot (unconsumed slots retain WAL indefinitely, which can fill the disk), the message broker and the consumer offset state.

Comparison: outbox vs CDC

Criterion	Outbox pattern	CDC with Debezium
Typical latency	250ms – 2s	20ms – 200ms
Delivery guarantee	At-least-once	At-least-once
Operational complexity	Low (Postgres only)	High (Debezium + broker)
WAL retention risk	None	High if the slot stalls
Idempotency required	Yes (in the indexer)	Yes (in the consumer)
Multi-table support	Manual	Automatic (any table)
Backpressure	Natural (polling)	Needs explicit design
When to choose it	Corpus < 100k docs/day, small team	Corpus > 1M docs/day, low latency critical

Rule of thumb: start with outbox. Move to CDC when the change volume goes past ~50k events/hour or when latency measured in seconds becomes unacceptable for the use case (for example, real-time news indexing).

Microservice architecture

The ingestion pipeline is made up of three microservices with cleanly separated responsibilities:

Microservice 1: Ingestor

Responsibilities: receive documents, split them into chunks and request embeddings. It does not write to Qdrant directly.

# ingestor/main.py (simplified)
from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
 chunk_size=512, # tokens, not characters
 chunk_overlap=64,
 length_function=token_count,
)

def ingest_document(doc: Document, db: Session) -> None:
 chunks = splitter.split_text(doc.content)
 with db.begin():
 db.execute(
 "UPDATE documents SET checksum=$1 WHERE id=$2",
 [doc.checksum, doc.id]
 )
 db.execute(
 """INSERT INTO outbox_events (aggregate_id, event_type, payload)
 VALUES ($1, 'document.updated', $2)""",
 [doc.id, {"chunks": chunks, "tenant_id": doc.tenant_id,
 "model": "bge-m3", "model_version": "1.0.0"}]
 )

Microservice 2: Indexer

It reads the outbox, generates embeddings by calling the inference server (vLLM or TEI) and upserts into Qdrant.

# indexer/worker.py
import asyncio
from qdrant_client import QdrantClient
from qdrant_client.models import PointStruct, VectorParams, Distance

qdrant = QdrantClient(host="qdrant-service", port=6333)

async def process_event(event: dict) -> None:
 payload = event["payload"]
 chunks = payload["chunks"]
 doc_id = str(event["aggregate_id"])

 # Batch embedding on TEI (Text Embeddings Inference)
 embeddings = await embed_batch(chunks, model="bge-m3")

 points = [
 PointStruct(
 id=f"{doc_id}_{i}",
 vector=emb,
 payload={
 "document_id": doc_id,
 "tenant_id": payload["tenant_id"],
 "chunk_index": i,
 "text": chunks[i],
 "model_version": payload["model_version"],
 }
 )
 for i, emb in enumerate(embeddings)
 ]

 # If it is an update, delete the previous chunks first
 if event["event_type"] in ("document.updated", "document.deleted"):
 qdrant.delete(
 collection_name="corpus",
 points_selector=filter_by_doc_id(doc_id)
 )

 if event["event_type"] != "document.deleted":
 qdrant.upsert(collection_name="corpus", points=points)

Microservice 3: Reconciler

The reconciler is the safety net. Periodically (every hour, for example) it compares the set of document_id values in Postgres with the set of document_id values in Qdrant. IDs present in Qdrant but absent from Postgres are ghosts: they get deleted.

# reconciler/diff.py
async def reconcile(tenant_id: str) -> int:
 pg_ids = set(await fetch_all_doc_ids(tenant_id))
 qdrant_ids = set(await scroll_all_doc_ids(tenant_id)) # paginated scroll

 orphans = qdrant_ids - pg_ids
 if orphans:
 logger.warning("Orphan chunks for %d documents", len(orphans))
 for doc_id in orphans:
 qdrant.delete("corpus", filter_by_doc_id(doc_id))

 return len(orphans)

Throughput and ingestion maths

Embedding throughput

The bge-m3 model (1024 dimensions, dense + sparse + colbert support) on a node with 4×H100 SXM (320 GB NVLink) running via vLLM or HuggingFace TEI reaches roughly 2,000 chunks/second with batch size = 256 and 512-token sequences.

$$\text{throughput} = 4 \times 500 \text{ chunks/s/GPU} = 2{,}000 \text{ chunks/s}$$

The figure of 500 chunks/s per GPU comes from public TEI benchmarks with bge-m3 on H100 SXM5, batch=256, seq_len=512 ¹.

Total re-ingestion time

For a corpus of 10M chunks:

$$t = \frac{10{,}000{,}000 \text{ chunks}}{2{,}000 \text{ chunks/s}} = 5{,}000 \text{ s} \approx 83 \text{ minutes}$$

That is pure embedding time. Adding Qdrant write latency (~0.5ms per upsert batch of 100 points):

$$t_{\text{qdrant}} = \frac{10{,}000{,}000}{100} \times 0.5\text{ ms} = 50{,}000 \text{ ms} = 50 \text{ s}$$

Estimated total for a full re-ingestion: ~85-90 minutes on a 4×H100 node.

Storage cost in Qdrant

Each bge-m3 vector has 1024 dimensions in float32 (4 bytes):

$$\text{size per vector} = 1{,}024 \times 4 \text{ B} = 4{,}096 \text{ B} = 4 \text{ KB}$$

For 10M chunks (dense vectors only):

$$\text{total vectors} = 10^7 \times 4{,}096 \text{ B} = 40.96 \text{ GB}$$

Adding the JSON payload (estimated at ~500 bytes/chunk):

$$\text{payload} = 10^7 \times 500 \text{ B} = 5 \text{ GB}$$

HNSW index (roughly 1.2× the vector size for $m=16$):

$$\text{HNSW} \approx 40.96 \text{ GB} \times 1.2 = 49.15 \text{ GB}$$

Estimated total on disk: ~95 GB for 10M chunks with dense bge-m3.

With scalar quantization (int8), the vector size shrinks 4×:

$$\text{with int8 quantization} \approx \frac{40.96}{4} + 5 + \frac{49.15}{4} \approx 27.5 \text{ GB}$$

Configuration	Vectors	HNSW	Payload	Total
float32, no quantization	40.96 GB	49.15 GB	5 GB	~95 GB
int8 scalar quantization	10.24 GB	12.29 GB	5 GB	~28 GB
binary quantization	1.28 GB	1.54 GB	5 GB	~8 GB

Binary quantization loses recall precision (~2-5% in NDCG@10), but it lets you host much larger corpora in RAM. For production where recall is critical, int8 is the usual balance point.

Recommended on-premise hardware

For a continuous ingestion pipeline in production:

Embedding node: 4×H100 SXM (320 GB, NVLink), 2× 64-core CPU (EPYC 9654), 1 TB DDR5 RAM, 100 GbE. Runs vLLM or TEI serving bge-m3. Sustained throughput: ~2,000 chunks/s with an asynchronous batch pipeline.

Qdrant node: 32-core CPU, 256 GB RAM (to keep the HNSW index in memory with 10M chunks and no quantization), 2 TB NVMe (Qdrant snapshot and WAL writes). Qdrant recommends that the HNSW index fit in RAM for p99 latency < 5ms.

PostgreSQL node: 16-core CPU, 128 GB RAM, 4 TB NVMe for the WAL (especially relevant if you use CDC with a logical replication slot; the slot retains WAL until Debezium consumes it).

Broker (if CDC): 3-broker Kafka with 500 GB NVMe per node, or NATS JetStream with 3 nodes for more modest loads.

Kubernetes manifests

Indexer Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
 name: rag-indexer
 namespace: rag-pipeline
spec:
 replicas: 3
 selector:
 matchLabels:
 app: rag-indexer
 template:
 metadata:
 labels:
 app: rag-indexer
 spec:
 containers:
 - name: indexer
 image: registry.example.com/rag-indexer:1.0.0
 env:
 - name: POSTGRES_DSN
 valueFrom:
 secretKeyRef:
 name: pg-credentials
 key: dsn
 - name: QDRANT_HOST
 value: "qdrant-service.qdrant.svc.cluster.local"
 - name: EMBEDDING_ENDPOINT
 value: "http://tei-service.embeddings.svc.cluster.local:8080"
 - name: POLL_INTERVAL_MS
 value: "500"
 resources:
 requests:
 memory: "512Mi"
 cpu: "500m"
 limits:
 memory: "2Gi"
 cpu: "2"

Reconciler CronJob

apiVersion: batch/v1
kind: CronJob
metadata:
 name: rag-reconciler
 namespace: rag-pipeline
spec:
 schedule: "0 * * * *" # hourly
 concurrencyPolicy: Forbid
 jobTemplate:
 spec:
 template:
 spec:
 restartPolicy: OnFailure
 containers:
 - name: reconciler
 image: registry.example.com/rag-reconciler:1.0.0
 env:
 - name: POSTGRES_DSN
 valueFrom:
 secretKeyRef:
 name: pg-credentials
 key: dsn
 - name: QDRANT_HOST
 value: "qdrant-service.qdrant.svc.cluster.local"
 - name: SCROLL_PAGE_SIZE
 value: "1000"
 resources:
 requests:
 memory: "256Mi"
 cpu: "250m"
 limits:
 memory: "1Gi"
 cpu: "1"

Production gotchas

Reindexing when the embedding model changes

This is the most painful problem. If you move from bge-m3 to nomic-embed-text-v2, the vectors are incompatible: they live in different embedding spaces and the cosine distances between them are meaningless.

The answer is dual-index aliasing:

Create a new collection in Qdrant: corpus_v2.
Re-embed the whole corpus with the new model and load corpus_v2.
When the new collection is complete and validated (recall test), switch the corpus_prod alias from corpus_v1 to corpus_v2.
Delete corpus_v1 once traffic has migrated.

During the migration the two indexes coexist. The retriever uses the alias, not the collection’s direct name.

Index versioning

Store model_version in the payload of every point in Qdrant. This lets you:

Filter by version during retrieval (useful in model A/B testing).
Have the reconciler detect points with an old version and reprocess them selectively.
Audit: know which model generated each embedding.

# Filter by model version at retrieval time
results = qdrant.search(
 collection_name="corpus",
 query_vector=query_embedding,
 query_filter=Filter(
 must=[
 FieldCondition(key="model_version", match=MatchValue(value="1.0.0")),
 FieldCondition(key="tenant_id", match=MatchValue(value=tenant_id)),
 ]
 ),
 limit=10
)

Namespace per tenant (multi-tenancy)

There are two strategies in Qdrant:

Strategy	Pros	Cons
Collection per tenant	Total isolation, no extra filter	N collections = N HNSW indexes in RAM
Payload filter per tenant	A single collection, less RAM	The filter adds ~10-15% search latency

For fewer than 100 tenants with large corpora (> 1M chunks/tenant), use a collection per tenant. For hundreds or thousands of tenants with small corpora, use a payload filter with tenant_id indexed:

qdrant.create_payload_index(
 collection_name="corpus",
 field_name="tenant_id",
 field_schema=PayloadSchemaType.KEYWORD
)

What we have not covered

Streaming corpus updates with near-real-time CDC: selective chunk invalidation when only one section of a document changes (incremental chunking based on content diff, not full re-chunking).
Multi-tenant corpus isolation with per-chunk ACLs: going beyond the tenant_id filter towards group, role or even individual document permissions applied at retrieval time.
Federated corpus: corpora distributed across cross-border silos where regulation (GDPR, CCPA) prevents centralising the embeddings; federated search patterns without moving data.
Zero-downtime incremental reindexing using dual-index aliasing: the full model migration protocol with rollback, recall regression testing and progressive traffic splitting.

References

HuggingFace Text Embeddings Inference — benchmarks oficiales con modelos de la familia bge en hardware A100/H100. https://github.com/huggingface/text-embeddings-inference ↩︎

Embeddings in 2026: the three families (dense, sparse, multi-vector), the model zoo and the decision that matters in production

Wed, 03 Jun 2026 03:00:00 +0200

This post opens the data sub-saga inside the six-stage LLMOps pipeline by going into the piece that holds up three-layer retrieval: the embedder. If the librarian of curation decided what enters the index and the faculty committee decided what reaches the model’s face, this post looks at the one in the middle: the cartographer who draws the map you search over.

TL;DR

The conversation about embeddings has been simplified in production to the point where “which embedder do you use” always gets the same answer: OpenAI text-embedding-3-large in the demo, bge-m3 in the “sovereign-ready” version. That simplification hides the fact that an embedder is three different models at once, dense single-vector, learned sparse (SPLADE) and multi-vector late-interaction (ColBERT), and that in 2026 the leading models do not compete within the same family: gte-Qwen2-7B-instruct and NV-Embed-v2 break MTEB in dense single-vector, SPLADE-v3 and the sparse head of bge-m3 dominate the learned lexical descriptor, Jina-ColBERT-v2 and ColNomic-7B are the strongest in multilingual multi-vector, and Snowflake Arctic Embed L 2.0 has slipped in as the small multilingual favourite with decent Matryoshka. This post takes apart the three families with their maths (InfoNCE with τ, MaxSim, SPLADE’s FLOPS regularization, Matryoshka Representation Learning), goes through the open source model zoo with the dimension, licence and niche of each one, poses the specific problem of multilingual Spanish, which cuts the list of viable models to fewer than six, gives the real storage cost per million chunks with int8 / binary / TurboQuant, describes how they are served on-premise with TEI, Infinity and vLLM --task embed, sets the minimum hardware at an RTX 4090 and the good hardware at a 4×H100 cluster, lists the seven operational traps that sink quality without warning (corpus drift, forgotten normalisation, badly chosen Matryoshka dimension, missing hard negatives, chat template slipped into the embedder, tokenizer drift, MTEB overfit) and closes with a license-clean stack for sovereign production.

The analogy: three librarians cataloguing the same book

A technical library receives a new book and, before putting it on the shelves, has to generate a searchable identifier for it. Three librarians with three different trades live in the library, and all three catalogue the same book at once:

Librarian A, the thematic one. He reads the whole book and gives it a single rich RFID tag. That tag is a vector of 1,024 numbers where each coordinate encodes a latent semantic axis (one the librarian never verbalises: he has learned it by reading a hundred million previous books). Two books about Kubernetes in production will end up with RFID tags very close together in the space even if one talks about Linkerd and the other about Cilium, because they share thematic axes. To search, you compare the question’s tag with the book’s and return the nearest ones by cosine. It is fast, it scales to millions, and it loses fine nuance. This is the dense single-vector embedder: bge-m3 in dense mode, gte-Qwen2-7B-instruct, Snowflake Arctic Embed L 2.0, multilingual-e5-large-instruct.

Librarian B, the lexical one. He summarises nothing. What he does is write on a card the weighted list of relevant terms from the book, expanded with synonyms from the field. The Kubernetes book carries on its card “kubernetes 4.2, linkerd 3.8, cilium 3.7, service-mesh 4.1, sidecar 3.2, mtls 2.9, ebpf 2.6, k8s 4.0…”, each term with a weight. The trick is that the model does the expansion: if your question says “service mesh” and the original book only said “Linkerd”, librarian B’s card did record “service-mesh 4.1” because he understood the relationship. To search, you intersect the question’s card with the book’s the old-fashioned way: inverted index, posting lists. It is decisive when the reader writes few, very specific words (product names, errors, jargon). This is the learned sparse embedder: SPLADE-v3 or the sparse head of bge-m3. It is the modern successor to BM25, not its rival; we will see why.

Librarian C, the copyist. He gives up on summarising. He takes every word on every page of the book and gives it a 128-dimension mini-RFID. He ends up with a 30,000-token book turned into 30,000 mini-RFIDs. When you search for something, the librarian compares every word of your question with every word of the book and keeps the maximum for each word of the question, adding them up. It captures nuance the other two lose by construction (proper nouns, numbers, specific phrasings), but its filing system is an order of magnitude larger in disk space. This is ColBERT-v2 / Jina-ColBERT-v2 / ColNomic-7B: late interaction, MaxSim.

And then there is the all-rounder librarian who does all three jobs in a single pass: bge-m3. A single 568 M parameter model that simultaneously returns the thematic RFID tag (1,024-d), the weighted list of terms (sparse) and the set of per-token mini-RFIDs (128-d). That is why it has settled in as the standard embedder for multilingual on-premise RAG: a single model.encode(chunk) produces the three outputs that feed hybrid retrieval without orchestrating three different models.

The same chunk indexed by three librarians. The all-rounder bge-m3 runs all three in a single 568 M parameter pass.

What an embedding really is

A text embedding is a function f : text → ℝᵈ trained so that two “semantically similar” texts produce nearby vectors in that space. The important part is not “vector”, any fixed-length hash is one, but what closeness means. Closeness is defined implicitly by the loss the model is trained with.

Almost every modern embedder is trained with InfoNCE (also called Multiple-Negatives Ranking Loss in sentence-transformers):

\mathcal{L}_{\text{InfoNCE}} = -\log \frac{\exp(\text{sim}(q, d^+)/\tau)}{\sum_{d \in \mathcal{B}} \exp(\text{sim}(q, d)/\tau)}

For each (query, doc⁺) pair the model has to assign the positive more similarity than all the other documents in the batch B, where the other documents act as free in-batch negatives. The temperature τ (typically 0.02–0.07, almost always 0.05) controls how sharp the distribution gets: lower τ means the model becomes more demanding about the positive but less stable. A large batch size |B| means many more negatives per gradient, so a better informed model. That is why embedders are trained with batch ≥ 1,024 on H100 clusters with AllGather across nodes to stack every negative in the cluster into a single effective batch.

On top of the in-batch negatives comes hard negatives mining: documents deliberately picked because they are “almost in the answer” (typically the next 10-100 neighbours from a previous BM25 / dense retrieval). Without hard negatives the model learns to discriminate the trivial and real quality on BEIR / MTEB drops 5-10 points.

This matters because the embedder family depends on what you feed into sim(·,·):

Family	`sim(q, d)`	Model output
Dense single-vector	dot product of two normalised 1024-d vectors	`f(q), f(d) ∈ ℝ^d`
Learned sparse	dot product of two 30,522-d vectors (≈80 non-zero each)	`f(q), f(d) ∈ ℝ^V`, `V =
Multi-vector	`Σᵢ maxⱼ ⟨qᵢ, dⱼ⟩` (MaxSim)	`f(q) ∈ ℝ^{

The loss is the same in all three cases, InfoNCE, but the geometry of the space changes, and with it quality, storage cost and search latency.

The three families in detail

Dense single-vector, the cartographer

The embedder reads the full chunk, passes it through an encoder transformer (XLM-RoBERTa, BERT, a Mistral decoder with a prompt) and aggregates the token representations into a single vector by:

CLS pooling: uses the embedding of the [CLS] token. Standard in BERT-base.
Mean pooling: plain average of the embeddings of all tokens. Standard in multilingual-e5-large-instruct, bge-m3 dense, Snowflake Arctic Embed L 2.0.
Last-token pooling: for decoder-LLM based embedders (e5-mistral-7b-instruct, gte-Qwen2-7B-instruct, NV-Embed-v2) that take the final token as the aggregate. It works because the model is causal and the last token “has seen” the whole context.
Latent-attention pooling: new in NV-Embed-v2. A learnable attention layer that weights tokens instead of averaging them. +2-3 MTEB points over mean pooling.

After pooling the vector is normalised to norm 1: v ← v / ‖v‖₂. With normalised vectors, cosine and dot product coincide:

\cos(q, d) = \frac{q \cdot d}{\|q\|\|d\|} = q \cdot d, \qquad \|q - d\|^2 = 2 - 2\,(q \cdot d)

That is why almost every vector DB indexes by inner product and leaves it to the user to normalise beforehand (Qdrant, Faiss IP, Milvus). If you forget to normalise the query vector but the corpus vectors are normalised, retrieval degrades silently: the query’s magnitude distorts the ranking. It is the number one bug in RAG in production.

A minimal numerical example, two normalised 4-d vectors:

q = [0.5, 0.5, 0.5, 0.5] ‖q‖ = 1
d₁ = [0.6, 0.4, 0.5, 0.5] ‖d₁‖ = 1.005, normalised [0.597, 0.398, 0.498, 0.498]
q · d₁ = 0.5·0.597 + 0.5·0.398 + 0.5·0.498 + 0.5·0.498 = 0.995

Closeness of nearly 1, as expected. If d₁ had not been normalised, q · d₁ = 1.0 and it would appear closer than any d that is perfectly aligned but with a norm < 1.005. Normalisation is not a detail: it is the contract of the space.

Learned sparse, the lexical descriptor

SPLADE-v3 (Naver, March 2024) has consolidated the modern version of the lexical librarian. Internally it is a small BERT (~110 M parameters, DistilBERT/BERT base) that produces, for each input token, a distribution over the whole vocabulary (30,522 dimensions in BERT WordPiece), and then does max-pool over the tokens:

w_j = \max_{i \in \text{seq}} \log\bigl(1 + \text{ReLU}(W_{ij})\bigr)

where Wᵢⱼ is the logit of input token i for vocabulary term j. The log(1+ReLU) saturates high logits (it stops a single word from dominating the vector) and the ReLU cuts the negatives. The result is a 30,522-dimension vector of which typically 50-200 entries stay non-zero.

The key part is the FLOPS regularization added to the loss during training:

\mathcal{L}_{\text{FLOPS}} = \lambda \cdot \sum_{j=1}^{V} \bar{w}_j^2, \qquad \bar{w}_j = \frac{1}{|B|}\sum_{i \in B} w_{ij}

It penalises the expected cost of the posting lists: if a vocabulary word appears on average in many documents, adding it to a new document costs double. The model learns to generate sparse vectors by construction.

What does this do to the text “Linkerd 3.8 brings mTLS by default”? The model does not only write the literal terms, it also writes, with a smaller but non-zero weight, “service-mesh”, “kubernetes”, “tls”, “sidecar”, “envoy”, “istio” (its competitor, also semantically related), “encryption”, “k8s”. That semantic expansion of the document is what sets SPLADE apart from BM25. BM25 only knows what was literally in the text; SPLADE knows what an expert would add as a descriptor.

In practice SPLADE-v3 beats BM25 by 3-6 MRR@10 points on MS MARCO and dominates BEIR zero-shot. The cost is ~2-4× the query latency of BM25 over the same inverted index, which static pruning can mitigate.

For the multilingual case, bge-m3 with its sparse head is the only maintainable option: SPLADE-v3 is trained in English and the multilingual ports are in an experimental state.

Multi-vector, the copyist

ColBERT-v2 (Stanford, NAACL 2022) introduced the late interaction paradigm. Instead of compressing the document to a single vector, it leaves it as a matrix (|d|, k) with a k-dimension vector per token. The similarity between query and document is computed token by token and aggregated with MaxSim:

s(q, d) = \sum_{i \in q} \max_{j \in d} \langle q_i, d_j \rangle

What is being computed: for each word of the query, find the document token that fits it best and add that similarity. The sum is over the query, not over the document. This lets a 30,000-token document compete fairly with a 200-token one, because the query always adds |q| terms.

Why does it give more quality than dense single-vector? Because compressing to one vector loses information about where each idea was. If the query is “which version brought mTLS by default in Linkerd”, the dense summary of the document only knows that the chunk is about “Linkerd and mTLS”; ColBERT’s copyist can match “which version” with “3.8” because it stores the embedding of the 3.8 token separately. On BEIR / out-of-domain, late interaction beats single-vector by between +2 and +6 nDCG@10 with the same backbone.

The price is storage. Per document, a 1024-dimension dense single-vector in fp16 takes 1024 × 2 = 2 KB. ColBERT-v2 with 128-dimension tokens for a 256-token chunk takes 256 × 128 × 2 = 65,536 B ≈ 64 KB: 32× more space. With ColBERT-v2’s nbits=2 residual compression it drops to ~16 KB (8×). Jina-ColBERT-v2 adds Matryoshka over the token dimensions (truncatable to 128 / 96 / 64), cutting another 50%.

For 1 million chunks:

Family	Per doc	Total
Dense fp32 (1024-d)	4,096 B	4.0 GB
Dense fp16 / halfvec	2,048 B	2.0 GB
Dense int8 (SQ)	1,024 B	1.0 GB
Dense binary (1 bit/d)	128 B	128 MB
SPLADE (≈ 80 terms × 8 B)	~640 B	~640 MB
ColBERT fp16 (256 tok × 128 d)	65,536 B	64 GB
ColBERT residual `nbits=2`	~16,000 B	~16 GB
Jina-ColBERT-v2 (MRL token 64)	~8,000 B	~8 GB

ColBERT in on-premise production is reserved for corpora of up to a few million chunks, or applied only as a reranker over the top-100 from the first committee (dense + sparse), as described in the reranker post.

Matryoshka, the truncatable dimension

An operational lever that changed the conversation about embeddings between 2024 and 2026 is Matryoshka Representation Learning (Kusupati et al., NeurIPS 2022). The trick: during training, on top of the loss over the full D-dimension vector, the same loss is computed over prefixes of the vector:

\mathcal{L}_{\text{MRL}} = \sum_{k \in \{64, 128, 256, 512, 1024\}} \alpha_k \cdot \mathcal{L}_{\text{InfoNCE}}\bigl(\text{emb}[:k]\bigr)

The first k dimensions of the embedding are trained to be, on their own, a valid embedding. At inference time, if you want a cheaper embedding, you truncate the vector: the first quarter is already a usable embedding. Without Matryoshka, truncating wrecks the geometry: the first 256 dimensions of an embedding trained only at 1024-d encode nothing coherent.

Typical degradation in MTEB nDCG@10 when truncating an MRL embedder:

Truncation	Average loss
1024 → 512	-1 to -2 points
1024 → 256	-3 to -5 points
1024 → 128	-5 to -8 points
1024 → 64	-8 to -12 points

Native MRL models in 2026 (the ones that let you pick the dimension at runtime without retraining):

jina-embeddings-v3 (1024 → 32, fine steps, CC-BY-NC-4.0)
jina-embeddings-v4 (2048 → 128, multimodal text+image, CC-BY-NC-4.0)
nomic-embed-text-v2-moe (768 → 256, Apache 2.0)
Snowflake-arctic-embed-l-v2.0 (1024 → 256, Apache 2.0)
mxbai-embed-large-v1 and mxbai-embed-2d-large-v1 (the latter also truncatable in layer depth)
Stella_en_1.5B_v5 (multiple steps 512 / 768 / 1024 / 2048 / 4096 / 6144 / 8192, MIT, English only)
text-embedding-3-large (OpenAI, 3072 → 256, API only)
voyage-3 family (1024 → 256 / 512 / 1024 / 2048, API only)

Production recommendation: always use an MRL model even if you do not truncate at the start, because it simplifies the future quantisation decision. And above all, evaluate truncation on your real corpus: the average MTEB degradation of “-3 points” becomes “-12 points” in a niche domain.

The 2026 open source model zoo

What follows is a spec sheet per model. Verified as of 2026-06: HuggingFace cards + reference papers + MTEB / MMTEB leaderboard.

Dense single-vector

Model	Params	Dim	Tokens	Languages	Licence	Distinctive
`BAAI/bge-m3` (dense)	568 M	1024	8192	100+	MIT	Tri-mode (dense + sparse + colbert) in one forward. De facto standard for multilingual on-prem.
`Snowflake/snowflake-arctic-embed-l-v2.0`	568 M	1024 (MRL → 256)	8192	~100	Apache 2.0	Trained for multilingual + English without degrading either. MIRACL 55.8.
`intfloat/multilingual-e5-large-instruct`	560 M	1024	512	~100	MIT	Veteran multilingual baseline. Short window.
`intfloat/e5-mistral-7b-instruct`	7.1 B	4096	4096	English	MIT	First decoder-as-embedder to break MTEB. English.
`Alibaba-NLP/gte-Qwen2-7B-instruct`	7 B	3584	32,768	100+	Apache 2.0	The only one with 32k context. Top MTEB-en, strong multilingual.
`nvidia/NV-Embed-v2`	7.85 B	4096	32,768	English	CC-BY-NC-4.0	Latent-attention pooling. Top quality. Licence blocker in prod.
`Linq-AI-Research/Linq-Embed-Mistral`	7 B	4096	4096	English	CC-BY-NC-4.0	Top MTEB retrieval Aug 2024. Non-commercial.
`NovaSearch/stella_en_1.5B_v5`	1.54 B	8192 (multiple MRL)	8192	English	MIT	Small + rich MRL. English.
`BAAI/bge-multilingual-gemma2`	9 B	3584	8192	100+	Gemma	High quality, the Gemma licence restricts redistribution.
`BAAI/bge-en-icl`	7 B	4096	8192	English	MIT-style	In-context learning from examples in the prompt.
`mixedbread-ai/mxbai-embed-large-v1`	335 M	1024 (MRL)	512	English	Apache 2.0	MRL + native binary. Short window.
`jinaai/jina-embeddings-v3`	570 M	1024 (MRL → 32)	8192	89	CC-BY-NC-4.0	Per-task LoRA. Non-commercial without a licence.
`nomic-ai/nomic-embed-text-v2-moe`	475 M / 305 M active	768 (MRL → 256)	512	~100	Apache 2.0	First general-purpose MoE in embeddings.

Learned sparse

Model	Params	Vocab	Tokens	Languages	Licence	Distinctive
`naver/splade-v3`	110 M	30,522	512	English	CC-BY-NC-SA-4.0	SOTA learned sparse. Non-commercial.
`BAAI/bge-m3` (sparse head)	568 M	XLM-R vocab	8192	100+	MIT	The only license-clean multilingual option.

Multi-vector (late interaction)

Model	Params	Dim/token	Tokens	Languages	Licence	Distinctive
`colbert-ir/colbertv2.0`	110 M	128	512	English	MIT	The original paper, the basis for everything.
`jinaai/jina-colbert-v2`	560 M	128 / 96 / 64 (MRL)	8192	89	Apache 2.0	The license-clean multilingual multi-vector.
`nomic-ai/colnomic-embed-multimodal-7b`	7 B	3584 (Qwen2-VL)	—	~100	Apache 2.0	Multimodal text+image multi-vector. Vidore-v2 open SOTA.

The leaderboard, with caution

MTEB / MMTEB (Massive Multilingual Text Embedding Benchmark, Enevoldsen et al., arxiv 2502.13595) is the standard thermometer. Top retrieval on MMTEB in mid-2026 is dominated by Qwen3-Embedding-8B (~70.6 multilingual avg) and Llama-Embed-Nemotron-8B. Below them, the 7B models (gte-Qwen2-7B, NV-Embed-v2) and the 568M ones (bge-m3, Snowflake-Arctic-L-2.0) compete task by task.

Trap: MTEB has started to saturate through dataset contamination. The higher the ranking, the more likely it is that the model saw subsets of the evaluation datasets during training. The rule in production: the leaderboard is for discarding bad models, not for picking the best one. The final decision is made on your own domain eval set, generated with the LLM-as-judge or evals recipe.

The specific problem of multilingual Spanish

For a Spanish customer serving corporate, legal or support documentation in Spanish (and often with Catalan / Portuguese / English mixed in), the embedder zoo narrows to fewer than six viable models. The operational exclusions:

English-only models: e5-mistral-7b-instruct, stella-en-1.5B-v5, Linq-Embed-Mistral, mxbai-embed-large-v1, NV-Embed-v2, SPLADE-v3. They cut Spanish performance below an acceptable level: translating the query into English before searching is one route, but it introduces latency, tokenisation drift and another model dependency.
Models with a non-commercial licence: jina-embeddings-v3, jina-embeddings-v4, NV-Embed-v2, Linq-Embed-Mistral. They work for a PoC, but commercial production needs an explicit agreement with the vendor. Unless you have the licence signed, they have to be excluded.
Models with a Gemma licence: bge-multilingual-gemma2. Allowed for internal use, awkward to redistribute weights to a customer.

The ones left, ordered by practical choice in sovereign production:

BAAI/bge-m3 — MIT, 568 M, 100+ languages (including Spanish and Catalan, explicitly trained), 8,192 tokens, tri-mode dense+sparse+colbert. A reasonable default. Fits in an RTX 4090. TEI and Infinity serve it natively.
Snowflake/snowflake-arctic-embed-l-v2.0 — Apache 2.0, same size, explicit Matryoshka, better MIRACL/CLEF than bge-m3 on some multilingual tasks, no sparse head. If the priority is pure MMTEB in Spanish.
intfloat/multilingual-e5-large-instruct — MIT, 560 M, veteran baseline. Its 512-token window is the big limitation: long documents have to be split first. If what you already have in production works, do not migrate out of fashion.
Alibaba-NLP/gte-Qwen2-7B-instruct — Apache 2.0, 32k context, high quality in Spanish (Qwen2 is well trained on Spanish). If the chunks are long (more than 4k tokens) and you have the GPU to serve it (it does not fit in a 4090; it does in an H100). It fits alongside an LLM on an H100 80GB with care.
nomic-ai/nomic-embed-text-v2-moe — Apache 2.0, 305 M active, MRL, ~100 languages. If latency and cost per token rule: the MoE gives it disproportionate throughput for its quality.
jinaai/jina-colbert-v2 — Apache 2.0, multilingual multi-vector, as a reranker or as the main retrieval on a small corpus (< 1 M chunks). The only license-clean multi-vector in Spanish.

The rule of thumb: bge-m3 as dense+sparse on the front line, jina-colbert-v2 as a third reranking layer when the use case warrants it, and Snowflake Arctic L 2.0 as the alternative if the corpus-specific eval prefers its geometry.

Serving embeddings on-premise

Three engines split the on-prem embedding serving landscape in 2026, with different profiles.

Text Embeddings Inference (TEI), the standard

huggingface/text-embeddings-inference is a server written in Rust with a Candle / ONNX backend, integrated FlashAttention and dynamic batching by tokens. It exposes an OpenAI-compatible /v1/embeddings API and supports the three bge-m3 modes simultaneously from version 1.5 onwards. For multilingual production it is the obvious default.

# values.yaml — TEI serving multilingual bge-m3 on an RTX 4090
image: ghcr.io/huggingface/text-embeddings-inference:1.5
args:
 - --model-id=BAAI/bge-m3
 - --pooling=cls
 - --max-batch-tokens=16384
 - --max-concurrent-requests=512
 - --dtype=float16
resources:
 limits:
 nvidia.com/gpu: 1

Indicative throughput with bge-m3, fp16, 512-token sequence, batch 32:

RTX 4090 (24 GB): ~8–15 k tokens/s
A100 80 GB: ~60 k tokens/s sustained
H100 80 GB: ~40–80 k tokens/s, with fp8 about 50% more

(The ranges are approximate and depend on the real batch, the average sequence length and compilation with FA2/FA3.)

Infinity, the flexible one

michaelfeil/infinity (MIT) is a multi-model FastAPI server able to load bge-m3, Snowflake Arctic, Jina-v3, Nomic, ColPali, CLAP and rerankers simultaneously behind the same OpenAI-style API. Backend PyTorch + Optimum (ONNX/TensorRT) or CTranslate2. Useful when you need to serve several different embedders (one for text, another for code, another for images) behind a single endpoint, or when the model does not yet have TEI support.

vLLM `--task embed`, for the 7B embedders

When the embedder is really an LLM decoder turned into an embedder (e5-mistral-7b-instruct, gte-Qwen2-7B-instruct, NV-Embed-v2, Stella-1.5B), the natural place to serve it is vLLM, which already has the PagedAttention and continuous batching stack in production:

vllm serve Alibaba-NLP/gte-Qwen2-7B-instruct \
 --task embed \
 --dtype bfloat16 \
 --max-model-len 32768 \
 --trust-remote-code

vLLM detects the correct pooling (last-token on the Qwen / Mistral based ones) and exposes an OpenAI-compatible /v1/embeddings. For inference clusters already running vLLM with an LLM on another port, it is the natural way to serve the embedder without standing up another stack.

fastembed, the lightweight one

qdrant/fastembed loads bge-small, MiniLM, ColBERT and BM25/SPLADE sparse in ONNX-CPU. It is not competitive on throughput against TEI/Infinity with a GPU, but it is the right option when you have to serve embeddings on a NUC node without a GPU (see mixed NVIDIA + Intel environments) or when the embedder is part of the client (a preview in a UI, prior scoring at the edge).

Storage, quantisation and the corpus arithmetic

The embedding does not stay in the embedder’s memory: it lives in the vector DB index and is materialised every time you ingest a new chunk. The storage cost calculation is what decides the final dimension, not MTEB quality. For 1 million chunks with a dense embedder at 1024-d:

fp32 : 1,024 dims × 4 B × 1 M = 4,096 MB ≈ 4.0 GB
fp16/halfvec: 1,024 dims × 2 B × 1 M = 2,048 MB ≈ 2.0 GB
int8 (SQ) : 1,024 dims × 1 B × 1 M = 1,024 MB ≈ 1.0 GB
binary : 128 B × 1 M ≈ 128 MB

The quantisation options in order of real use (mid-2026):

halfvec (fp16): the default in pgvector 0.7+ and in any serious vector DB. Zero MTEB loss, 2× compression. Always turn it on.
Scalar Quantization int8 (SQ): each vector component is mapped to int8 with a global min/max. Typical recall@10 loss: 0–1%. 4× compression. The Qdrant default, supported in Milvus and Weaviate.
Binary quantisation: bit = sign(v_i - μ_i). 32× raw compression. Cold loss 5–15%. Mitigated with a Hadamard / TurboQuant pre-rotation (Qdrant 1.18, Dec 2025): it pre-multiplies by a random orthogonal matrix that spreads the energy across dimensions before binarising. After TurboQuant the loss drops to 1–3%. It also combines with rescoring over the original fp16 for the top-100 candidates.
Product Quantization (PQ): the FAISS classic. Up to 64× compression, 2–5% loss. More complex to operate (it needs a trained codebook); in 2026 it has given ground to binary + rescoring.

A corpus of 100 million chunks (a real figure for a large corporate RAG) with dense bge-m3:

Format	Total
fp32	400 GB
fp16	200 GB
int8	100 GB
binary + Hadamard	12.5 GB

The difference between 200 GB and 12.5 GB is the difference between needing a dedicated vector DB node with 8 NVMe in RAID and fitting in the RAM of a single node. For large corpora, quantisation is no longer an optimisation: it is the only way to operate.

The vector DB integration

The vector DBs of 2026 have become hybrid DBs that index all three types at once. The quick map:

Vector DB	Native hybrid	Multi-vector / ColBERT	Quantisation
Qdrant ≥1.10	RRF/DBSF in `query_points` with dense + sparse + colbert in one collection	Yes, native (one-shot MaxSim)	SQ int8, binary, TurboQuant 1.18
Weaviate	`hybrid(alpha=0.75)` BM25 + dense, multi-target named vectors	Yes, as a multi-vector named vector	PQ, SQ, rotational 8-bit BBQ
Milvus ≥2.4	Multi-vector + sparse in the schema; 2.5 adds native BM25 full-text	Multi-vector field, MaxSim orchestrated from the client	SQ, PQ, CAGRA GPU
pgvector 0.7+/0.8	`halfvec`, `sparsevec`, `bit`; HNSW for all three	Not native (separate-table workaround)	`binary_quantize()`, halfvec, rescoring with exact `<#>`
Elasticsearch / OpenSearch	`sparse_vector` (ELSER, SPLADE) + `dense_vector` HNSW; RRF	OpenSearch 3.x yes	ES 9: `int8_hnsw` by default, BBQ binary quantization

For sovereign on-prem production in Spanish, the easiest combination to operate in 2026 is Qdrant + bge-m3: a single collection indexes the three modes of the same model, the hybrid query with RRF is done in one call, TurboQuant quantisation brings the corpus down to manageable levels, and the operator is a Go binary with simple backups to S3/MinIO. pgvector + bge-m3 is the other reasonable option when you already have Postgres with HA and do not want to add a second DB to the operational inventory; you lose native multi-vector, but you gain cross-cutting SQL over the chunks.

The HNSW parameters you absolutely have to touch:

M: connections per node in the graph. 16–32 typical. Higher means more recall, more RAM. For small corpora (<1 M) M=16; for medium corpora (10 M) M=24; for large corpora M=32+IVF-PQ or M=32+binary.
ef_construction: search width during construction. 100–400. Higher means a better graph, slower construction. Build with ef_construction=400 even if it is slow; you pay it once.
ef_search: width during the query. 50–200. The main knob for the recall/latency trade-off at runtime. Start at 64 and measure.

Implications for on-premise inference

The embedder does not share hardware with the LLM as comfortably as it might seem. The numbers:

bge-m3 (568 M) takes about 568 × 2 = 1,136 MB in fp16 for the weights, plus the batch KV cache, plus temporary activations. In practice it is served comfortably in 6–8 GB of VRAM even at a high batch. It fits alongside a 7B-Q4 LLM on an RTX 4090.
gte-Qwen2-7B-instruct needs ~14 GB fp16 for weights alone. It does not fit next to a 7B LLM on a 4090; on an H100 80 GB it does, with care over simultaneous batching.
jina-colbert-v2 (560 M) takes ~1.1 GB of weights, but the storage of the multi-vector index is the real cost: 8 GB per million chunks even with Matryoshka and compression.

On the RTX 4090 (24 GB)

A realistic minimum stack for a Spanish RAG with a corpus under 1 M chunks:

GPU 24 GB ┐
 ├─ TEI bge-m3 (dense + sparse + colbert) │ ~6 GB VRAM, ~12 k tok/s
 └─ vLLM Qwen2.5-7B-Instruct AWQ Q4 │ ~8 GB VRAM, ~80 tok/s
CPU/RAM ┐
 ├─ Qdrant with bge-m3 dense + sparse + colbert │ ~3 GB RAM per M chunks
 └─ FastAPI gateway (LiteLLM)

It serves a few tens of RAG QPS with decent multilingual quality. It is the PoC configuration and the deployment for a small site.

On the 4×H100 80 GB cluster

For the production case with several million chunks and an SLO of p99 < 500 ms:

H100 #1 (80 GB) ── vLLM Qwen3-72B-Instruct AWQ + Qwen2.5-7B speculative ┐
H100 #2 (80 GB) ── vLLM gte-Qwen2-7B-instruct (embedding 32k ctx) │ LLM + large embed
H100 #3 (80 GB) ── TEI bge-m3 multi-tenant + jina-colbert-v2 reranker │ medium embed
H100 #4 (80 GB) ── Hold-out for canary / shadow │ see canary post
 ┘
Qdrant cluster (3 CPU nodes + NVMe) ── 100 M chunks indexed (binary + TurboQuant + rescoring)

This configuration separates the large LLM from the large embedder (they share the Qwen2 architecture but compete for VRAM if put on the same GPU) and leaves a whole H100 for variants in canary. bge-m3 fits with room to spare alongside the reranker on a single H100, serving tens of thousands of requests/min.

The seven operational traps of the embedder

Not normalising the query vector. Cosine and dot product coincide only when both vectors are unit vectors. If you forget v ← v / ‖v‖₂ in the client, the results are “almost right”, the top-1 is still correct on trivial queries, the top-10 no longer is, and nobody notices until RAG quality drops 8 points. Fix: bake normalisation into the embedder adapter, not into the client.
Chat template slipped into the embedder. Some LLM-based embedders (e5-mistral-7b-instruct, gte-Qwen2-7B) expect a specific instruction prompt before the text to embed ("Instruct: Retrieve relevant passages\nQuery: ..."). Forgetting it leaves performance ~5 MTEB points lower. Fix: read the usage_template in the model card and put it into the embedder wrapper.
Badly chosen Matryoshka dimension. The default in many Qdrant / pgvector clients is dim=768. If your embedder is natively MRL at 1024 → 768, fine. If it is 1024 without MRL, truncating to 768 wrecks the space (typical loss -8 MTEB points). Fix: use the model’s native dim and truncate only when storage demands it, and only on MRL models.
Missing hard negatives in fine-tuning. When the embedder is fine-tuned with your own data (which should be standard practice for corporate RAG), if the mini-batch only carries positives and in-batch negatives from the same domain, the model learns that anything outside the domain is negative, but inside the domain it does not discriminate. Fix: mine hard negatives with BM25 / dense from your own corpus before fine-tuning.
Corpus drift without reindexing. When you retrain or replace the embedder but only apply the new model to new chunks, you end up with the index mixing two incompatible geometries. Chunks from the old and the new model are not comparable by cosine. Fix: every change of embedder is a full reindexing of the corpus, planned as an operational retrain.
Tokenizer drift between client and model. The Python client that prepares the queries uses its own tokenizer (sometimes tiktoken by default) and truncates at 8,192 tokens. The embedder uses XLM-R with sentencepiece and truncates at 8,192 of its own tokenizer. Long queries are truncated differently; the corpus embeddings are consistent but the query ones are not. Fix: use the model’s tokenizer in the client or in the wrapper.
MTEB overfit as a selection guide. The MTEB leaderboard has become a contaminated metric: there is evidence that leading models saw subsets of the evaluation datasets during training. The model that is +0.5 points above the second is not necessarily better for your domain. Fix: your own domain eval set (100-300 labelled query-doc pairs) run with the evals recipe decides.

License-clean stack for sovereign production

Let us write down the recommended ending. For a Spanish organisation serving corporate RAG on-prem under ENS / ISO 42001 / EU AI Act, with a corpus of 1-50 M chunks in Spanish + English + Catalan:

Layer	Component	Licence	Rationale
Dense embedder	`BAAI/bge-m3`	MIT	Robust multilingual, 8k tokens, license-clean, served by TEI
Sparse embedder	`bge-m3` sparse head	MIT	Same pass as the dense one, no second model needed
Layer 2 reranker	`BAAI/bge-reranker-v2-m3`	MIT	Multilingual cross-encoder from the same team
Layer 3 reranker (optional)	`jinaai/jina-colbert-v2`	Apache 2.0	License-clean multilingual multi-vector
Embed server	`TEI` + `Infinity` for multi-model	Apache 2.0 / MIT	Supported stack
Vector DB	`Qdrant` (preferred) or `pgvector 0.8`	Apache 2.0 / PostgreSQL	Native hybrid + quantisation
Quantisation	`int8 SQ` + `binary + TurboQuant + rescoring`	Apache 2.0	Cuts the corpus 16×–32× with < 3% loss
Minimum hardware	RTX 4090 24 GB	—	For PoC and small sites
Production hardware	4×H100 80 GB cluster	—	For RAG with an SLO of p99 < 500 ms

The alternative stack, if explicit MMTEB in Spanish weighs more than the tri-modality of bge-m3: replace bge-m3 with Snowflake/snowflake-arctic-embed-l-v2.0 (Apache 2.0, MRL → 256) and explicitly add SPLADE-v3 or plain BM25 for the sparse layer. It loses the elegance of the single forward and gains 1-2 points on Spanish MIRACL.

Conclusion

The embedder is the easiest piece of a RAG to oversimplify and the one that decides real quality the most once everything else is in place. The three families (dense, sparse, multi-vector) are not three options to choose between but three trades that bge-m3 performs in a single pass and that hybrid retrieval consumes in parallel. The maths that matters is modest, InfoNCE with τ, MaxSim, FLOPS regularization, MRL, but the operational traps are many and quiet: forgotten normalisation, missing chat template, badly chosen Matryoshka dimension, tokenizer drift. For sovereign production in Spanish the list of viable models fits in fewer than ten, and the real decision comes down to “bge-m3 or Snowflake Arctic L 2.0”, with jina-colbert-v2 added as layer three when the fine-grained quality justifies the cost. The license-clean stack fits on an RTX 4090 for a PoC and scales to a 4×H100 cluster for real production.

References

Chen et al. M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. arXiv:2402.03216. https://arxiv.org/abs/2402.03216
Sturua et al. Jina Embeddings v3: Multilingual Embeddings With Task LoRA. arXiv:2409.10173. https://arxiv.org/abs/2409.10173
Günther et al. Jina Embeddings v4: Universal Embeddings for Multimodal Multilingual Retrieval. arXiv:2506.18902. https://arxiv.org/abs/2506.18902
Nussbaum et al. Nomic Embed v2: Multilingual Mixture of Experts. arXiv:2502.07972. https://arxiv.org/abs/2502.07972
Wang et al. Improving Text Embeddings with Large Language Models (E5-Mistral). arXiv:2401.00368.
Yu et al. Arctic-Embed 2.0: Multilingual Retrieval Without Compromise. Snowflake, 2024-12. https://www.snowflake.com/blog/arctic-embed-2-multilingual/
Lee et al. NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models. arXiv:2405.17428.
Khattab y Zaharia. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. SIGIR 2020.
Santhanam et al. ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction. arXiv:2112.01488.
Jha et al. Jina-ColBERT-v2: A General-Purpose Multilingual Late Interaction Retriever. arXiv:2408.16672.
Lassance et al. SPLADE-v3. arXiv:2403.06789.
Kusupati et al. Matryoshka Representation Learning. NeurIPS 2022, arXiv:2205.13147.
Enevoldsen et al. MMTEB: Massive Multilingual Text Embedding Benchmark. arXiv:2502.13595.
Hugging Face Text Embeddings Inference. https://github.com/huggingface/text-embeddings-inference
Michael Feil. Infinity. https://github.com/michaelfeil/infinity
Qdrant. TurboQuant 1.18 release notes. https://qdrant.tech/articles/turboquant-quantization/
pgvector. Release notes 0.7 / 0.8. https://github.com/pgvector/pgvector
Hugging Face. Embedding Quantization. https://huggingface.co/blog/embedding-quantization

Reranker and hybrid retrieval: the committee that decides the 5 chunks the LLM will actually read

Mon, 25 May 2026 14:00:00 +0200

This post takes apart the retrieval layer inside the RAG piece of the six-stage LLMOps pipeline. It sits directly on top of the corpus curation post: if the librarian decided what enters the index, the committee in this post decides what leaves the index for the LLM’s face. It is the stage that moves the real quality of a RAG in production the most, and the one most teams solve with “dense top-k and off we go”.

TL;DR

The most widespread antipattern in real RAG is: curated corpus, dense embeddings with bge-m3, the user’s query straight through, top_k=5 of cosine similarity and off to the LLM. The system delivers answers that look good in the demo, mediocre with real traffic and dreadful with three-word queries, technical jargon, internal abbreviations or anything outside the embedder’s fine-tuning domain. The root cause is almost never the model: it is that a single pass of dense retrieval over the raw query is structurally insufficient for the cases a real user produces. In 2026 the field has consolidated the three-layer retrieval pattern: a broad layer that mixes dense + sparse (BM25 / SPLADE) via Reciprocal Rank Fusion to secure recall, a fine layer with a cross-encoder reranker (BAAI/bge-reranker-v2-m3, Cohere Rerank 3, mxbai-rerank-large) that reorders the 30-50 candidates by true relevance, and optionally a late interaction layer with ColBERT-v2 or an LLM reranker for critical cases. On top of it all, a query rewriting pattern that rewrites malformed queries and HyDE, which generates a hypothetical document so you embed the expected answer instead of the question. This post goes through them one by one: why each layer exists, what latency budget it consumes, the minimum maths of RRF and of the recall/precision trade-off, the dominant OSS stack in 2026, the on-premise hardware for serving it under data sovereignty, and the seven operational traps that kill the stage.

The analogy: the committee for a professorial chair

A university department has a chair to fill and receives 3,000 applications. Nobody is going to read 3,000 CVs in depth. The real process is organised in rounds:

Round 1, mass pre-screening. A system of automatic filters reads each CV in seconds. One route looks at the keywords of the post (publications in specific journals, languages, certifications); another route looks at the overall profile (research area, h-index, career). The two routes run in parallel. Each proposes its shortlist of 30-50 candidates. The two rankings are combined and what remains is a list of some 50-100 candidates for the next round to act on. The metric that matters here is recall: not losing the good candidate through a filter error.

Round 2, careful reading. A member of the panel reads each CV on the shortlist thoroughly and scores it against the real profile of the post. It is slow, 10-20 minutes per CV, but it discriminates far better than the automatic filter. It reorders the 100 candidates into a fine ranking and keeps the 5 finalists. The metric that matters here is precision: that the top-5 is genuinely the real top-5.

Round 3 (optional), personal interview. Long interviews are held for the 5 finalists. Face to face, questions adapted to each candidate’s profile, a check against the reality of the post. It is extremely expensive and very slow, but for critical chairs it justifies the cost.

The retrieval of a modern RAG works exactly like this. Change the vocabulary and the structure holds:

The equivalence is exact. The broad layer guarantees that the right chunk is on the shortlist (recall). The fine layer orders well inside the shortlist (precision). Each layer is justified by an operational asymmetry: the broad one is cheap and parallelises, the fine one is expensive and sequential. Solving everything with the expensive one, a single cross-encoder over the whole corpus, is operationally unsustainable. Solving everything with the cheap one, a single dense top-k, is functionally insufficient.

The problem: why dense on its own is not enough

The argument against dense-only is not theoretical. It is that the queries a real user produces have patterns the dense bi-encoder does not capture well by construction:

Short queries. “IRPF dates 2026” goes through the embedder and produces a vector close to any chunk that mentions dates and IRPF, including chunks from 2023, chunks of generic calendars, chunks of corporate IRPF when you were asking about personal income tax. BM25 disambiguates better because the weight of the infrequent terms (“2026”, “IRPF”) dominates the score.

Infrequent or internal technical terms. Product codes (FBR-X42-PRO), organisation abbreviations (PYM-23-bis), internal variable names. The embedder, trained on a public corpus, breaks them down into sub-tokens and pushes them into the “noisy” zone of the latent space. Two completely different codes can end up 0.02 cosine distance apart. BM25 treats them as exact tokens and scores them correctly.

Polysemy. “Java” as a language vs Java as an island vs Java as coffee. The dense embedding averages the meanings; the ranking becomes mediocre. Here dense can beat BM25 if the query carries enough context, but that “if” is rarely met with real user queries.

Synonyms and rephrasings. “How do I cancel my subscription” vs a chunk titled “Service termination procedure”. Here dense beats BM25: the semantic space brings them together even though they share no tokens. It is exactly the case where dense shines and BM25 fails.

The operational conclusion: no single retriever covers the four cases. The combination covers them all. And for the cases where both retrievers fail (very ambiguous queries, contradictory context between chunks), the third layer comes in: the reranker that reads the query and each chunk together with a cross-encoder, and produces a real relevance score far more discriminating than cosine similarity.

Layer 1 — Hybrid retrieval: dense + sparse

Why BM25 is still alive in 2026

BM25 is from 1994. It is not deep learning. It needs no GPU. And it is still competitive with dense retrievers of hundreds of millions of parameters on many retrieval benchmarks (BEIR, 2021-2024). Its formula is simple:

$$ \mathrm{BM25}(q, d) = \sum_{t \in q} \mathrm{IDF}(t) \cdot \frac{\mathrm{tf}(t,d) \cdot (k_1 + 1)}{\mathrm{tf}(t,d) + k_1 \cdot (1 - b + b \cdot \frac{|d|}{\overline{|d|}})} $$

Where $\mathrm{tf}(t,d)$ is the frequency of term $t$ in document $d$, $|d|$ the document length, $\overline{|d|}$ the mean length of the corpus, and IDF the inverse document frequency. $k_1$ and $b$ are hyperparameters (typically $k_1 = 1.2$, $b = 0.75$). In practice you use pyserini, Tantivy, Elasticsearch or OpenSearch and never write the formula.

SPLADE (Formal et al., 2021) is the neural generation of BM25: it learns sparse weights over the BERT vocabulary, expanding each token to its related terms. It keeps BM25’s interpretability (you can see which query terms match which document terms) and beats plain BM25 on BEIR. The price is that you need GPU inference over the query (and over each document at indexing time, once).

The rule of thumb in 2026:

Corpus < 10M chunks, queries with a broad vocabulary: plain BM25 via Elasticsearch / Tantivy. Trivial indexing, 1-5ms queries.
Corpus 10M-100M, specific domain: SPLADE-v3 indexed in Elasticsearch or Vespa. Slow indexing (you need a GPU once), 5-20ms queries.
Corpus > 100M, latency critical: BM25 with external query expansion (synonyms, query2doc) is the operational option, with SPLADE reserved for the critical segments.

Dense bi-encoder: the second lane

The bi-encoder produces one vector per chunk and another per query, and relevance is the cosine similarity (or dot product) between the two. The dominant multilingual model in 2026 is BAAI/bge-m3: 568M params, 1024 dimensions, support for up to 8,192 context tokens per chunk, multivector (dense + sparse + ColBERT-style multi-vec in the same embedder), trained on 100+ languages. Alternatives:

Embedder	Params	Dim	Multilingual	Comment
`BAAI/bge-m3`	568M	1024	yes	the default all-rounder in 2026
`intfloat/multilingual-e5-large-instruct`	560M	1024	yes	direct competitor, instruct-style queries
`nomic-ai/nomic-embed-text-v1.5`	137M	768	English	fast, variable Matryoshka dimension
`jinaai/jina-embeddings-v3`	570M	1024	yes	with task-specific LoRAs per domain
`mixedbread-ai/mxbai-embed-large-v1`	335M	1024	English	top English on MTEB
`Snowflake/arctic-embed-l-v2.0`	568M	1024	yes	enterprise-oriented, Apache licence

The practical selection criterion: multilingual no matter what if your corpus or queries need it (bge-m3, multilingual-e5, jina-v3, arctic-l-v2), Matryoshka if you want to save vector store memory by reducing dimensions without re-embedding (nomic), and the MTEB leaderboard ranking only as a tie-breaker, never as the sole criterion, since MTEB benchmarks are contaminable and a 1-2 point difference rarely translates into a real improvement in your domain.

The fusion: Reciprocal Rank Fusion (RRF)

You have two rankings, one from BM25 and one from dense. How do you combine them? The naive option, adding the scores, fails because the scores are not comparable (BM25 produces numbers of 0-30, cosine 0-1, L2 distances 0-2…). The option that has consolidated is Reciprocal Rank Fusion (Cormack, Clarke, Buettcher 2009), which ignores absolute scores and uses only the rankings:

$$ \mathrm{RRF}(d) = \sum_{r \in R} \frac{1}{k + \mathrm{rank}_r(d)} $$

Where $R$ is the set of retrievers and $\mathrm{rank}_r(d)$ is the position of document $d$ in the ranking of retriever $r$. The constant $k$ is typically $60$ and smooths the weight of the first positions.

A numerical example. You have BM25 with top-5: [d_a, d_b, d_c, d_d, d_e] and dense with top-5: [d_b, d_f, d_a, d_g, d_h]. You compute the RRF of each candidate:

Doc	BM25 rank	dense rank	$\frac{1}{60+r_{\text{BM25}}}$	$\frac{1}{60+r_{\text{dense}}}$	total RRF
`d_b`	2	1	0.01613	0.01639	0.03252
`d_a`	1	3	0.01639	0.01587	0.03226
`d_c`	3	—	0.01587	0	0.01587
`d_f`	—	2	0	0.01613	0.01613
`d_d`	4	—	0.01562	0	0.01562

d_b wins because it appears well ranked in both. d_a follows it. Those that appear in only one list stay below. With nothing tuned, RRF tends to push the consensus candidates to the top, which is exactly what you want as a coarse filter before the reranker.

Variants: weighted RRF assigns a multiplier per retriever ($w_r \cdot \frac{1}{k + \mathrm{rank}_r(d)}$) when you have reasons to trust one more; learning-to-rank trains a model (LambdaMART or a LightGBM ranker) over features from the two retrievers, a more powerful option but one that demands relevance labels most teams do not have.

In Qdrant, Vespa, Weaviate and Elastic 8.11+ the RRF fusion is built in as a native mode: you ask for hybrid search with a lexical query and a vector and they return the combined top-k without you having to compute the fusion.

Layer 2 — The reranker: bi-encoder vs cross-encoder

The bi-encoder looks at query and document separately and compares the resulting vectors. It is extremely fast (a dot product) and lets you index the document embeddings once and for all. The price is that the model never sees query and document together, and the compressed representation loses nuance.

The cross-encoder looks at query and document concatenated: [CLS] query [SEP] document [SEP] goes through the model and the output is a relevance score. It discriminates far better, since shades of ordering and of local relationship between terms are captured, but the cost explodes: each (query, doc) pair requires a full inference of the model. You cannot precompute anything. If you want to rerank 50 candidates, that is 50 inferences.

Aspect	Bi-encoder	Cross-encoder
Inference per chunk at indexing	yes, once	not precomputable
Inference at query time	1 (query) + dot product per chunk	1 per (query, chunk) pair
Typical latency for top-50	5-20 ms (CPU possible)	80-300 ms (GPU recommended)
Discrimination	medium	high
Typical use	recall (layer 1)	precision (layer 2)

This asymmetry is why the canonical pattern is a two-step funnel: the bi-encoder does cheap broad recall and the cross-encoder does expensive fine precision, over 30-50 candidates instead of millions.

2026 OSS cross-encoder reranker stack

Reranker	Params	Multilingual	top-50 latency (A100)	Comment
`BAAI/bge-reranker-v2-m3`	568M	yes (100+ languages)	~90 ms	the de facto default in 2026
`BAAI/bge-reranker-v2-gemma`	2B	yes	~250 ms	more quality, more cost
`mixedbread-ai/mxbai-rerank-large-v2`	1.5B	English	~180 ms	top English on BEIR rerank
`jinaai/jina-reranker-v2-base-multilingual`	278M	yes	~50 ms	fast and enough for many cases
`Alibaba-NLP/gte-multilingual-reranker-base`	306M	yes	~55 ms	direct competitor to jina
`Cohere Rerank 3`	(closed)	yes (100+ languages)	~120 ms (API)	commercial, not on-prem
`Voyage rerank-2`	(closed)	yes	~100 ms (API)	commercial, not on-prem

For deployments under data sovereignty the obvious choice is bge-reranker-v2-m3 as a starting point, with jina-reranker-v2-base as the lightweight alternative. If the domain is exclusively English and the hardware budget allows it, mxbai-rerank-large-v2 takes a point or two more on BEIR. Cohere and Voyage are the option when you can reach out to an external API, ruled out in ENS / NIS2 scenarios with sensitive data.

Late interaction: ColBERT-v2 as a compromise

There is a third option gaining traction: late interaction, embodied in ColBERT-v2 (Santhanam et al., 2022). The idea is that the document is embedded not as a single vector but as a matrix of embeddings, one per token. And at query time the query is also embedded as a matrix, and the similarity is the sum of the maxima per column (MaxSim):

$$ S(q, d) = \sum_{i \in q} \max_{j \in d} \langle E_q[i], E_d[j] \rangle $$

The result is retrieval with quality close to a cross-encoder but at a much lower cost (there is no need to run the whole model over each pair). The price is space: each chunk is ~150-200 vectors instead of 1, and that multiplies the index size by ~100. For small to medium corpora (<10M chunks) it is manageable; for huge corpora PLAID compression and ColBERT-v2 quantization make it viable up to 100M.

In 2026 the reference implementations are RAGatouille (Python wrapper), JaColBERT (Japanese), Vespa.ai (a search engine with native ColBERT support as a first-class citizen), and the native integration in bge-m3 that produces the ColBERT-style multivectors as part of the same dense model.

My operational recommendation: start with a cross-encoder reranker over the top-30 of the hybrid retrieval. Only if the precision numbers fall short, try late interaction as an alternative to the cross-encoder (not as an additional layer). Late interaction as a third layer rarely pays back the operational cost for most cases.

Query rewriting and HyDE: correcting the query before retrieval

The user’s query is the worst-controlled input in the whole system. The most frequent problems:

Queries that are too short: “GDPR fines” → retrieves any chunk about GDPR or about fines.

Queries with typos or jargon: “i wnat to know how to cancel the premium subscripton” → the embedder is noisy because of the typos.

Multi-intent queries: “compare the premium and enterprise plans and tell me which has the better SLA” → dense will try to embed the three questions into one vector.

Queries with references resolved in context: “and for next month?” → without the previous context, dense does not know what it is about.

Canonical 2026 patterns to mitigate:

Query rewriting with a light LLM. Before retrieval, a small model (Qwen2.5-7B, Llama-3.2-3B, Phi-4-mini) rewrites the user’s query into a “canonical” version: it expands abbreviations, fixes typos, breaks multi-intent into sub-queries, resolves references using the conversation history. Cost: one cheap inference (~50-100 ms). Benefit: the four pathologies above are noticeably reduced.

Multi-query: the LLM generates 3-5 rephrasings of the query and you retrieve with each one; then you fuse with RRF. It increases recall at the cost of time and LLM tokens.

Step-back prompting (Zheng et al., 2023): the LLM generates a more general question than the original query (“what is a premium subscription in this system?”), you retrieve with both and combine. It helps with very specific queries that need context.

HyDE — Hypothetical Document Embeddings (Gao et al., 2022): instead of embedding the question, you ask a light LLM to generate a hypothetical answer and you embed that. The reasoning is that the semantic space of answers is closer to that of the real chunks than the space of questions. It works especially well in domains with very specific vocabulary. Cost: one inference of the light LLM (~100-200 ms). Benefit: in many cases +5-15 points of Recall@10.

The operational trade-off is the added latency. Each technique adds 50-300 ms to the pipeline. In synchronous interactions with a 2s SLO there is still headroom; in agentic interactions with tool-use loops, every ms counts. The practical choice in 2026: rewriting with a light LLM as the default, HyDE for specific cases where it has been measured to help, multi-query and step-back only when the case justifies it with numbers.

Minimum maths: latency budget and metrics

Latency budget of a canonical pipeline

A typical hybrid + rerank pipeline in 2026 production over a corpus of 5M chunks with bge-m3 + bge-reranker-v2-m3 on an H100 with TEI:

Stage	p50 latency	p95 latency	Comment
Query rewrite (LLM 3B)	60 ms	150 ms	only if enabled
HyDE (LLM 3B)	90 ms	220 ms	only if enabled
BM25 / SPLADE top-50	8 ms	20 ms	Elasticsearch or local Tantivy
Dense embedding of the query	12 ms	35 ms	bge-m3 with batch=1
Dense top-50 (filterable HNSW)	6 ms	18 ms	Qdrant with scalar quantization
RRF fusion → top-30	<1 ms	<1 ms	arithmetic
Cross-encoder rerank top-30	60 ms	180 ms	bge-reranker-v2-m3 batched
Top-5 selection + format	<1 ms	<1 ms
Total without rewrite/HyDE	~90 ms	~270 ms	realistic range on an H100
Total with rewrite + HyDE	~240 ms	~640 ms	+ ~150-370 ms

On an RTX 4090 (~3× slower than an H100 on a large cross-encoder, similar on everything else) the numbers shift: ~180 ms p50 without rewrite, ~400 ms p50 with rewrite + HyDE. It is perfectly serviceable for a synchronous assistant with a 2-3s SLO, but tight for one with a 500 ms SLO.

Metrics: nDCG, MRR, Recall@k

There are three classic metrics that any retrieval eval reports:

Recall@k: is the relevant chunk among the top-k? A binary metric, it ignores ordering. The one that matters for the broad layer.

$$ \mathrm{Recall@k} = \frac{|\{\text{relevant}\} \cap \{\text{top-k}\}|}{|\{\text{relevant}\}|} $$

Mean Reciprocal Rank (MRR@k): in which position does the first relevant chunk appear? It penalises placing it far down.

$$ \mathrm{MRR@k} = \frac{1}{|Q|} \sum_{q \in Q} \frac{1}{\mathrm{rank}_q} $$

where $\mathrm{rank}_q$ is the position of the first relevant item for query $q$ (or $\infty$ if it does not appear in the top-k).

nDCG@k: it takes into account multiple relevant chunks with degrees of relevance. The most used metric for evaluating rerankers:

$$ \mathrm{DCG@k} = \sum_{i=1}^{k} \frac{2^{\mathrm{rel}_i} - 1}{\log_2(i+1)} \quad \mathrm{nDCG@k} = \frac{\mathrm{DCG@k}}{\mathrm{IDCG@k}} $$

The operational heuristic: Recall@50 measures the quality of your broad layer (we want > 95%; otherwise layer 2 will never be able to recover what was lost), nDCG@5 measures the quality of your reranker (we want > 0.75 on the golden set to consider the system “good”) and MRR@5 measures the quality of your top-1 (it matters especially in systems with top_k=1 or where the LLM’s answer is based mainly on the first chunk).

Costs and throughput

Assuming a bge-reranker-v2-m3 cross-encoder (568M params) on an H100 SXM with TEI 1.7 and dynamic batching:

Throughput: ~280 queries/sec with top-30 each (= ~8,400 chunk-query pairs/sec).
VRAM used: ~3.5 GB for the model + ~6 GB of KV cache + activations at batch=32.
Energy cost: an H100 SXM draws ~700 W. At 8,400 pairs/sec, the energy cost of pure reranking is ~0.083 mJ per pair. In Spanish industrial electricity bill terms (~0.12 €/kWh, May 2026): ~0.000003 € per rerank of 30 candidates. The bill is negligible compared with the cost of the downstream LLM.

On an RTX 4090 with TEI: ~95 queries/sec at top-30 (~2,850 pairs/sec), VRAM ~3.5 GB + ~5 GB. Serviceable for an internal assistant with modest traffic (~5 QPS sustained, peaks at 20-30).

The canonical pattern — which piece goes where

The 2026 operational reference stack for an on-premise hybrid + rerank RAG under data sovereignty:

 ┌─────────────────┐
User query ────────▶│ Query rewriter │ (optional, LLM 3B)
│ (Qwen2.5-7B-IT) │
└────────┬────────┘
▼
┌─────────────────┐
│ HyDE │ (optional)
│ (same LLM) │
└────────┬────────┘
┌─────────────────┼─────────────────┐
▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ BM25/SPLADE │ │ Dense bi-enc │ │ (metadata │
│ Elasticsearch│ │ bge-m3 + TEI │ │ filters │
│ top-50 │ │ + Qdrant │ │ per tenant) │
│ │ │ top-50 │ │ │
└──────┬───────┘ └──────┬───────┘ └──────┬───────┘
└─────────┬───────┘ │
▼ │
┌──────────────┐ │
│ RRF k=60 │◀─────────────────┘
│ → top-30 │
└──────┬───────┘
▼
┌──────────────────┐
│ Cross-encoder │
│ bge-reranker-v2 │
│ TEI batch=32 │
│ → final top-5 │
└──────┬───────────┘
▼
┌──────────────────┐
│ LLM context │
│ ~2,000 tokens │
└──────────────────┘

Key services:

Sparse index: Elasticsearch 8.16, OpenSearch 2.18 or embedded Tantivy. Native BM25, SPLADE optional via plugin.
Vector index: Qdrant 1.13 (filterable HNSW, scalar quantization, multivectors). Covered in detail in the PostgreSQL + Qdrant in ingestion post.
Embedder serving: HuggingFace’s TEI (Text Embeddings Inference) for bge-m3, multilingual-e5 and compatible models. OpenAI-compatible endpoint.
Reranker serving: TEI too, for the rerankers of the bge-reranker, jina-reranker and mxbai-rerank families. Separate /rerank endpoint.
LLM for rewriter/HyDE: vLLM 0.7+ serving Qwen2.5-7B-Instruct or Phi-4-mini-Instruct. If the main LLM is the same, it shares resources; if not, a dedicated pod with GPTQ-INT4 quantization.
Orchestration: the RAG client (FastAPI, LangChain, LlamaIndex) composes the three lanes. Each parallel call (Sparse + Dense) is made concurrently; rewriter, HyDE and rerank are sequential.

Minimum Kubernetes manifests are already developed in the vLLM on Kubernetes post and the full orchestration piece fits the pattern of the multi-tenant H100 cluster.

On-premise hardware: what fits in an RTX 4090 vs a generic 4×H100 SXM configuration

The operational reality matters. Two typical profiles:

Modest configuration — 1× RTX 4090 (24 GB Ada Lovelace)

It comfortably serves:

bge-m3 (568M) for dense embedding of the query → ~1 GB VRAM
bge-reranker-v2-m3 (568M) cross-encoder → ~3.5 GB VRAM at batch=32
A light LLM for rewriter/HyDE (Qwen2.5-7B GPTQ-INT4) → ~5-6 GB VRAM
The main LLM (Llama-3.1-8B-Instruct FP8) → ~11-12 GB VRAM

Total VRAM used: ~21 GB. That leaves ~3 GB for the main LLM’s KV cache, enough for 4-8 concurrent users with ~4K token contexts. For an internal corporate assistant it is viable. If you need a larger context or more concurrency, you have to split things up: either the main LLM is served outside the 4090 (another GPU, another node), or you fall back on quantising the reranker (not advisable: you lose the nDCG points that justify the cost of adding it).

Generic configuration — 4×H100 SXM (320 GB total, NVLink)

It comfortably serves everything above multiplied by 30-50× in throughput and opens the door to:

A large main LLM (Llama 3.3 70B FP8 or Qwen2.5-72B FP8) on 2 GPUs with tensor parallelism
Large rerankers (bge-reranker-v2-gemma 2B) with quality ~2 points higher
Late interaction (ColBERT-v2) on one of the nodes for critical cases
Several light LLMs in parallel for rewriter, HyDE and the evals judge

The practical rule: the reranker is one of the most efficient things you can put on a GPU. The VRAM used is modest, the throughput is high, and the quality improvement per euro of extra hardware is among the best in the RAG stack.

The seven traps that kill retrieval

Trap 1 — Top-k to the LLM with no reranker. Dense retrieval returns a top-5 and those 5 chunks go straight to the LLM. Without a reranker, the order of the 5 is the cosine similarity order, which is not the real relevance order. The LLM answers based much more on the first chunks than the last ones, and the nuance is lost. Symptom: confident answers with citations that are not the most relevant in the corpus.

Trap 2 — Dense-only with no BM25. Short queries and queries with internal jargon work badly, the demos look pretty, real traffic complains. Symptom: on QA with real support tickets, the system fails specifically on queries of fewer than 5 words or with product codes.

Trap 3 — Reranker badly calibrated on top-k. A layer 1 top-k that is too small (top-10) cuts before the reranker can do its job. A top-k that is too large (top-200) blows up latency without improving nDCG. The canonical sweet spot is top-30 to top-50 from the hybrid → top-5 to top-10 from the reranker. Symptom: the reranker’s nDCG@5 stuck no matter which reranker model you try.

Trap 4 — Embedder and reranker in different languages. The corpus is Spanish + Catalan, the embedder is all-MiniLM (English only), the reranker is bge-reranker-v2-m3 (multilingual). The broad layer retrieves badly because of the embedder, and the reranker has little to rerank. Symptom: Recall@50 < 80%, unrecoverable by changing the reranker.

Trap 5 — RRF with k=1. Someone reads the paper, copies the formula and sets k=1 “because they do not understand what it does”. The result: the first positions of the worst ranking (whichever it is) dominate. The fusion stops averaging and becomes “winner-takes-all”. Symptom: the final ranking is almost identical to that of just one of the retrievers.

Trap 6 — A query rewriter that changes the meaning. The LLM rewriter “improves” "how do I cancel" into "termination procedure for the premium service for corporate customers". It retrieves chunks about corporate customers when the user was a private individual. Over-specification by the rewriter is worse than no rewriter. Symptom: queries from “normal” users come back worse than the old ones. Mitigation: the rewriter must keep the user’s intent and only add context, never assume.

Trap 7 — No retrieval telemetry. The system serves, the user complains, you do not know whether the fault was the corpus, the retrieval, the reranker or the LLM. Without emitting traces with retrieved_chunks_ids, retrieved_chunks_scores, rerank_scores, query_rewritten_to, selected_top_k, debugging is theatre. Symptom: each incident takes days to investigate. The tracing with OTel and MCP piece covers the canonical pattern.

All seven are operational. Just as with the corpus, retrieval does not break because the maths is wrong: it breaks because the discipline relaxes. And as in Eval, the metrics can go up while the real experience gets worse, because the golden set accommodates itself to the system instead of the system accommodating itself to the golden set.

What we have not covered (upcoming posts)

Embedding model selection and fine-tuning: how to choose among the embedders on the MTEB leaderboard without falling into goodharting, when and how to fine-tune an embedder on your own domain with MNR loss or triplet loss, which synthetic dataset is generated with an LLM for training (GPL, InPars, Promptagator), and the gotchas of massive re-embedding when you change model.
Semantic cache for RAG: how to cache semantically similar queries to serve answers without going through retrieval or the LLM, GPTCache, MeanCache, and the cache’s precision/coverage trade-off. It saves 30-70% of the cost on workloads with repeated queries.
Multi-vector and ColBERT-v2 at scale: how PLAID and CITADEL indexes are designed to serve corpora of 100M+ chunks with late interaction without burning the memory budget.
RAG-specific eval — RAGAS deep dive: faithfulness, answer relevance, context precision, context recall, noise sensitivity. How the RAG-specific golden set is built (with chunks labelled as relevant/irrelevant) and which metrics correlate with real user satisfaction.
Function calling and tool-augmented retrieval: when the LLM decides which retriever to invoke (SQL for structured data, vector for unstructured, web search for real time), the ReAct pattern, handling tool errors.
Agentic retrieval loops: when a single retrieval pass is not enough and the agent iterates (planning, sub-queries, summary-and-refine). The latency / quality trade-off and the anti-patterns (infinite loops, diverging sub-queries).

References

BEIR: Thakur et al. (2021). “BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models.” NeurIPS 2021 Datasets and Benchmarks Track. https://github.com/beir-cellar/beir
SPLADE: Formal et al. (2021). “SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking.” SIGIR 2021. https://arxiv.org/abs/2107.05720
ColBERT-v2: Santhanam et al. (2022). “ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction.” NAACL 2022. https://arxiv.org/abs/2112.01488
Reciprocal Rank Fusion: Cormack, Clarke, Buettcher (2009). “Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods.” SIGIR 2009. https://dl.acm.org/doi/10.1145/1571941.1572114
HyDE: Gao et al. (2022). “Precise Zero-Shot Dense Retrieval without Relevance Labels.” https://arxiv.org/abs/2212.10496
Step-back prompting: Zheng et al. (2023). “Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models.” https://arxiv.org/abs/2310.06117
BGE-M3: Chen et al. (2024). “BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation.” https://arxiv.org/abs/2402.03216
MTEB leaderboard: https://huggingface.co/spaces/mteb/leaderboard — útil como punto de partida, peligroso como criterio único (overfitting al benchmark, dataset contamination).
TEI (Text Embeddings Inference): https://github.com/huggingface/text-embeddings-inference — el motor de serving de HuggingFace para embedders y rerankers de tamaño pequeño-mediano. Endpoint OpenAI-compatible.
Qdrant hybrid search: https://qdrant.tech/documentation/concepts/hybrid-queries/ — implementación nativa de RRF y multi-vector queries.
RAGatouille: https://github.com/AnswerDotAI/RAGatouille — wrapper Python para ColBERT-v2 que reduce drásticamente la curva de entrada.
Cohere Rerank 3: https://docs.cohere.com/docs/rerank-2 — referencia técnica del reranker comercial multilingüe líder.

RAG corpus curation: the active librarian who decides what goes in, what goes out and what gets signed

Mon, 25 May 2026 11:00:00 +0200

This is the curation layer inside stage 1 (Data) of the six-stage LLMOps pipeline. It complements the other Data posts: dataset versioning for the four versionable artefacts, PostgreSQL + Qdrant ingestion in microservices for the outbox + CDC pattern, and RAG over Kafka and a datalake for streaming transport. This post is not about moving data: it is about what to do with data before you let a model read it.

TL;DR

A RAG system serving mediocre answers is rarely the retriever’s fault, nor the model’s. The root cause is usually the corpus: three nearly identical versions of the same PDF that make top-k return the same thing three times, an old manual nobody removed that contradicts the current one, a free-text field with customer numbers that the model quotes verbatim, a scanned PDF with dirty OCR that the chunker cut in the middle of a sentence. None of that is fixed by changing the model, the embedder, the reranker or the prompt. It is fixed by curating the corpus. This post takes apart the five operational layers of curation (schema-validated ingest, three-level deduplication, PII anonymisation measured with precision/recall, anti-contamination against the golden eval set, chunk→trace lineage), the minimum maths needed to avoid fooling yourself, the 2026 stack (Presidio, Unstructured, Argilla, LangChain text splitters, OpenLineage, Marquez, Great Expectations), the seven traps that reduce the stage to theatre, and the on-premise hardware to sustain all of it without sending anything sensitive to external APIs.

The analogy: the active librarian

A serious librarian does not accept books by the kilo. When someone offers a new volume:

They look at the spine, the ISBN and the stamp: is it legible? Is it catalogued correctly? Does it belong to a recognised collection? Without valid metadata, it does not get in. This is the schema check.
They check whether they already hold a copy: is it exactly the same book? Is it a later edition of the same one? Is it a translated version of something already on the shelves? If so, they decide explicitly what to do (replace, archive the old one, withdraw both from lending). This is dedup at its three levels.
They flag what is restricted: if the book contains identifiable personal data, some pages cannot be lent as they are, so they must be redacted, anonymised or moved to the reserved section. This is PII anonymisation.
They verify it is not this year’s final exam book: if it is, it stays out of the public collection until the syllabus changes, because if students can consult it the exam stops measuring what it is meant to measure. This is anti-contamination against the golden eval set.
They write it into the register: this book, this edition, this provenance, this date, this person who approved the entry. This is lineage.

If the book passes all five, it joins the collection. If it fails any of them, it goes to an auditable quarantine shelf with the reason for rejection. The difference between a good collection and a mediocre one is not size: it is how much discipline you apply across the five layers, every day, to every new book that arrives.

A RAG corpus is exactly that. The only differences are scale (thousands or millions of documents per month) and the fact that the “readers” are LLMs that cannot tell a duplicate from a reinforced truth, nor a PII value from a synthetic example, nor a contaminated fragment from an authentic one.

The four data artefacts and where the RAG corpus fits

Before going down to the five layers it is worth being clear about which corpus we are curating. The Data stage of the pipeline manages four distinct artefacts, each with its own discipline. The data versioning post lists them; here I reorder them from a curation perspective:

Artefact	Who consumes it	Dominant curation
Training dataset	Tune (fine-tuning the model or an adapter)	aggressive dedup + quality filters + label balancing
RAG corpus	Deploy (retrieval at request time)	the 5 layers in this post
Golden eval set	Eval (promotion gates)	strict hold-out + stratification + maintenance driven by incidents
Retrain enriched dataset	Retrain (closing the loop)	production feedback + human triage

The RAG corpus is the most volatile of the four and the one most exposed to the end user: every answer the system serves literally contains fragments of it. A duplicate in the training dataset degrades learning but ends up buried in the weights; a duplicate in the RAG corpus shows up in today’s answer and tomorrow’s. That justifies the extra discipline that follows.

Layer 1 — Schema-validated ingest

Every piece entering the corpus has to arrive with structured metadata validated against a schema. This is not bureaucracy: it is the only way to make the following layers (dedup, PII, lineage) work without friction.

The canonical pattern is to define a schema in JSON Schema or Pydantic that every document must satisfy:

class CorpusDocument(BaseModel):
 source_system: str # e.g. "confluence", "salesforce", "manual_pdf"
 source_id: str # unique ID in the source system
 version: str # document version (semver or date)
 language: str # ISO 639-1: "es", "en"
 title: str
 body: str
 captured_at: datetime
 captured_by: str # pipeline or human
 sensitivity: Literal["public", "internal", "restricted"]
 schema_version: str # version of the schema itself, not of the document

Any piece that fails this contract is rejected at ingest and never reaches the following layers. Validation is done with Great Expectations (declarative suites), Pandera (more pythonic, integrates with pandas) or Soda (oriented towards continuous data quality). The choice is a matter of style; what matters is:

Validation suites live in code and are versioned with the pipeline, not in a notebook off to one side.
Rejection generates an auditable event (quarantine) with the specific schema failure as its reason, not a log line lost in stdout.
The schema itself is versioned. When it changes, previous documents are reprocessed or explicit backward compatibility is maintained.

The RAG over Kafka post covers the Schema Registry pattern (Confluent Schema Registry or Apicurio) that materialises this in streaming: every message on the topic is validated against the registered schema before propagating downstream. For batch or pull, Great Expectations is the equivalent.

Common trap: leaving the body field free with no further validation. You have to tighten it up with minimum/maximum length (a PDF that yields 12 characters after extraction is almost certainly broken), valid encoding (UTF-8 with no control characters), proportion of alphanumeric characters (dirty OCR returns a soup of symbols). These are simple rules that filter 80% of the noise with no need for AI.

Layer 2 — Three-level deduplication

The most expensive and most silent mistake in a RAG corpus is the duplicate. A document that appears three times in the corpus makes retrieval’s top-k return it three times, wasting two slots and reinforcing a single source. The LLM reads it as if three independent sources agreed, when in fact it is the same thing repeated.

Deduplication is done at three levels, in this order by cost:

Level A — Exact dedup (SHA-256 hash)

Compute the hash of the normalised content (trim, lower-case where applicable, remove redundant whitespace) and compare it against an index of already ingested hashes. On a match, discard or replace. Cost: $O(1)$ per document. It catches literal duplicates (the same PDF uploaded twice, two byte-for-byte copies of the same HTML).

Level B — Near-duplicate (MinHash + LSH)

Nearly identical documents with minor differences (a different header, an updated date, one version in Castilian and another in Galician with minimal changes). The canonical algorithm is MinHash with Locality-Sensitive Hashing (LSH), which approximates Jaccard similarity over shingles of k tokens. For n documents, comparing all against all is $O(n^2)$, which is unworkable for large corpora. LSH cuts the cost to $O(n)$ most-likely buckets.

A typical threshold is Jaccard ≥ 0.80 over shingles of 5 tokens. The standard libraries are datasketch (Python, MIT) or dedup (Python, MIT). A numerical example: for 1 M short documents (300 tokens each), datasketch.MinHashLSH with 128 permutations and threshold 0.8 takes about 2 GB of RAM and processes the full corpus in around 30 minutes on a modern CPU. The fraction of duplicates found in a real enterprise corpus is usually between 5% and 25%; removing them cuts storage and improves retrieval quality at the same time.

Level C — Semantic dedup (cosine over embeddings)

Documents saying the same thing in different words, such as paraphrases, translations or rewritten versions, are not caught by MinHash. This is where semantic similarity comes in: compute the embedding of each document and compare the cosine between pairs.

The problem is quadratic cost: for n documents, computing every pair is $O(n^2)$. For n = 1 M and 768-dimensional embeddings (a typical model such as BAAI/bge-base-en-v1.5), that is 5×10^11 dot products, which is unworkable. The solution is the same idea as LSH but over dense vectors: HNSW (Hierarchical Navigable Small World) or IVF (Inverted File) to build an approximate search index. For each new document you run a k-NN query against the index and examine only the k nearest neighbours.

A sensible threshold for treating something as a semantic duplicate: cosine ≥ 0.95. Below 0.95 the documents are related but distinct; above it, they are almost always the same information rewritten. The exact threshold is calibrated by observing precision/recall over a human-annotated sample; 100 pairs confirmed by a reviewer is reasonable for setting it.

A numerical example: with qdrant or pgvector as the HNSW index and k=10 neighbours per query, deduplicating 1 M documents against the existing corpus takes on the order of 2-4 hours on an RTX 4090 (including embedding computation). If the embedder is self-hosted with vLLM, the cost per token is negligible against compute time.

Policy on what to do with a duplicate

Detection is not enough, you have to decide. Three common policies, in order of complexity:

Drop: discard the more recent one, keep the older one. Simple, with no extra lineage.
Replace: discard the old one, index the new one. More volatility but it reflects the update.
Merge with provenance: mark the new one as a “shadow” of the old one, keep both in lineage but index only one. Better for regulated audit.

The policy has to be explicit and applied uniformly, not an ad-hoc decision per document.

Layer 3 — PII anonymisation with measured precision/recall

This layer is the one that most easily turns into theatre. The typical mistake: install Presidio, run it over the corpus, assume the output is clean. Without measuring the detector’s precision and recall against an annotated golden set, you know nothing.

A PII detector can fail in two ways:

False negative (low recall): it does not detect a national ID written as “12345678-A” because your model was trained on the 12345678A format without a hyphen. The RAG serves personal data unredacted.
False positive (low precision): it redacts a number from a configuration manual thinking it is a phone number. The RAG loses useful information.

Both are problems; regulation (GDPR, ENS, NIS2) penalises the first, and user experience degrades with the second. The acceptable ratio depends on the domain. In medical data, practically zero false negatives is non-negotiable; in internal technical documentation you can tolerate more recall in exchange for less precision.

The standard metric is F1 over an annotated golden set:

\[ \text{precision} = \frac{TP}{TP + FP}, \quad \text{recall} = \frac{TP}{TP + FN}, \quad F_1 = 2 \cdot \frac{\text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}} \]

To build the PII golden set, annotate around 200 documents by hand with every entity marked (national ID, IBAN, email, phone, address, personal name). Then run the detector and compute the metrics per category, not just aggregated, because a global F1 of 0.90 can hide a recall of 0.55 on IBANs.

2026 stack for this layer:

Microsoft Presidio (MIT, Microsoft): the most complete OSS option. Configurable detectors, recognises around 50 entities by default, extensible with your own regex patterns or with fine-tuned NER models.
spaCy NER (MIT, Explosion AI): a base for custom detectors; useful when Presidio does not cover a domain entity.
Llama Guard 4 (Llama Community License, Meta): a safety classifier that also detects PII in one pass, an option when you already have a GPU for inference and prefer a single pass.
DataFog (Apache 2.0): a more recent alternative, specialised in streaming pipelines.

Recommended hybrid pattern: Presidio for rule-based + regex detection (fast, deterministic) → Llama Guard as a second pass over what Presidio did not flag (an ensemble that raises recall without killing throughput). This is measured and reported as aggregate and per-category F1 on every release of the detector.

Common fallacy: trusting that a detector with F1 0.95 “is very good”. If you have 1 M documents and each contains on average 1 PII entity, F1 0.95 means 50,000 mishandled entities (between false positives and negatives). With sensitive data you have to design so that false negatives go to human quarantine, not to the public corpus.

Layer 4 — Anti-contamination against the golden eval set

If the RAG corpus contains fragments of the golden eval set, Eval metrics measure memorisation. The model returns the exact answer because it has it literally in its context, not because it generalised anything. Deploy then promotes models that shine in the exam and fail in production.

This layer is the easiest to implement and the easiest to forget:

The golden eval set has its own versioned hash.
Before indexing any new document into the RAG corpus, run a token-by-token overlap check (or by shingles, similar to MinHash) against the golden set.
If overlap exceeds a threshold (typically ≥ 30% of 5-token n-grams), the document is not indexed. It stays in quarantine flagged as “contamination risk vs golden_v12”.
A human reviews the rejections. Sometimes they are false positives (a short quotation, a boilerplate sentence). Sometimes they are real contamination that a supplier introduced without realising.

The deeper reason: the RAG corpus and the golden set are enemies by design. The golden set measures how well the system generalises to questions it has not seen. If those questions are in the RAG, the system “sees” them on every query. The metric stops measuring generalisation.

This check is computationally trivial, a hash join over n-grams. The difficulty is keeping it up: every time the golden set changes (monthly or quarterly), the full corpus has to be re-validated against the new golden set. Without that discipline, contamination sneaks in through the back door when someone updates the golden set with real cases that the RAG was already serving.

Layer 5 — End-to-end lineage: from document to trace

The last layer is the one that closes the auditable chain. Every chunk indexed in the vector store carries metadata that lets you answer the forensic question:

“The system generated this answer on 14 March at 16:23. Which exact document did the quoted fragment come from? When did that document enter the corpus? Which version of the embedder processed it? Who approved its ingest?”

Without lineage, that question is unanswerable. With lineage done properly, it is four queries.

The canonical pattern:

Every indexed chunk carries in its metadata: source_system, source_id, document_version, chunk_index, embedder_version, dataset_hash, ingested_at, ingested_by, schema_version.
Every RAG answer in production emits a trace span that includes the retrieved chunk_id values.
The central tracing system (Langfuse, Phoenix or OpenLLMetry) joins chunk_id → chunk metadata → document metadata → corpus dataset_hash → embedder version → and so on.

The tools that standardise this glue are OpenLineage (Apache 2.0, LF AI & Data) and Marquez (Apache 2.0, its server implementation). They define a lineage event schema that interoperates between systems; an ingest job emits an event “produced corpus_v12.3 from source X with embedder bge-base-v1.5”; a retrieval job emits “consumed corpus_v12.3 with query Q produced response R”. The graph is reconstructed automatically.

This layer is the only way to pass real audits under regulations such as the EU AI Act, GDPR or ENS, where traceability of which data went into which answer is a requirement, not an option. Without it, the answer “we do not know which document this came from” is not acceptable, and it is the default answer if lineage is not built from day one.

The maths that matter

Beyond dedup thresholds and PII F1 scores, there are three mathematical pieces that any serious team ends up using.

Chunk size vs retrieval quality. Chunk size affects retrieval quality in a non-monotonic way: chunks that are too small fragment ideas (retrieval returns a piece without context), and chunks that are too large dilute the signal (the embedding mixes several topics and similarity drops). The empirical sweet spot for technical text in 2026 is between 256 and 768 tokens per chunk, with 15-25% overlap between contiguous chunks to preserve continuity.

Numerically, for a corpus of 1 M documents with a mean length of 2,000 tokens, chunking at 512 tokens with overlap 100 gives $\frac{2000}{512 - 100} \approx 5$ chunks per document, a total of roughly 5 M indexed chunks. With 768-dimensional embeddings in float32, that occupies $5 \cdot 10^6 \cdot 768 \cdot 4 \approx 15$ GB of vector memory, manageable in any modern vector store.

PII golden set coverage. To know whether the annotated PII golden set is representative enough, compute the proportion of categories covered: if your golden set of 200 documents has 5 IBAN examples and production has 12,000 IBANs per day, the measured F1 on IBANs is statistical noise. Rule of thumb: a minimum of 30 examples per category for per-category metrics to mean anything.

Cost of re-embedding when rotating the model. Changing the embedder invalidates the whole index. For a corpus of 5 M chunks with a model such as BAAI/bge-base-en-v1.5 (768 dim, around 110 M parameters) served in vLLM on 1× H100, throughput is on the order of 8,000-15,000 chunks/second. Full re-embedding: about 5-10 minutes. For a larger embedder (bge-large, 1024 dim, around 335 M parameters), a factor of 3× worse, around 15-30 minutes. The bottleneck is usually vector store I/O, not GPU compute. The dual-index pattern, keeping the old index serving while the new one is built and doing an atomic swap at the end, avoids downtime and allows rollback.

Applied to typical on-premise hardware

For an on-premise deployment that keeps all curation in-house without sending data to external APIs:

RTX 4090 (24 GB): covers layer 1 (the schema check with Great Expectations is CPU-bound), layer 2 levels A and B (hash + MinHash are CPU-bound), and layer 2 level C semantic dedup with a bge-base-class embedder (8-15k chunks/s, enough for corpora of up to 5-10 M chunks in hours). For Presidio in NER mode (layer 3) it runs comfortably. It is the sensible GPU for the whole curation pipeline on a mid-size corpus.
Generic 4×H100 SXM configuration (320 GB total, NVLink): only needed if the corpus exceeds around 50 M chunks or if you want frequent re-embeddings with large models (bge-large, e5-mistral). In practice, two GPUs serve the embedder at TP=2 with throughput above 50k chunks/s, and the other two go to the PII judge (Llama Guard 4) or to serving the main inference model. Capacity for corpora of hundreds of millions of chunks.

The stubborn arithmetic: with a 4090, corpus curation is an overnight job; with 4×H100, it is minutes. The decision depends on corpus size and on how often you rotate the embedder or the PII rules.

The seven traps that kill this stage

Trap 1 — No validated schema at ingest. Malformed documents reach the chunker, the chunker slices them into nonsense, garbage embeddings enter the index. The RAG answer quotes incoherent text and nobody knows why.

Trap 2 — Dedup only at exact hash level. The corpus fills up with paraphrases and translations of the same document. Retrieval’s top-k returns the same source 3 times. The LLM reads it as three confirmations.

Trap 3 — PII detector without precision/recall measurement. Everyone assumes Presidio “works”. IBANs in non-standard formats slip through. The RAG serves personal data.

Trap 4 — Golden eval set contaminated with the corpus. Eval metrics measure memorisation. Promotions approve models that fail in real production.

Trap 5 — No lineage down to the chunk. The question “where did this quotation come from?” has no answer. Regulatory audit fails. Incidents cannot be investigated.

Trap 6 — Maintenance as a one-off event. The corpus is curated once when the system is initialised, and afterwards everyone assumes it is fine. Six months later the documents are out of date, the new PII rules are not applied retrospectively, dedup has not been re-run after adding new sources. The corpus degrades silently.

Trap 7 — Quarantine without human review. Rejected documents go into a table nobody looks at. False positives pile up, real contamination cases go uninvestigated, the team’s trust in curation erodes and the pressure to “relax the thresholds” begins.

All seven are operational, not technical. Corpus curation does not break because of an algorithm bug: it breaks because discipline slackens. It is the exact equivalent of the kind of degradation that kills Eval suites, and in both cases the symptom is the same: the metrics improve or hold steady while the real experience gets worse.

What we have not covered (upcoming posts)

Vector store versioning proper: an embedding index is not versioned like a raw dataset because it depends on the embedding model. Changing the embedder rewrites the whole index. It is a different animal with its own patterns (index branching, selective reembedding, recall-aware ANN parameters).
Streaming corpus updates with CDC: when the corpus has to be updated in near-real time from an OLTP system. The Postgres + Qdrant ingestion post covers the mechanics; still pending is the pattern for selectively invalidating chunks that depend on deleted rows.
Multi-tenant corpus isolation: how to set up a shared corpus versus one with per-tenant namespaces, with ACLs over individual chunks. Especially relevant for multi-customer RAG under data sovereignty constraints.
Federated corpus: corpora distributed across silos that the system queries without centralising the content. An emerging pattern for companies with several sites and cross-border restrictions.
Reranking-aware curation: how curation discipline changes when there is a reranker (Cohere Rerank, ColBERTv2, BGE-Reranker) reordering the top-k after retrieval. Some duplicates you would tolerate without a reranker become intolerable when the reranker pushes them up the ranking.

References

Presidio: https://microsoft.github.io/presidio/ — official docs, list of supported entities, guide to extending it with custom NER.
OpenLineage: https://openlineage.io/ — event schema spec and libraries per language.
Marquez: https://marquezproject.ai/ — server implementation of OpenLineage.
datasketch (MinHash + LSH): https://ekzhu.com/datasketch/ — the reference Python library for near-duplicate deduplication at scale.
Great Expectations: https://docs.greatexpectations.io/ — declarative data quality suites.
Unstructured: https://docs.unstructured.io/ — parsing and normalisation of heterogeneous documents (PDF, HTML, DOCX, eml) before chunking.
Argilla: https://docs.argilla.io/ — human annotation UI for building the PII golden set and other calibration sets.
Llama Guard 4: Meta technical paper, multimodal safety classifier — useful as a second PII detection layer.
GDPR, EU AI Act, ENS, NIS2 — the regulatory frameworks whose compliance depends, in practice, on the discipline of layers 3 (PII) and 5 (lineage). The CEN/CENELEC technical standards for conformity assessment of GenAI systems under the EU AI Act are still pending final publication.

RAG on Kafka: a technical reference architecture for streaming datalakes, with fresh embeddings and vector stores that are always up to date

Thu, 21 May 2026 06:00:00 +0200

TL;DR

The piece that blocks enterprise GenAI projects most in 2026 is not the model, nor even the guardrails: it is data ingestion for RAG. Companies hold valuable information in OLTP databases, in operational logs, in SaaS systems, and all of it is quietly changing every second. Batch RAG systems that reindex every night arrive late (the model’s answer is backed by a snapshot from 18 hours ago) and open the door to operational hallucinations even when the retriever is perfect. The dominant answer in production in 2026 is to build the RAG piece on top of Kafka as the source of truth: an immutable log, massive throughput, managed schema evolution, and a mature stream processing ecosystem (Flink, Kafka Streams, RisingWave) that makes it possible to transform and embed events as they happen, delivering them to vector stores (Milvus, Qdrant, Weaviate, pgvector) in milliseconds. The canonical pattern: source → CDC with Debezium → Kafka topics → Flink SQL with an embedding UDF → sink connector to a vector store → serving with vLLM or equivalent. The 2026 developments that change the game: Confluent Tableflow turns Kafka topics into Iceberg/Delta tables automatically (read from Snowflake/Databricks/Trino with no ETL, 30-50 % lower TCO); native Flink SQL brings openai_embedding() and vector search integrated with Cosmos DB and Amazon S3 Vectors; Confluent’s official MCP server lets AI agents query Kafka/Flink/Tableflow in natural language. This post develops the end-to-end architecture with manifests, Flink SQL code and concrete numbers.

This is the second post in the MLOps series specific to LLMs. The first (2026 landscape) set out the framework. Here we go down to the most operational piece of the stack: how a real enterprise system connects to an LLM agent while keeping RAG fresh without falling into explosive complexity.

The analogy: Kafka as the “single source of truth”

Anyone who has spent time in distributed systems has seen the pattern again and again: an immutable, append-only, replicated, time-ordered log has become the canonical primitive for rebuilding complex systems. DBAs know it as the write-ahead log (PostgreSQL WAL, MySQL binlog). Event system developers know it as event sourcing. Data architects know it as the Kappa architecture. Kafka is the massive, distributed, mature implementation of that primitive: a log that lives on disk, partitioned to scale, replicated for durability, retained by time or size, readable from any point in history.

When you think about RAG, this is exactly what you need. A well-designed RAG system has two critical questions: how do you keep the index fresh? And how do you rebuild the index when something breaks? Kafka answers both naturally: fresh because every change at the source is published as an event to the log and the pipeline processes it in milliseconds; rebuildable because the whole log is there: you drop the vector store, you have the Kafka topic from offset 0, and you build the index back exactly as it was.

There is a second layer to the analogy as well. For a modern GenAI architecture, Kafka plays the role of the WAL of the entire system. Just as the Postgres WAL is the gospel of the database’s state (if you lose the DB but keep the WAL, you can rebuild it), the Kafka log is the gospel of the state of the business as a whole: orders, users, transactions, documents. Connecting your AI agent to Kafka means connecting it to the real pulse of the system, not to stale snapshots.

The static RAG problem

Before presenting the architecture, it is worth pinning down what concrete problem we are solving. The antipattern that trips up most GenAI projects:

A team builds RAG on a static dataset: it dumps Confluence documents, product PDFs, database snapshots.
It embeds the lot with a nightly cron that regenerates the index every 24 hours.
It ships the product.
Day 2: a user asks about a change that happened two hours ago. RAG does not have it; the model answers about the old version.
The team adds fragile logic: “if the query mentions a recent date, escalate to a human agent”.
Day 30: the dataset has moved so much that half the RAG is out of date. The team decides to refactor and migrate to streaming.

It is the repeated story of so many projects that the ecosystem has learned the lesson: streaming from day 1, even when the volume is low. The operational complexity of a well-designed streaming pipeline is constant; the complexity of migrating from batch to streaming on a live project is enormous.

From Lambda to Kappa to Streaming RAG

Three architectures in historical order:

Lambda (the classic big-data design from 2014): two parallel pipelines, one batch for accuracy and one streaming for freshness. The query combines both. It works, but it means maintaining two pipelines.

Kappa (Jay Kreps, 2014, mainstream since 2020): a single streaming pipeline. Batch is a special case of streaming (reprocessing from the beginning). It simplifies a great deal.

Streaming RAG (emerging in 2025-2026): a specific variant of Kappa where the pipeline’s output is embeddings indexed in a vector store that the LLM queries at runtime. The Kafka log is the source of truth, the vector store is a queryable projection.

The mental conversion: think of the vector store as the materialised view of the Kafka log. If the view is corrupted, you rebuild it from the log. If you want a new view (a different embedding model, a different chunking strategy), you create another consumer of the log and build a second view in parallel.

The reference architecture

On to the diagram. I am going to present the canonical architecture that stabilised in 2026, showing where each component fits:

[OLTP DB (Postgres)] [Other sources]
│ │
│ WAL via logical decoding │
▼ ▼
┌──────────────────────────────────────────────────────────┐
│ Debezium / Kafka Connect (Sources) │
└──────────────────────────────────────────────────────────┘
│
▼ produces events
┌──────────────────────────────────────────────────────────┐
│ Kafka cluster │
│ ┌───────────────────────────────────────────────────┐ │
│ │ topic: orders.raw (3 partitions, RF=3) │ │
│ │ topic: users.raw (3 partitions, RF=3) │ │
│ │ topic: documents.raw (6 partitions, RF=3) │ │
│ └───────────────────────────────────────────────────┘ │
│ + Schema Registry (Avro/Protobuf) │
└──────────────────────────────────────────────────────────┘
│
▼ consumes and transforms
┌──────────────────────────────────────────────────────────┐
│ Flink SQL streaming jobs │
│ - chunking text │
│ - calls to the embedding model (UDF) │
│ - enrichment with metadata │
│ - sink to a curated topic: documents.embedded │
└──────────────────────────────────────────────────────────┘
│
┌───────────┼────────────────────┐
▼ ▼ ▼
[Vector store] [Tableflow] [Iceberg/Delta]
Milvus/Qdrant auto-convert for analytics
/pgvector/ topics →
Weaviate tables
│
▼ queried at runtime
┌──────────────────────────────────────────────────────────┐
│ LLM serving (vLLM / SGLang) + Retriever │
│ - receives the agent's query │
│ - searches top-K in the vector store │
│ - builds prompt + context │
│ - generates an answer with citations │
└──────────────────────────────────────────────────────────┘

The five layers you can see (source, ingestion (CDC), transport (Kafka), processing (Flink), storage (vectors + tables)) are the ones that structure any serious RAG on a datalake in 2026. Let us go through each one.

Layer 1 — Sources: your OLTP as the starting point

The typical source is an OLTP database (Postgres, MySQL, SQL Server). It is where the live state of the business lives. The technique for extracting changes in real time is Change Data Capture (CDC): reading the database’s transaction log (PostgreSQL WAL, MySQL binlog) and turning each commit into a Kafka event.

The OSS standard is Debezium. It supports Postgres, MySQL, SQL Server, MongoDB, Oracle, Cassandra and others. The typical deployment is a Kafka Connect cluster with Debezium connectors.

An example Debezium configuration for PostgreSQL:

{
 "name": "postgres-orders-connector",
 "config": {
 "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
 "tasks.max": "1",
 "database.hostname": "postgres.prod.internal",
 "database.port": "5432",
 "database.user": "debezium",
 "database.password": "${secret:postgres-creds}",
 "database.dbname": "ecommerce",
 "database.server.name": "ecommerce-prod",
 "table.include.list": "public.orders,public.users,public.products",
 "publication.autocreate.mode": "filtered",
 "slot.name": "debezium_slot",
 "plugin.name": "pgoutput",
 "topic.prefix": "ecommerce",
 "key.converter": "io.confluent.connect.avro.AvroConverter",
 "value.converter": "io.confluent.connect.avro.AvroConverter",
 "key.converter.schema.registry.url": "http://schema-registry:8081",
 "value.converter.schema.registry.url": "http://schema-registry:8081"
 }
}

For each commit in the database, this produces an Avro event on the corresponding topic (ecommerce.public.orders, ecommerce.public.users, and so on) carrying the change: type (INSERT/UPDATE/DELETE), values before and after, commit timestamp, position in the WAL.

A simpler alternative for 2026: RisingWave can read the Postgres WAL directly, with no Debezium or Kafka Connect in between. When the case is CDC only, with no other sources, it is operationally simpler. For architectures with multiple sources (CDC + APIs + scrapers + logs), Debezium remains the standard piece.

Layer 2 — Kafka as transport and persistence

The Kafka cluster is where all events land. Key operational decisions:

Topics: raw vs curated

The convention that settled in 2026:

*.raw: the raw event exactly as it arrived. Untransformed CDC, unparsed application logs.
*.cleaned: after dedup, schema validation, type normalisation.
*.enriched: after adding metadata (geolocation, cross identifiers, and so on).
*.embedded: the event with its vector embedding already computed.

Multi-stage topics allow debugging per layer and partial reprocessing: if you change the embedding model, discarding *.embedded and rebuilding from *.enriched costs hours; rebuilding from *.raw costs days.

Schema Registry

Without a schema registry, topics break silently when someone changes the schema at the source. Confluent Schema Registry or the OSS Apicurio are the dominant options.

Common formats:

Avro: versioned schema, strict evolution rules. The historical default.
Protobuf: compatible with gRPC stacks, good performance.
JSON Schema: textual, debuggable by eye, less efficient.

For RAG on Kafka we recommend Avro by default. Schema evolution matters because source tables change over time, and an unversioned schema breaks downstream consumers.

Partitions, replication and retention

Operational decisions for RAG topics:

Partitions: typically 3-12. More partitions means more parallelism in the Flink consumer, but more overhead. The rule of thumb: partitions = expected peak events/s ÷ 1000.
Replication factor: 3 as a minimum in production. Replication protects against broker failure; with RAG, the cost of losing a topic can be weeks of re-embedding.
Retention: for topics that feed RAG, use long retention or compaction by key. If document doc-42 changes 100 times, compaction keeps only the last state per key, leaving a smaller, rebuildable log. For data that is never updated (historical logs), retention by time (90 days, 1 year).

Cross-cluster replication

For multi-region or multi-cloud deployments, MirrorMaker 2 or Cluster Linking (Confluent) replicate topics between Kafka clusters. RAG can then query the local cluster without crossing regions.

Layer 3 — Flink as the streaming processor

Apache Flink is the dominant stream processing piece in 2026. Apache 2.0, a mature distribution, a broad ecosystem. The main alternative is Kafka Streams (simpler, Java-only); RisingWave is the emerging option for pure SQL cases.

What Flink adds to Kafka:

Stateful streaming: time-based aggregations, joins between streams, sessions.
Exactly-once semantics: with checkpoint coordination.
Watermarks: correct handling of out-of-order events.
UDFs in Python/Java: including calls to LLM models.

Flink SQL: the most operational piece

Flink SQL is the most usable part of Flink for data engineers who are not streaming experts. Here is a realistic RAG pipeline example:

-- 1. Define the source: Kafka topic with CDC events from documents
CREATE TABLE documents_raw (
 doc_id STRING,
 title STRING,
 body STRING,
 category STRING,
 updated_at TIMESTAMP_LTZ(3),
 PRIMARY KEY (doc_id) NOT ENFORCED
) WITH (
 'connector' = 'upsert-kafka',
 'topic' = 'ecommerce.public.documents',
 'properties.bootstrap.servers' = 'kafka:9092',
 'key.format' = 'avro-confluent',
 'value.format' = 'avro-confluent',
 'value.fields-include' = 'EXCEPT_KEY'
);

-- 2. Define the sink: vector store via an intermediate Kafka topic
CREATE TABLE documents_embedded (
 doc_id STRING,
 chunk_id INT,
 title STRING,
 chunk_text STRING,
 category STRING,
 embedding ARRAY<FLOAT>,
 embedded_at TIMESTAMP_LTZ(3),
 PRIMARY KEY (doc_id, chunk_id) NOT ENFORCED
) WITH (
 'connector' = 'upsert-kafka',
 'topic' = 'rag.documents.embedded',
 'properties.bootstrap.servers' = 'kafka:9092',
 'key.format' = 'json',
 'value.format' = 'json'
);

-- 3. UDF for chunking (defined in Python or Java)
-- CREATE TEMPORARY FUNCTION chunk_text AS 'com.example.ChunkingUDF';

-- 4. Pipeline: chunk, embed, write to the sink
INSERT INTO documents_embedded
SELECT
 doc_id,
 chunk_idx AS chunk_id,
 title,
 chunk AS chunk_text,
 category,
 OPENAI_EMBEDDING(chunk,
 'text-embedding-3-small') AS embedding,
 CURRENT_TIMESTAMP AS embedded_at
FROM documents_raw
CROSS JOIN UNNEST(chunk_text(body, 512, 64))
 WITH ORDINALITY AS t(chunk, chunk_idx);

What happens here, line by line:

The documents_raw table reads the CDC topic in upsert-kafka mode (each new event on the same key replaces the previous one). This correctly reflects the semantics of “this is the latest version of doc 42”.
The documents_embedded table will be the intermediate topic where Flink writes the embedded chunks.
The chunk_text UDF (defined in Python or Java) splits each doc into 512-token chunks with an overlap of 64.
The INSERT INTO query runs continuously: every new event in documents_raw is chunked, each chunk is embedded with OPENAI_EMBEDDING (a built-in Flink SQL function in Confluent Cloud 2026), and written to the embedded topic.

OPENAI_EMBEDDING can be swapped for a custom function that calls a self-hosted model (vLLM with an encoder), SentenceTransformers, or a managed service. The syntax is the same; you change the provider.

Watermarks and late events

For cases where an event can arrive late (for example, the Postgres WAL falls behind because of a network blip), Flink lets you define watermarks:

CREATE TABLE documents_raw (
 doc_id STRING,
 title STRING,
 body STRING,
 updated_at TIMESTAMP_LTZ(3),
 WATERMARK FOR updated_at AS updated_at - INTERVAL '5' MINUTE
) WITH (...)

This tells Flink “assume no event arrives more than 5 minutes late relative to the event timestamp”. For temporal joins and aggregations, Flink uses the watermark to decide when to “close” a window.

Layer 4 — Sinks to vector stores

The last step is indexing the embeddings in a vector store. Three patterns in 2026:

Pattern A — direct Kafka Connect sink

Each vector store has its official connector:

Milvus: official Zilliz sink connector. Supports named/unnamed dense/sparse vectors.
Qdrant: official sink connector. Supports dense, sparse, multi-vector.
pgvector: has no dedicated connector, but the JDBC Sink Connector is used with custom SQL.
Weaviate: community connector.
LanceDB: community connector.

An example Milvus sink configuration:

{
 "name": "milvus-rag-embeddings-sink",
 "config": {
 "connector.class": "com.milvus.io.kafka.MilvusSinkConnector",
 "tasks.max": "3",
 "topics": "rag.documents.embedded",
 "milvus.host": "milvus.prod.internal",
 "milvus.port": "19530",
 "milvus.collection.name": "documents",
 "milvus.collection.dim": "1536",
 "milvus.collection.partition": "default",
 "key.converter": "org.apache.kafka.connect.storage.StringConverter",
 "value.converter": "org.apache.kafka.connect.json.JsonConverter",
 "value.converter.schemas.enable": false
 }
}

Three parallel tasks (tasks.max: 3) consume the embedded topic and write to the Milvus collection. Latency from “event in Kafka” to “indexable vector in Milvus” is typically <5 seconds.

Pattern B — pgvector with a direct CDC pipe

For teams that already live in PostgreSQL, pgvector is the lowest-friction option. The pattern: the same source Postgres cluster holds a second DB for embeddings with the pgvector extension enabled; the Flink pipeline writes to it directly over JDBC.

-- On the Postgres cluster with pgvector enabled
CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE document_embeddings (
 doc_id TEXT,
 chunk_id INT,
 chunk_text TEXT,
 category TEXT,
 embedding vector(1536),
 embedded_at TIMESTAMP,
 PRIMARY KEY (doc_id, chunk_id)
);

CREATE INDEX ON document_embeddings
 USING hnsw (embedding vector_cosine_ops)
 WITH (m = 16, ef_construction = 64);

Advantages: the same DBA operates everything, transactionality across tables, trivial joins with relational metadata. Limitation: beyond 10M vectors, pgvector performance starts to give way compared with dedicated systems.

Pattern C — Confluent Tableflow → Iceberg + Flink SQL vector search

This is the 2026 development that changes the mechanics. Confluent Tableflow materialises Kafka topics automatically as Apache Iceberg or Delta Lake tables. Features:

No ETL pipeline: you do not write Flink/Spark jobs to move Kafka into a table. Tableflow does it.
Automatic schema evolution: changes to the topic schema are reflected in the table.
Unified catalogue: the table appears in Glue, Unity Catalog, Snowflake, Databricks. Any analytical engine queries it without copying data.
Native CDC: handles inserts, updates and deletes correctly.
30-50 % lower TCO according to the figures Confluent publishes against traditional pipelines.

And since 2026, Tableflow + Flink SQL offer native vector search integrated with Cosmos DB and Amazon S3 Vectors. The RAG query can be made directly in Flink SQL:

SELECT doc_id, chunk_text, category
FROM documents_embedded
WHERE VECTOR_SEARCH(embedding,
 OPENAI_EMBEDDING('user query', 'text-embedding-3-small'),
 top_k => 10) > 0.7
ORDER BY VECTOR_SEARCH_SCORE DESC;

This unifies layers that used to be separate (vector store + analytics). For many cases, it removes the need to maintain a dedicated vector store.

Confluent’s official MCP server

One piece added in 2026 deserves a mention: Confluent has published an official MCP server that exposes Kafka, Flink and Tableflow as tools accessible to AI agents over MCP. Any MCP client (Claude Desktop, Cursor, your own agents) can:

List topics, read recent messages, publish to topics.
Run Flink SQL queries in natural language (“give me the orders from the last 24 hours worth more than 1000 €”).
Query Tableflow Iceberg tables.
Manage Kafka Connect connectors.

This closes the circle: as well as reading data from the datalake through RAG (with vector search), your AI agent can write data to the log (through MCP) and trigger transformations (through Flink SQL in natural language). It is the deepest point of fusion between LLM ops and data ops this year.

A connection with the earlier series: this MCP server emits traces with the OpenTelemetry GenAI MCP semantic conventions we covered in the MCP observability post. The spans show up in Langfuse, Phoenix or your OTel backend with the right cardinality. Zero instrumentation code.

Vector stores: a 2026 comparison

The five dominant options:

Vector store	Licence	Operation	Where it fits
pgvector	Postgres ext, OSS	Your DBA	<10M vectors, Postgres-heavy team
Qdrant	Apache 2.0	Self-host or managed	Mid-scale, performance focus
Milvus	Apache 2.0	Self-host or Zilliz Cloud	Large-scale, scalability focus
Weaviate	BSD-3	Self-host or managed	Native hybrid search, semantically rich
LanceDB	Apache 2.0	Embedded or serverless	Small-medium, simplicity

The choice depends on:

Scale: pgvector falls short beyond 10M vectors. Milvus and Qdrant scale to billions.
Hybrid search: Weaviate brings lexical + vector natively. Others support it but less integrated.
Operation: pgvector if you already run Postgres. Qdrant if you want simplicity. Milvus if you need maximum scale.
Cloud managed: Zilliz Cloud for Milvus, Qdrant Cloud for Qdrant, Pinecone if you want pure SaaS (with no OSS behind it).

Freshness vs accuracy: the operational trade-off

A critical decision any RAG system on Kafka must answer: when is a new document considered “live” in the index?

Three options:

Synchronous streaming: the event reaches Kafka, Flink embeds it, the sink writes it to the vector store, and only then is it considered live. Typical latency: 1-5 seconds. The best freshness. But if the embedding model fails or the vector store is slow, events pile up in the topic.

Asynchronous streaming with a baseline: the event is considered live immediately; a background process embeds it when it can. In the meantime, queries asking for that document do not find it. Typical latency: 5-60 seconds. Acceptable for most applications.

Micro-batch: processing happens in mini-batches every 1-5 minutes. Less efficient than continuous streaming but more stable under variable load. Latency: 1-5 minutes.

The decision depends on the product’s SLA. For customer support chatbots, 5-60 seconds is acceptable. For systems that react to critical events (financial prices, alarms), synchronous streaming is necessary.

Schema evolution and re-embedding

When the embedding model changes (you move from text-embedding-3-small to text-embedding-3-large, or from OpenAI to Cohere), the vectors already in the index are incompatible: different dimensions, different semantic spaces. The distance between an old vector and a new one means nothing.

The standard pattern for handling this: dual-index during the migration.

T0: the active index is V1 (embedding model A).
T1: a parallel pipeline starts writing to a V2 index (embedding model B), consuming the topic from offset 0 (reprocessing the whole log).
T2: V2 has caught up with the present.
T3: you switch the retriever over to V2.
T4: a week later, you discard V1.

The Kafka log makes this pattern feasible because it is immutable and reproducible. Without the log, this pattern becomes a weeks-long data migration project.

Operational pitfalls

Topics without adequate retention

Configuring topics with 7-day retention on the grounds that “I already have the vector store” means losing the ability to rebuild if the vector store fails. Long retention (90+ days) or compaction by key for topics that feed RAG.

Heavy CDC during peak load

Debezium reading the WAL at peak hours can hit the source database’s performance. A dedicated read replica for Debezium, not the production primary. Or use logical replication restricted to just the tables you need.

Embedding cost run-away

OPENAI_EMBEDDING on every event of a topic with millions of messages a day comes to thousands of USD per month. Strategies: filter before embedding (only embed what adds value); deduplicate by content hash; use self-hosted open-source embedding models (BGE, E5, GTE) when the cloud cost is prohibitive.

Slow re-embedding due to limited throughput

Recomputing 10M embeddings with the OpenAI API at 3000 req/min takes 55 hours. If you wait for an incident to re-embed, that is two days without service. Embedding throughput is an explicit capacity planning item; reserve capacity or keep an offline job ready to start.

Downstream schema breaks

A change to the raw topic’s schema breaks Flink jobs downstream. Schema Registry with BACKWARD compatibility made mandatory; never ALLOW_ALL. And test schema evolution in CI.

A vector store with no backup

Your vector store holds 50M vectors. It is the only copy (the topics expired). A failure wipes it. Vector stores must be backed up like any primary persistence layer. For Milvus/Qdrant: periodic snapshots. For pgvector: plain pg_dump.

Multi-region without cross-cluster replication

Your RAG serves users in the US and the EU. The vector store is in US-east. Latency from the EU = 100 ms+ per query. MirrorMaker or Cluster Linking to replicate topics and vector stores in both regions.

What we have not covered

Hybrid search in production: combining BM25/lexical + vector + reranker. A topic for its own post.
Multimodal RAG: indexing images, audio and video alongside text. Multimodal embeddings (CLIP, Imagebind), a specific architecture.
GraphRAG: using structured knowledge (knowledge graphs) as well as vector retrieval. Microsoft GraphRAG, LlamaIndex KnowledgeGraphQueryEngine.
RAG with multi-tenant ACLs: filtering by permissions at runtime. The pattern uses metadata in the vector store + server-side filters.
Query rewriting with an LLM: using a first LLM to expand the query before retrieval (HyDE, multi-query, step-back prompting).

References

Kafka and stream processing:

Apache Kafka y Debezium.
Confluent Schema Registry y Apicurio Registry.
Apache Flink y Flink SQL docs.
RisingWave — alternativa SQL streaming con embedding built-in.

Vector store connectors:

Tableflow and 2026 architecture:

Streaming RAG:

Cross-references:

Previous post: MLOps specific to LLMs in 2026: the landscape.
Data series: PostgreSQL + Qdrant at the ingestion stage, RAG corpus curation: the active librarian, Reranker and hybrid retrieval — the four pieces that close the Data block: streaming, ingest, curation and retrieval.
Post-tracing series: Evals, Guardrails, MCP observability, eBPF + drift.
eBPF series: eBPF from zero to Cilium, Tetragon, Hubble, AgentSight.

MLOps specific to LLMs in 2026: the landscape of three modalities, six stages and ten tools that make them work

Thu, 21 May 2026 05:30:00 +0200

TL;DR

This is the blog’s fourth series and it is called MLOps specific to LLMs. It takes the operational craft of traditional MLOps (reproducible pipelines, model registries, dataset versioning, eval gates, controlled deployments) and redraws it for a world where the model is probabilistic, the outputs are subjective, the dependencies include external vendors that update weights without warning, and the “application” is not a model but an orchestration of models, embeddings, retrievers, guardrails and routers. Gartner predicts that more than 50% of enterprise GenAI deployments will fail before the end of 2026, and the main cause is not the model: it is that deterministic software assumptions were applied to probabilistic systems. This post opens the series with the frame: the seven structural differences between LLMOps and classic MLOps; the six-stage pipeline (data → tune → eval → deploy → observe → retrain); the three modalities of preparing a model (continuous fine-tuning, RAG over datalakes, agent training) with their decision matrix, where 60% of 2025-2026 deployments use hybrid because each modality solves a different problem: “fine-tune for behaviour, RAG for volatile knowledge”; and the 2026 tool landscape, which already forms reasonably stable layers: MLflow 3.10 (March 2026) as a GenAI-aware registry, W&B Weave and ZenML for tracing and pipelines, Kubeflow + KServe vLLM 0.8.1+ for serving, BentoML for flexibility, DVC + lakeFS (joined since November 2025) for data, Langfuse for prompts and observability. The three following posts will go down into the detail of the most critical pieces.

This is the opening of series 4: MLOps for LLMs. It continues the tradition of the previous series: LLM inference (the first), eBPF (the second) and post-tracing (the third). Here we enter the discipline that ties all the pieces together: how an LLM system is operated in production over months, not just deployed once.

The analogy: the SRE’s craft redrawn

Anyone who has worked for years as an SRE or a platform engineer recognises the classic pillars: reproducibility (same code + same data + same config = same result), observability (what happens can be measured), safe rollback (if something goes wrong, I go back within minutes), gradual rollout (the new thing goes to 1% before it goes to 100%). These pillars are not negotiable. The question is whether they hold up when the central component is an LLM.

The answer is: same pillars, radically different mechanics. Reproducibility: versioning code and data is no longer enough; you have to version prompts, retrieval configurations, snapshots of the vendor’s model (which change without warning). Observability: error and latency metrics are no longer enough; you have to measure subjective quality via LLM-as-judge and embedding drift. Rollback: dropping the binary’s version is no longer enough; you have to keep the old model cached because loading a new one takes minutes. Gradual rollout: a % of traffic is no longer enough; you have to decide what % of what kind of queries by segment.

It is the same craft, practised with partly new tools and reflexes. MLOps specific to LLMs, or “LLMOps”, as the field has christened itself, is the discipline that codifies those reflexes.

The seven structural differences between LLMOps and traditional MLOps

Before going down to the pipeline, let us fix the differences that make this territory new rather than a mere continuation. Each one has concrete practical consequences.

1. Non-deterministic outputs

Traditional MLOps: the model receives structured input and returns a bounded, reproducible prediction. Same input → same output. Unit tests work.

LLMOps: same input → different output every time (through sampling, through temperature, through the order of tools invoked, through the retrieval context that changed). The idea of a “unit test” breaks.

Operational consequence: tests over properties (was the tone maintained?, does it mention the source?, does it respect the JSON schema?), not over equality. Statistical evals over a distribution, not over samples.

2. Behaviour metrics, not statistical accuracy

Traditional MLOps: F1, accuracy, AUC, RMSE. Metrics with a clear number.

LLMOps: subjective rubric scores (G-Eval, faithfulness, helpfulness, toxicity), judge LLMs, human feedback. The “number” depends on who judges.

Operational consequence: platforms have to treat evals as versioned artifacts, not just “model v3 scored 0.87”, but “model v3 evaluated with the judge claude-3-5-sonnet-20251022 over the dataset gold-rag-v7 with the prompt judge-v2 scored 0.87”. Versioning the judge is as important as versioning the model being evaluated.

3. The model is an external dependency, not an internal asset

Traditional MLOps: you train the model, it lives in your registry, it does not change until you change it.

LLMOps: the base model belongs to Anthropic, OpenAI, Google, Meta. They change it on you without warning. The claude-3-5-sonnet version that answered well yesterday answers somewhat differently today.

Operational consequence: drift detection becomes far more critical (previous post). Pinning to specific snapshots (claude-3-5-sonnet-20251022) when the vendor allows it. For high-commitment apps, self-hosting the base model to guarantee reproducibility.

4. The application is an orchestration, not a model

Traditional MLOps: an app calls a model and consumes its output.

LLMOps 2026: an app connects foundation model + LoRA adapters + retrievers + vector stores + guardrails + routers + tool servers (MCP) + evaluators, all composing the final behaviour. Any component can degrade the result.

Operational consequence: cross-component debugging requires distributed tracing with OTel (covered in previous posts). The registry does not only store “the model” but the composition: which prompt version + which adapter + which vector store + which retriever config.

5. Cost per inference, not per training

Traditional MLOps: the high cost is training; serving is cheap. You optimise training.

LLMOps: the high cost is serving (every token costs, every call to the vendor is paid for, the GPUs doing the serving are switched on 24/7). You optimise inference.

Operational consequence: cost accountability per tenant, per agent, per tool. Metrics such as gen_ai.usage.input_tokens aggregated at customer and product level. Model decisions based on cost per query, not only on quality.

6. GPU-heavy infra with specific primitives

Traditional MLOps: CPU + some GPU for training. Standard Kubernetes.

LLMOps: Hopper/Blackwell SXM GPUs, NVLink/NVSwitch, tensor parallel, paged attention, KV cache. Infra that only fits into Kubernetes with primitives such as LeaderWorkerSet, GPU Operator, KEDA with LLM metrics (covered in vLLM on Kubernetes).

Operational consequence: the orchestration stack includes specialised operators (OME, vLLM Production Stack, NVIDIA Dynamo, llm-d, see LLM K8s Operators) that traditional MLOps does not contemplate.

7. RLHF and human feedback as a first-class citizen

Traditional MLOps: human feedback means labelling data before training.

LLMOps: human feedback lives inside the model in production, whether through RLHF of the foundation models (Anthropic, OpenAI), through RLAIF, through DPO, or through explicit user feedback that is fed back into fine-tuning.

Operational consequence: bidirectional production → training pipelines. Datasets grow with real incidents. Model decisions are taken with continuous feedback, not in a training project every N months.

Why Gartner predicts 50%+ failures

Gartner published that more than 50% of enterprise GenAI deployments will fail before 2026. The causes are not technical ones about the model but about the system:

Hallucinated outputs from poor grounding: badly designed RAG, poor retrieval, insufficient context.
Data architectures that are not ready: companies have data in silos, with no stable schemas, with no controlled freshness. Connecting an LLM to this data without a serious pipeline produces erratic answers.
Lack of structured workflows for prompt-driven systems: teams that treat prompts as code in hardcoded strings, with no versioning, no tests, no gates.

The conclusion the field draws: LLMOps is not optional. Companies that deploy GenAI without operational discipline fall into one of the three failure modes. Those that apply it (MLflow/W&B for tracking, DVC/lakeFS for data, Langfuse for prompts and evals, KServe or vLLM Production Stack for serving, drift detection in production) are the ones that keep the system working six months after the first release.

The six-stage LLMOps pipeline

On to the pipeline. The six stages that any serious LLM system goes through, in order:

[1. Data] → [2. Tune] → [3. Eval] → [4. Deploy] → [5. Observe] → [6. Retrain]
│
└─→ back to 1

Each stage is an operational domain of its own, with its own tools and traps:

Stage 1 — Data

What happens: ingestion, cleaning, curation, versioning, indexing of the corpus. It is where real projects suffer most, because companies have data in heterogeneous silos.

Typical sub-tasks: extraction from source (CDC over Kafka, batch from data lakes, scraping), cleaning (PII removal, dedup, format), curation (labelling for fine-tuning, golden datasets for eval), versioning (DVC + lakeFS), indexing (embeddings + vector store for RAG).

Traps: schema drift at the source, undetected PII, poor dedup that injects redundancy into training, a vector store that is not updated.

Stage 2 — Tune

What happens: preparing the model for your use case. Three modalities (we go deeper into them shortly): fine-tuning, RAG, agent training.

Typical sub-tasks: base model selection, adapter preparation (LoRA, QLoRA), training loop with continuous eval, hyperparameter sweep (Optuna, W&B Sweeps), saving the checkpoint.

Traps: catastrophic forgetting if the fine-tuning is very aggressive, overfitting to the golden dataset, no independent validation set.

Stage 3 — Eval

What happens: validating that the model + adapters + RAG configuration is acceptable before promoting. Covered in Evals.

Typical sub-tasks: running an eval framework (DeepEval, Promptfoo, Ragas) against the golden dataset, judge LLM evaluations, human review over a sample, gates with thresholds.

Traps: a golden dataset that ages, an uncalibrated judge, evals that pass in CI but fail in production because of distribution shift.

Stage 4 — Deploy

What happens: going from “the model evaluated well” to “the model serves real traffic”. Covered in LLM K8s Operators.

Typical sub-tasks: serving with vLLM/SGLang/TRT-LLM, runtime configuration, gradual rollout (canary, shadow, blue-green), routing between models (LiteLLM, OpenRouter, LangChain routers).

Traps: a naive rolling update that cuts sessions, autoscaling by CPU% that does not respond to LLM metrics (covered), a new model that performs worse in production than in eval.

Stage 5 — Observe

What happens: seeing what is happening in real time. Covered in the whole post-tracing series and the eBPF series.

Typical sub-tasks: tracing (Langfuse, LangSmith, Phoenix, OpenLLMetry), metrics (TTFT, TPOT, queue depth, cost per query), active guardrails (NeMo, Llama Guard), drift detection (Evidently, NannyML, WhyLabs).

Traps: cardinality explosion in metrics, batch evals with no tail-sampling over real traces, drift that is ignored until an incident makes it real.

Stage 6 — Retrain

What happens: closing the loop. Production feedback (incidents, worst-scoring cases, detected drift) generates new data to go back to stage 1.

Typical sub-tasks: extraction of problematic logs, human labelling of the sample, incorporation into the golden dataset, re-fine-tuning if applicable, decision on a new release.

Traps: an “open” loop where production never informs the dataset, human feedback that gets lost, lack of a clear retrain cadence.

The three modalities of “preparing the model”

Stage 2 (Tune) is where there is most confusion. In 2026 three modalities coexist, each solving a different problem:

Fine-tuning

What it does: modifying the model’s weights (or those of a LoRA/QLoRA adapter on top) so that it learns specific behaviour patterns: tone, output structure, idiomatic decisions of the domain.

When: when your main failure is behavioural inconsistency between calls. The model sometimes answers formally, sometimes not; sometimes it structures the JSON, sometimes not; sometimes it follows the company’s conventions, sometimes it invents them. Fine-tuning stabilises it.

When NOT: when what you need is up-to-date knowledge. Fine-tuning fixes knowledge in frozen weights; the day after the fine-tuning, the model knows nothing new.

RAG (Retrieval-Augmented Generation)

What it does: leaving the model intact and, on each call, retrieving fresh context from a knowledge base (vector store + lexical search, typically) and passing it to the model so that it answers on that basis.

When: when the knowledge you need is dynamic or very large. Documentation that changes, a product catalogue that is updated, an internal knowledge base that grows.

When NOT: when the problem is behavioural (RAG does not teach the model how to behave, it only gives it information). Or when retrieval is so noisy that the context arriving is worse than nothing.

Agent training

What it does: going beyond conventional fine-tuning with Reinforcement Learning techniques. RFT (OpenAI’s Reinforcement Fine-Tuning), classic RLHF, RLAIF (with AI feedback), DPO (Direct Preference Optimization) over datasets of (good, bad) pairs.

When: when the model needs to learn complex multistep trajectories: when to choose each tool, how to decompose a task, when to ask the user for confirmation. It is what is turning Claude, Gemini and GPT into agents capable of long tasks.

When NOT: when your case is simple chat or RAG. It is overkill, expensive and complicated for problems that the previous modalities solve.

Decision matrix

Observed problem	Modality
Inconsistent answers in tone/structure	Fine-tuning
The model makes things up (hallucinates)	RAG
Out-of-date knowledge (>1 year)	RAG
The model chooses tools badly	Agent training (RLAIF/RFT)
Mixed behaviour + knowledge	Hybrid (fine-tune + RAG)
Multi-step trajectory fails	Agent training
A specific language/regional style	Fine-tuning

The 2026 verdict: hybrid is the default

Multiple reports agree that in 2025-2026, around 60% of production projects use hybrid: fine-tuning for behaviour + RAG for knowledge. The key insight:

Fine-tune for behaviour (brand voice, decision protocol, output structure); use RAG for volatile knowledge you need the model to cite. Do not force a single tool to do both jobs.

A practical observation: the biggest quality improvements of 2025-2026 come from better reranking in RAG (cross-encoders), not from better embeddings. Rerankers add 15-35% of quality with little complexity.

On cost: combined fine-tuning + RAG tends to be 30-50% cheaper than pure RAG with frontier models at high volume, because the fine-tuned model can be smaller and cheaper while keeping equivalent quality.

The 2026 tool landscape

On to the concrete pieces, grouped by function. The field has matured enough for each piece to have 2-3 reasonable options and a couple of leaders.

Experiment tracking and model registry

MLflow remains the de facto standard, now with LLM-specific traction. MLflow 3 was published in June 2025; version 3.10.1 (March 2026) added:

A GenAI Overview dashboard with ready-made metrics for LLM apps.
Multi-workspace support for large teams.
Cost tracking in traces (gen_ai.usage.* aggregated per experiment).
MemAlign: a new specific eval algorithm.
Integrated native OpenTelemetry tracing.
First-class support for LangChain, LlamaIndex, AutoGen as frameworks.

MLflow treats prompts and agents as first-class citizens alongside classic models. It is the biggest change relative to MLflow 2.x.

Weights & Biases (W&B) with its LLM-specific product Weave offers tracing + eval + debug with a very polished UI. More commercial, less self-host friendly, but excellent UX.

ZenML is the piece that most cleanly integrates “classic MLOps + emerging LLMOps” into a single framework. Its automatic artifact versioning captures prompt templates, retrieval chunks and agent conversation histories with no extra work. Open-source. The most complete unification option that exists.

Dataset versioning

DVC remains the OSS standard. It extends Git to large files and pipelines. Important news from November 2025: lakeFS acquired DVC, consolidating the two OSS data versioning projects under one organisation. The combined roadmap is oriented towards LLM training and RAG datalakes specifically.

Typical pattern: Git for code + DVC for data/models + MLflow or W&B for experiment tracking + registry. Few teams use just one; the combination is what covers the cycle. Detailed in the dedicated post on data versioning for LLMOps — the four artifacts to version separately, schema contracts, end-to-end lineage, and a golden eval set with a strict holdout.

Prompt versioning and observability

Covered in depth in the AgentSight post where we go deeper into Langfuse as the OSS reference. Summary here:

Langfuse: MIT, self-host, prompt management with v1/v2/v3 versioning + labels + cache + linkage with traces.
LangSmith: if your stack is LangChain.
Arize Phoenix: ELv2, OTel-native.

Pipeline orchestration

For the LLMOps pipeline steps, the dominant options:

Kubeflow Pipelines: the K8s-native standard. KServe (the serving part of Kubeflow) has its vLLM runtime upgraded to v0.8.1+ with support for reasoning models, tool calling, embeddings, reranking, Llama 4 and Qwen 3.
ZenML: already mentioned; also a pipeline orchestrator.
Metaflow (Netflix-originated): Python-first pipelines, less LLM-specific but workable.
Argo Workflows: a pure K8s OSS alternative.
Flyte: Kubernetes-native, OSS.

Serving

Covered in depth in vLLM on Kubernetes and LLM K8s Operators. Summary:

vLLM Production Stack: a curated Helm chart.
KServe vLLM runtime: K8s-native, vLLM 0.8.1+ with full agentic support.
BentoML: flexible serving, popular in startups for its simplicity.
NVIDIA Dynamo: the successor to Triton.
llm-d: CNCF Sandbox.
OME: an LMSYS operator with native SGLang.

Evals and guardrails

Covered in Evals and Guardrails. Ultra-short summary:

Evals CI: DeepEval, Promptfoo, Ragas.
Evals platform: Langfuse, LangSmith, Phoenix, Braintrust.
Guardrails: NeMo Guardrails, Llama Guard 4, Llama Prompt Guard 2, LLM Guard, Lakera.

Drift detection and observability

Covered in the closing eBPF post. Summary:

Drift: Evidently AI, NannyML, WhyLabs.
Tracing: Langfuse, OpenLLMetry, Phoenix.
eBPF: AgentSight, Hubble, Tetragon, ProfInfer.

The typical 2026 stack table

Stage	Dominant piece	Alternatives
Data ingestion + versioning	DVC + lakeFS (unified Nov 2025)	Pachyderm, Quilt
Vector store / RAG index	Milvus, Qdrant, pgvector, Weaviate	LanceDB, Pinecone, Chroma
Experiment tracking	MLflow 3.10	W&B Weave, Neptune
Pipeline orchestration	Kubeflow + Argo	ZenML, Metaflow, Flyte
Model registry	MLflow registry	W&B Models, KServe ModelMesh
Prompt versioning	Langfuse	LangSmith, MLflow Prompts
Serving	vLLM Production Stack	KServe, BentoML, Dynamo, llm-d, OME
Evals CI	DeepEval, Ragas	Promptfoo, OpenAI Evals
Evals platform	Langfuse, Phoenix	LangSmith, Braintrust
Guardrails	NeMo + Llama Guard	LLM Guard, Lakera
Tracing	OpenLLMetry + Langfuse	Phoenix, LangSmith
Drift detection	Evidently AI	NannyML, WhyLabs
eBPF observability	AgentSight + Tetragon + Hubble	(new territory, few alternatives)

13 pieces. No org uses them all; any serious org uses at least six. This is the current LLMOps stack.

The operational reality: nobody uses a single tool

Multiple comparisons agree on one thing: the teams that win combine. Recurring patterns:

ZenML to orchestrate + MLflow for tracking + KServe for serving: the most popular OSS stack in companies coming from classic MLOps.
Kubeflow + W&B + BentoML: for teams with a research focus.
Langfuse + DeepEval + Phoenix + LiteLLM: for pure-LLM teams with no classic MLOps background.
MLflow + DVC + Argo + KServe: an idiomatic cloud-native stack without additional LLM-specifics (with its limitations).

The choice depends on the team’s background, on the licensing model they can afford, on the level of self-hosting they need, and on which frictions blocked them most in previous projects. There is no “one right answer”; there is a stable meta-pattern of layers worth covering.

Common operational traps

Treating the prompt as text in code

Hardcoding prompts in strings in the repo. Changing them requires a PR + redeploy. Result: teams that do not iterate on prompts because every change costs hours of pipeline. Solution: externalised prompt management (Langfuse, MLflow Prompts) with versioning, labels, hot-reload.

Skipping dataset versioning

“DVC is complicated, we will add it later”. Result: two months later, nobody knows which dataset trained which model. Impossible to reproduce incidents. Solution: DVC + lakeFS from day 1, even if only with a small subset.

Mixing layers in the same pipeline

Teams that put ingestion, fine-tuning, eval and deploy into a single giant pipeline. When something fails, the whole pipeline fails. Solution: independent pipelines per stage, with versioned artifacts as the interfaces between them.

Tracking with no structure

Logging everything to stdout and “we will look at it in CloudWatch later”. Result: impossible to correlate, compare, debug. Solution: OTel from day 1 with gen_ai.* semantic conventions.

Evals that block nothing

You have evals, you run them, you look at them, but they do not prevent the deploy if they drop. Eventually it drops gradually and nobody notices. Solution: eval gates in CI/CD that block the merge if critical metrics regress by more than X%.

No retrain cadence

You launch v1 and never go back to the model. Six months later, drift has degraded it but the team is on other projects. Solution: a formal retrain cadence (monthly, quarterly) tied to the production incident queue.

Invisible vendor lock-in

You start with the OpenAI API + LangSmith + Pinecone. When you want to self-host, you discover that migrating is a 3-month project. Solution: abstraction layers (LiteLLM, OpenLLMetry) and vendor-neutrality from the start.

What is coming in the following posts of the series

Post 2 — RAG over datalakes with Kafka: end-to-end technical architecture — the most hands-on one. Kafka as source-of-truth, Flink CDC, embedding pipelines, continuous indexing in Milvus/Qdrant, a complete example with real numbers and manifests.
Post 3 — The six-stage LLMOps pipeline: global architecture — the master map of the complete system with a reusable “you are here” SVG for the following posts. A deep dive into each of the six stages (Data, Tune, Eval, Deploy, Observe, Retrain).
Post 4 — PostgreSQL + Qdrant in the ingestion stage — synchronisation patterns (dual-write, outbox + CDC, event-driven), a complete microservices architecture, a Qdrant cluster manifest.
Upcoming posts — still to be decided: the cluster as a multi-tenant platform, Constitutional AI / alignment runtime, continuous fine-tuning in depth, edge LLMs.
Synthesis post — Anatomy of an LLM request in production, May 2026 — the complete journey of a real request through the six stages and the two cross-cutting components, with cross-links to each dedicated post. It works as a mental map of the blog and as an integrator’s guide.
Parallel OSS vs hyperscalers catalogue — The parallel catalogue: the six LLMOps stages in OSS and in AWS / GCP / Azure — for each stage of the pipeline, which tools the blog’s reference OSS stack uses and what the cloud equivalents are, with summary tables, identification of real gaps and the multi-tenant chatbot ported to an AWS stack component by component. A neutral editorial stance.
OSS catalogue entry by entry — The OSS catalogue for LLMOps in six stages: entry by entry — the zoom in on the open source side: ~150 words of description per core tool (vLLM, Langfuse, DVC, Qdrant, Airflow, NeMo Guardrails, Presidio…), licence and governance, a decision matrix per stage, a diagram of the connected OSS stack and a master table of licences and EE / SaaS offerings.

References

LLMOps vs MLOps:

Tools:

MLflow — registry + tracking + serving.
Weights & Biases Weave — LLM tracing.
ZenML — pipeline orchestration MLOps + LLMOps.
Kubeflow — K8s-native MLOps.
KServe — model serving K8s.
BentoML — flexible serving.
Metaflow — Netflix’s pipelines.
DVC — dataset versioning.
lakeFS — enterprise data versioning, acquired DVC in Nov 2025.

2026 comparisons:

RAG vs Fine-Tuning:

Cross-references (the three previous series):

LLM inference series: KV cache, vLLM on K8s, PagedAttention, LLM K8s Operators.
eBPF series: eBPF from zero to Cilium, Tetragon, Hubble, AgentSight.
Post-tracing series: Evals, Guardrails, MCP observability, eBPF + drift.

Rag on lo0 — Tech Blog

Serving embeddings and rerankers with TEI in production

TL;DR

The analogy: the stamping press that only stamps vectors

What TEI is, exactly

The endpoints

The backends

Why a dedicated server and not calling the model inline

Dynamic batching: the mechanics of the press

The number: throughput with batching vs without batching

Deployment: compose for embedder and reranker

Embedder (bge-m3) on GPU

Reranker (bge-reranker-v2-m3) on GPU

CPU for batch ingestion

Integration: how ingestion and the gateway consume it

Sizing: memory, replicas and latency budget

Footprint of a 568M encoder

Latency budget: short query vs top-50 rerank

Applied to the generic 4×H100 cluster

Conclusion

See also

References

End-to-end document ingestion: from PDF to indexed chunk

TL;DR

The analogy: an archive’s cataloguing chain

The six-stage pipeline

Stage 1 — Parsing: layout-aware vs plain text

OCR for scans

Stage 2 — Cleaning, normalising and deduplicating

Deduplication: why it matters

Stage 3 — Chunking

What the 2026 benchmarks say (and why semantic disappoints)

A worked numerical example of chunking

Stage 4 — Enriching with metadata

Stages 5 and 6 — Embedding and indexing

A cross-cutting stage — Incremental ingestion and idempotency

The arithmetic: sizing the corpus and the index

Corpus sizing and ingestion time

Vector index size

Applied to the generic 4×H100 cluster

Closing: quality is decided upstream

See also

References

Taking RAG to the CPU: separating the data plane from the generation plane

TL;DR

The analogy: the library and the librarian

The three phases and their compute profiles

Why ingestion fits on CPU: the embedder is not an LLM

Runtimes that already do this effortlessly

Viability table: CPU for each component?

The numbers, with honest methodology

Decision tree: CPU or GPU for this piece?

Reference architecture (a): CPU-only

Reference architecture (b): the recommended hybrid

Applied to the generic 4×H100 cluster

The auditability angle: ENS / NIS2

When NOT to take it to CPU

See also

References

Aggressive RAG in small models: trading parameters for retrieval

TL;DR

The analogy: the open-book exam

The capacity argument: how many facts fit in the weights

The central tension: retrieving more is not putting in more

(a) SLMs make worse use of long context

(b) Prefill grows with the context and is compute-bound

The maths of prefill

Numerical example: TTFT before and after compressing, RTX 4090

The five levers to resolve the tension

1. Aggressive reranking: precision over recall

2. Context compression: distilling the crib sheet

3. Prefix caching of stable context

4. Semantic caching of answers

5. Structured output and function calling: lean on tools, not on memory

The complete pipeline

Implications for on-premise inference

On the RTX 4090 (24 GB, Ada Lovelace)

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

What we have not covered

See also

Embedder (`bge-m3`) on GPU

Reranker (`bge-reranker-v2-m3`) on GPU