Embeddings on lo0 — Tech Blog

Serving embeddings and rerankers with TEI in production

Thu, 11 Jun 2026 09:40:00 +0000

Sixth piece in an operational series about squeezing a generic on-premise LLM cluster of 4×H100 SXM 80 GB. If RAG on the CPU argued where the data plane runs and the embedder zoo decided which model you serve, this post looks at the engine that serves them: the embeddings and rerankers server. The sisters in this batch, document ingestion from PDF to indexed chunk, GitOps for the inference stack with Flux and hardening and secrets for the sovereign stack, build the rest of the system around this piece. The closer of the series, the end-to-end sovereign assistant with LibreChat + LiteLLM + RAG, is still in draft.

TL;DR

Serving embeddings is not model.encode(text) scattered through the ingestion code. Just as you do not put a transformers.generate() inside your API and call it vLLM, you do not put the embedder inline either: you put it behind a dedicated server that does batching, controls concurrency, emits metrics and exposes a single HTTP contract that both ingestion and query-time reuse. In the 2026 open source ecosystem that server is TEI, Text Embeddings Inference from Hugging Face: a Rust binary that serves embedding, reranker (cross-encoder) and sequence classification models, with token-based dynamic batching, and that exposes OpenAI-compatible endpoints (/v1/embeddings) on top of the native /embed, /embed_sparse, /rerank and /predict (TEI repo, docs). It runs on CPU (ONNX Runtime, Intel MKL) and on CUDA (with FlashAttention and cuda graphs), with the same contract on both sides, so the rest of the system does not know, or care, what silicon sits behind it. The technical piece that gives almost all the performance is dynamic batching: TEI groups requests that arrive separately into a batch that shares the fixed cost of the forward, controlled by --max-batch-tokens (how many tokens fit per batch) and --max-concurrent-requests (backpressure). Grouping raises throughput, going from serving 1 sequence to 32 multiplies tokens/s at almost constant cost until the hardware saturates, in exchange for some latency while the tray fills. The numbers: a 568M encoder (bge-m3, bge-reranker-v2-m3) takes ~1.1 GB in fp16 and ~0.57 GB in int8, so several replicas fit even in a small H100 slice; embedding a short query online costs tens of ms, but reranking the top-50 costs linearly in candidates because a cross-encoder evaluates each (query, doc) pair separately. The split across the 4×H100 cluster: TEI-CPU on the fleet for batch ingestion (no SLA, throughput/€), TEI-GPU in a MIG slice of one H100 for high-QPS online embeddings and rerank, talking to ingestion and to the LiteLLM gateway through the OpenAI contract.

The analogy: the stamping press that only stamps vectors

Picture a workshop with a specialised stamping press. It does not make varied parts; it does one single thing: it takes flat material and, in one stroke, stamps a shape into it. In our case it takes text and stamps a vector. And like every industrial press, it has one property that defines all its behaviour: the cost of the stroke is almost fixed, no matter how much material you put on the tray.

Lowering the press, heating the die, aligning, pressing and lifting costs, say, one second. If you put a single sheet in, you spend that whole second stamping one part: a waste. If you fill the tray with thirty sheets and lower the press once, you spend nearly the same second and come out with thirty stamped parts. The cost per part collapses. The press performs far better when it fills the tray before pressing.

This is exactly the dynamic batching of an embeddings server. The “stroke” is the model’s forward pass on the GPU or the CPU: an operation that loads the weights, multiplies them by the activations and emits the vectors, with a large fixed cost that does not scale with how many sequences you process at once, until you saturate memory bandwidth or the compute units. Serving requests one at a time is stamping one sheet per stroke. Grouping them into a batch is filling the tray.

But the press has an operational dilemma, and here is the heart of the post. If you wait until the tray is completely full before pressing, the first sheets that arrived sit waiting for the rest: latency. If you press as soon as one sheet drops so nobody has to wait, you go back to the waste of one stroke per part: low throughput. The embeddings server solves this with a bounded wait window: it gathers whatever arrives within a window of microseconds to milliseconds, or until the batch token budget is full, and then it presses. That is the policy that turns a pile of independent requests into a few efficient strokes without anyone waiting too long.

The rest of the post is, at bottom, about configuring that press properly: how many sheets fit on the tray (max-batch-tokens), how much queue is tolerated before work is rejected (max-concurrent-requests), whether the press lives on the cheap CPU fleet (nightly ingestion, pure throughput) or in a slice of the expensive GPU (online, bounded latency), and why the reranker is a different press whose cost does grow with the number of parts.

What TEI is, exactly

Text Embeddings Inference (TEI) is Hugging Face’s inference server for the family of models that does not generate tokens: embedders, cross-encoder rerankers and sequence classifiers. It is to encoders what vLLM is to generative LLMs: a high-performance server, written in Rust, that handles tokenisation, batching, concurrency, metrics and the HTTP contract, leaving the application code to only make calls (TEI repo).

Three model types, three jobs:

Embedding (bi-encoder): turns a text into a reusable vector. bge-m3, Snowflake Arctic Embed, multilingual-e5. The vector is stored in the index and reused on every search.
Reranker (cross-encoder): takes a pair (query, document) and emits a relevance score, not a vector. bge-reranker-v2-m3. It produces nothing reusable: every pair is evaluated again.
Sequence classification: emits labels/probabilities over a text. Useful for lightweight guardrails, routing, language or toxicity detection.

The endpoints

TEI exposes both an OpenAI-compatible endpoint and its native endpoints, verified against the current documentation (Quick Tour, DeepWiki):

Endpoint	What for	Model type
`/v1/embeddings`	Embeddings with the OpenAI contract (drop-in for clients that already speak OpenAI)	embedder
`/embed`	Dense embeddings, TEI’s native API	embedder
`/embed_sparse`	Sparse embeddings (lexical head of models like `bge-m3`/SPLADE)	embedder
`/rerank`	Orders a list of texts by relevance against a query	reranker (cross-encoder)
`/predict`	Sequence classification (labels/scores)	classifier
`/embed_all`	Per-token embeddings (no pooling)	embedder
`/similarity`	Direct similarity between texts	embedder
`/metrics`, `/health`, `/info`, `/tokenize`, `/decode`	Operation, observability and utilities	all

The operational key is in the first row: TEI speaking the OpenAI contract on /v1/embeddings means the embedder is swappable without touching client code. The same code that called OpenAI’s text-embedding-3-large points at your on-prem TEI by changing the base_url, and the gateway (LiteLLM) routes it as one more provider. /rerank, by contrast, is not part of the OpenAI standard, no such endpoint exists in their API, so the gateway treats it as a reranking-specific endpoint (LiteLLM TEI rerank).

The backends

TEI runs on CPU and on GPU with the same HTTP contract, which is exactly the property the CPU/GPU split of RAG needs (TEI docs, hardware):

CPU: cpu-* image. ONNX Runtime (recommended) or Intel MKL backend. It exploits the integer compute paths (AVX-512+VNNI, AMX on 4th-gen Xeon) when the model is in int8. It is the batch ingestion engine.
CUDA: architecture-specific images (cuda-*, with variants for different compute capabilities). It uses FlashAttention and cuda graphs to squeeze the GPU. It requires compute capability ≥ 7.5 (Volta is out) and CUDA 12.2+ compatible drivers. It is the high-QPS online engine.
There is also Metal support (Apple Silicon) and experimental ROCm (AMD Instinct), less relevant to the on-prem case of this post.

The golden rule: the same bge-m3 served by TEI-CPU and by TEI-GPU exposes an identical /v1/embeddings. The client does not notice the silicon. That is what lets you put ingestion on CPU and online on GPU without rewriting anything.

Why a dedicated server and not calling the model inline

The temptation, especially in the prototype, is to load the embedder into the application process and call model.encode() directly. It works in the demo and breaks in production for five reasons, all solved by the dedicated server:

Batching. Your ingestion code processes documents; your query code embeds one query at a time. Inline, each encode() is an independent forward, one press stroke per sheet. A dedicated server joins requests from different origins (ingestion + several concurrent queries) into a single batch and amortises the fixed cost. This is what really multiplies throughput, and you cannot do it if the model lives inside each isolated process.
Concurrency and backpressure. Under load, what happens when 10,000 requests arrive at once? Inline, you run out of memory or queue without control. TEI has --max-concurrent-requests: above that limit it rejects instead of degrading everyone, which is the correct way to manage overload (TEI CLI args).
Metrics. TEI exposes /metrics in Prometheus format: latencies, batch size, tokens/s, queued requests. Inline you have none of this without hand-instrumenting every encode(). To observe the data plane you need that endpoint.
A reusable HTTP contract. The same server is consumed by batch ingestion, by online query-time and by any other service that needs vectors. One copy of the model in memory, one version point, one place to change the model. Inline, each service loads its own copy and the version drifts on its own.
Lifecycle separation. The embedder is updated, restarted or scaled without touching the application. If tomorrow you swap bge-m3 for Snowflake Arctic, you change the server container, not the code of N services. (Remember the embedder drift trap: changing embedder forces a reindex; having a single server makes that change atomic and auditable.)

It is the same argument for why nobody serves an LLM with transformers.generate() in production: you serve it with a dedicated engine. The embedder is no different.

Dynamic batching: the mechanics of the press

Here is the performance engine. TEI does token-based dynamic batching: instead of processing requests one at a time or grouping them by a fixed number of requests, it groups them by token budget (TEI README, Discussion #367).

Two knobs rule:

--max-batch-tokens (the default depends on the build, typically 16384): the total tokens that fit in a batch. With max-batch-tokens=1000 you get 10 requests of 100 tokens, or one of 1000. The docs say it clearly: this number should be as large as possible until the model becomes compute-bound. It is the size of the press tray, measured in tokens, not in parts, because the real cost scales with tokens, not with number of texts.
--max-concurrent-requests (default 512): how many requests it admits in flight before rejecting. It is the backpressure control, not the batching one: it protects against surges by returning an error instead of queuing indefinitely.

The policy: TEI keeps an internal queue. When there is work, it fills a batch up to the max-batch-tokens budget (or until there are no more requests waiting) and sends it to the model in one stroke. Requests that arrive while the previous batch is being processed wait in the queue and go into the next one. Watch out for a real operational detail: if max-batch-tokens is smaller than the model’s maximum input length, you can trigger a loop where a long request never fits into a batch (Issue #723); the token budget has to be at least as large as the longest sequence you accept.

The number: throughput with batching vs without batching

Let us model the press. Let $C$ be the fixed cost of a forward pass (loading weights, launching kernels) and $c$ the marginal cost per sequence inside the batch. For a batch of size $B$, the stroke time is roughly:

$$t(B) \approx C + c \cdot B$$

as long as the batch fits in the hardware (memory-bandwidth-bound, which is the normal regime of a small encoder on GPU). The throughput, sequences per second, is:

$$\text{throughput}(B) = \frac{B}{t(B)} = \frac{B}{C + c \cdot B}$$

Let us put in concrete numbers, indicative but of the right order for bge-m3 (568M) on a GPU serving ~256-token sequences. Say $C = 4$ ms of fixed cost per stroke and $c = 0.5$ ms per marginal sequence.

Without batching ($B=1$):

$$t(1) = 4 + 0.5 = 4.5 \text{ ms} \quad\Rightarrow\quad \text{throughput} = \frac{1}{4.5\text{ms}} \approx 222 \text{ sec/s}$$

With batching ($B=32$):

$$t(32) = 4 + 0.5 \times 32 = 20 \text{ ms} \quad\Rightarrow\quad \text{throughput} = \frac{32}{20\text{ms}} = 1600 \text{ sec/s}$$

The batch of 32 multiplies throughput by ~7.2× compared with serving one at a time, because the fixed cost $C$, which at $B=1$ was 89% of the time, is spread across 32. And the asymptotic limit, as $B \to \infty$, is $1/c = 2000$ sec/s: the press cannot go faster than its marginal cost, and we are already at 80% of that ceiling with $B=32$. Going up to $B=128$ would give ~1969 sec/s, a marginal improvement in exchange for a good deal more latency and memory. Batching has diminishing returns: most of the gain is in going from 1 to a few dozen.

The latency trade-off is the other face. The request that arrived first in the batch of 32 does not see its response until the whole stroke finishes: it waits for the tray to fill plus the 20 ms of the forward. If the 32 requests arrived within a window of, say, 5 ms, the first request suffers ~5 ms of batching wait + 20 ms of compute = 25 ms of latency, against the 4.5 ms it would have had being served alone. The p99 latency gets worse, that is the price of the full tray, but in exchange the system absorbs 7× more load with the same silicon. That is why the correct configuration depends on the use case: in batch ingestion (no user waiting) you maximise max-batch-tokens and latency does not matter; online (user waiting) you bound the wait window to keep p99 under control even at the cost of some throughput.

Deployment: compose for embedder and reranker

TEI is deployed as a container. The key arguments: --model-id (the model from the Hub), --pooling (how it aggregates tokens into the vector: cls, mean, splade, last-token), --dtype (precision), and the batching knobs already seen. Two separate services, since the embedder and the reranker are different models and different endpoints, each with its own container.

Embedder (`bge-m3`) on GPU

# docker-compose: TEI serving bge-m3 as an embedder, on GPU
services:
 tei-embed:
 image: ghcr.io/huggingface/text-embeddings-inference:cuda-1.9
 command:
 - --model-id=BAAI/bge-m3
 - --pooling=cls
 - --dtype=float16
 - --max-batch-tokens=16384 # big tray: the press performs
 - --max-concurrent-requests=512 # backpressure: reject above this
 ports: ["8081:80"]
 volumes: ["./hf-cache:/data"]
 deploy:
 resources:
 reservations:
 devices: [{driver: nvidia, count: 1, capabilities: [gpu]}]
 # exposes /v1/embeddings (OpenAI), /embed, /embed_sparse, /metrics

Reranker (`bge-reranker-v2-m3`) on GPU

 tei-rerank:
 image: ghcr.io/huggingface/text-embeddings-inference:cuda-1.9
 command:
 - --model-id=BAAI/bge-reranker-v2-m3  # cross-encoder, ~568M
 - --dtype=float16
 - --max-batch-tokens=16384
 - --max-concurrent-requests=256
 ports: ["8082:80"]
 volumes: ["./hf-cache:/data"]
 deploy:
 resources:
 reservations:
 devices: [{driver: nvidia, count: 1, capabilities: [gpu]}]
 # exposes /rerank — invoked ONLY over top-k (20/50), not over hundreds

bge-reranker-v2-m3 is a ~568M cross-encoder built on bge-m3, multilingual, that takes (query, document) and emits a relevance score directly, without producing a reusable vector (model card, BGE docs). It carries no --pooling because it emits no embedding: it emits a scalar.

CPU for batch ingestion

For the CPU fleet, it is enough to change the image and the dtype; the HTTP contract is identical:

 tei-embed-cpu:
 image: ghcr.io/huggingface/text-embeddings-inference:cpu-1.6
 command:
 - --model-id=BAAI/bge-m3
 - --pooling=cls
 - --dtype=float16  # on CPU, ONNX int8 if the export supports it
 - --max-batch-tokens=8192 # smaller tray: the CPU saturates sooner
 ports: ["8083:80"]
 volumes: ["./hf-cache:/data"]
 # ONNX/MKL backend: exploits AVX-512+VNNI / AMX on 4th-gen Xeon

Any OpenAI client embeds against the endpoint without knowing what is behind it:

from openai import OpenAI
client = OpenAI(base_url="http://tei-embed:80/v1", api_key="-") # api_key ignored
r = client.embeddings.create(model="BAAI/bge-m3", input=["user query"])
vec = r.data[0].embedding # same contract as OpenAI; the silicon is invisible

And the rerank, through its native (non-OpenAI) endpoint:

import requests
docs = ["retrieved chunk 1", "retrieved chunk 2", "...top-20 from retrieval..."]
r = requests.post("http://tei-rerank:80/rerank",
 json={"query": "user query", "texts": docs})
ranked = r.json() # [{index, score}, ...] ordered by relevance

Integration: how ingestion and the gateway consume it

The TEI server has two clients with opposite profiles, and that is precisely the reason to centralise it:

Ingestion (batch, no SLA) calls /v1/embeddings or /embed with large lots of chunks. Here throughput is what matters: ingestion pushes thousands of texts and TEI groups them into large batches against the high max-batch-tokens. It is the case of the press with a full tray. Ingestion then writes the vectors (dense + sparse) into the vector DB; the specific PDF-to-indexed-chunk pipeline is the sister piece of this series in preparation, and meanwhile the PostgreSQL + Qdrant schema in microservices covers the structure of the pipeline.

The gateway (query-time, online) consumes TEI through two different routes:

Embedding the query: a single call to /v1/embeddings with a short text. The gateway, LiteLLM or another L7 router, treats it as one more OpenAI embeddings provider, routed by model. That embedded query goes to the vector DB for the hybrid search.
Reranking the top-k: after retrieving 20-50 candidates from the retriever, the gateway calls /rerank on the reranker server with the query and the candidate list, and keeps the best ones. Since /rerank is not an OpenAI standard, LiteLLM exposes it through its own reranking contract (LiteLLM rerank with TEI).

The full query-time pattern: embed query (TEI dense) → search (vector DB, hybrid retrieval) → rerank top-k (TEI rerank) → send query + top-5 to the LLM. Two different TEI endpoints, one for dense and another for rerank, both behind the gateway, both swappable.

Sizing: memory, replicas and latency budget

Footprint of a 568M encoder

The weight footprint of bge-m3 (568M params) depends only on precision:

$$\text{fp16: } 568 \times 10^6 \times 2 \text{ B} \approx 1.14 \text{ GB} \qquad \text{int8: } 568 \times 10^6 \times 1 \text{ B} \approx 0.57 \text{ GB}$$

On top of this comes the space for the batch’s temporary activations (proportional to max-batch-tokens × hidden dimension) and, in TEI, the FlashAttention/cuda graphs structures. In practice, a bge-m3 in fp16 is served comfortably in ~6 GB of VRAM even at high batch, and in int8 on CPU the whole model (0.57 GB) fits in the cache and RAM of any server without blinking, activating the AMX/VNNI paths that make it viable, which is exactly the argument of RAG on the CPU.

How many replicas fit? If we reserve a 10 GB slice of an H100 for serving embeddings online, with an effective footprint of ~2.5 GB per replica (fp16 weights + activations + overhead), around 3-4 replicas of bge-m3 fit in that slice, splitting the QPS between them. In int8 the footprint drops and more fit, at the cost of a full point of quality. For the reranker, the weight footprint is the same (568M), but its batch pattern is different (query-doc pairs), so it is worth sizing it separately.

Latency budget: short query vs top-50 rerank

Here is the fundamental asymmetry between embedder and reranker, and it is what decides the sizing.

Embedding a short query (online): it is one forward over a text of tens of tokens. On GPU, inside a dynamic batch, it is tens of ms including the batching wait, the order of the earlier numerical example, ~5-25 ms. It is cheap and constant: a query is a query, no matter how many documents are in the corpus. That is why embedding online fits easily into an interactive latency budget.

Reranking the top-50 (online): a cross-encoder does not produce a reusable vector; it evaluates every pair (query, document) in a forward. The cost is linear in the number of candidates:

$$\text{cost}_{\text{rerank}} \propto k \times \text{forward}(\text{query} + \text{doc})$$

Reranking a top-50 is 50 forwards of (query + doc) pairs, where each pair is longer than the query alone (query + ~256-token chunk). TEI groups them into batches, and batching helps again there, but the total work is ~50× that of embedding one query. If embedding a query costs of the order of 10-20 ms, reranking 50 candidates costs of the order of hundreds of ms depending on the batch and the hardware. Hence the operational rule repeated throughout the series: broad, cheap recall in the retriever, precision rerank over FEW candidates. Reranking a top-20/50 fits in the online budget; reranking hundreds at high QPS does not, and that is where the reranker demands a dedicated GPU or a larger slice. The detail of why the cross-encoder is expensive but precise is in reranker and hybrid retrieval.

The query-time latency budget, added up:

Stage	Typical cost (order)	Scales with
Embed query (TEI dense)	~10-20 ms	constant (1 short text)
Hybrid search (vector DB, HNSW+sparse)	single-digit ms	index size (sublinear)
Rerank top-k (TEI cross-encoder)	~100-300 ms	linear in k
Generation (LLM, outside TEI)	seconds	output tokens

The reranker is the most expensive stage of the data plane, and the only one that scales badly with the width of retrieval. Size it as the critical component.

Applied to the generic 4×H100 cluster

Let us come down to the cluster of the series: 4×H100 SXM 80 GB plus a generic CPU fleet (Xeon with AMX, NUCs). TEI lives in two places depending on the load profile, and the identical OpenAI contract is what makes each place swappable.

TEI-CPU on the fleet → batch ingestion. Re-embedding the corpus is throughput-bound work with no latency SLA: its place is the CPU fleet with bge-m3 in int8 (ONNX/AMX). Here you maximise max-batch-tokens because nobody is waiting, and the press performs with the tray brimming. No H100 should spend a cycle reindexing a corpus that changes once a day, which is exactly the bad split that RAG on the CPU takes apart. The CPU fleet scales horizontally and cheaply; the corpus is chopped up and spread across nodes.

TEI-GPU in a MIG slice → high-QPS online embeddings and rerank. Query-time has a latency SLA and, if the system serves many requests per second, it needs the GPU’s throughput and low latency. But a 568M encoder does not deserve a whole H100: it takes ~2.5 GB effective, and leaving it 80 GB wastes 97% of the card. The right answer is a MIG slice: partition an H100 into hardware-isolated instances and give TEI one of the small slices (a 1g.10gb, for example), leaving the large slices for generation or for other tenants. The online embedder and reranker live there, with memory and compute isolation guaranteed by the partitioning. The how of the partitioning, MIG, MPS, time-slicing and when to use each, is in sharing a GPU; the reading for TEI is direct: MIG gives the hard isolation that an online service with bounded latency wants, while batch ingestion on CPU does not even touch the GPU.

The split, in one sentence: ingestion squeezes throughput/€ on CPU with a full tray; online squeezes latency/QPS in a MIG slice of an H100; both speak the same /v1/embeddings and /rerank, so moving load from one side to the other is changing a URL in the gateway.

If at some point you need a 7B embedder (gte-Qwen2, NV-Embed) whose quality bge-m3 cannot reach, that is no longer small-encoder work: it drags an LLM’s cost profile with it and is served where 7Bs are served, with vLLM --task embed in a large GPU slice, not with TEI-CPU. But it is the occasional exception, not the base load.

Conclusion

TEI is the stamping press of the RAG data plane: a specialised server that only stamps vectors and scores, and that performs far better when it fills the tray before pressing. The technical piece that gives the performance is token-based dynamic batching, controlled by max-batch-tokens and max-concurrent-requests, with a clear trade-off between throughput and p99 latency that is resolved differently in ingestion (tray brimming, no SLA) and online (bounded window, p99 protected). The architectural virtue is the OpenAI-compatible contract on /v1/embeddings, which makes the embedder swappable without touching code, plus the native /embed_sparse and /rerank endpoints for the sparse head of bge-m3 and for the bge-reranker-v2-m3 cross-encoder. The numbers drive the sizing: a 568M encoder fits in gigabytes, not tens of gigabytes, so several replicas live in a MIG slice; embedding a query is cheap and constant, but reranking is linear in candidates and is the critical stage of the latency budget. On the 4×H100 cluster, the right split is TEI-CPU for batch ingestion and TEI-GPU in a MIG slice for high-QPS online: two presses, two silicons, one contract.

References

Hugging Face — Text Embeddings Inference (TEI), repo y docs (Rust, backends CPU ONNX/MKL y CUDA, endpoints OpenAI-compatibles). https://github.com/huggingface/text-embeddings-inference · https://huggingface.co/docs/text-embeddings-inference
Hugging Face — TEI Quick Tour (endpoints /embed, /embed_sparse, /rerank, /predict, /v1/embeddings). https://huggingface.co/docs/text-embeddings-inference/quick_tour
Hugging Face — TEI CLI arguments (--model-id, --pooling, --dtype, --max-batch-tokens, --max-concurrent-requests). https://github.com/huggingface/text-embeddings-inference/blob/main/docs/source/en/cli_arguments.md
Hugging Face — TEI README / Discussion #367 (token-based dynamic batching; semántica de max-batch-tokens). https://github.com/huggingface/text-embeddings-inference/blob/main/README.md · https://github.com/huggingface/text-embeddings-inference/discussions/367
BAAI — BGE-M3 model card (568M, XLM-RoBERTa, dense+sparse+colbert, MIT). https://huggingface.co/BAAI/bge-m3
BAAI — bge-reranker-v2-m3 model card (cross-encoder ~568M sobre bge-m3, multilingüe). https://huggingface.co/BAAI/bge-reranker-v2-m3 · https://bge-model.com/bge/bge_reranker_v2.html
Chen, J., et al. BGE M3-Embedding. arXiv:2402.03216. https://arxiv.org/abs/2402.03216
BerriAI — LiteLLM: proveedor TEI para /rerank. https://github.com/BerriAI/litellm/issues/8372

End-to-end document ingestion: from PDF to indexed chunk

Thu, 11 Jun 2026 09:00:00 +0000

Fourth piece in an operational series about squeezing a generic on-premise LLM cluster of 4×H100 SXM 80 GB. Its siblings in this batch: serving embeddings and rerankers with TEI in production details the inference piece that this ingestion feeds; GitOps for the inference stack with Flux versions and deploys this whole pipeline; and hardening and secrets for the sovereign stack protects the corpus and the credentials this ingestion touches. The final consumer of what we build here, an end-to-end sovereign assistant, is assembled in a later instalment.

TL;DR

A RAG system inherits the quality of its corpus, and the corpus inherits the quality of the ingestion that manufactured it. This is the garbage-in/garbage-out of retrieval: a chunk badly extracted from a table, a page header repeated a thousand times, a scanned PDF from which you only got OCR noise; all of it gets embedded, indexed and reappears as poisoned context in the model’s answer. Document ingestion is not an afternoon’s script, it is a six-stage pipeline with engineering decisions at every one: (1) extract/parse, where almost everything is decided: layout-aware (Docling, which with Granite-Docling-258M preserves tables, formulas and structure, and which according to IBM avoids classic OCR and is up to ~30× faster) against plain text extraction (PyMuPDF), with OCR or a VLM for scans; (2) clean/normalise, removing boilerplate, normalising Unicode, rebuilding broken paragraphs; (3) chunk, and here 2026 has shifted the consensus: a February 2026 benchmark put recursive at 512 tokens in the lead (69% accuracy) ahead of semantic chunking (54%), and late chunking adds global context at no extra storage cost; (4) enrich metadata, with source, page, section, timestamp, ACL/tenant: for filtered retrieval, citation and auditability; (5) embed, through an embeddings server such as TEI; (6) index, in pgvector or Qdrant, with payload, dense + sparse. In between, exact deduplication (hash) and near-duplicate detection (MinHash/LSH or cosine with a threshold), because near-duplicates degrade retrieval recall and diversity. The numbers matter: a corpus of $N_{docs}\times$ pages $\times$ tokens/page gives the total tokens and, at a given CPU throughput, the ingestion time; and $N_{chunks}\times d\times\text{bytes}$ gives the index size, which in int8 falls 4× against fp32. The reading for the 4×H100: ingestion is CPU work, which links to the RAG-on-CPU piece; the GPU only comes in if you parse with a VLM (Granite-Docling) or use a 7B embedder.

The analogy: an archive’s cataloguing chain

Picture the cataloguing department of a large document archive. It is not one person putting papers into boxes; it is a chain of workstations, each with a different craft and its own quality standard. A document goes in raw at one end and comes out the other turned into a record you can find in seconds. If one station does its job badly, the ones downstream inherit the error and amplify it.

The first station is intake and reading. A heterogeneous box arrives: typed reports, crooked photocopies, folded tables, microfiche. An expert archivist really reads each item: distinguishes the body of the text from the margins, rebuilds a table that spans two pages, transcribes by hand what the scanner could not read. A novice archivist, by contrast, photocopies everything flat and hands over a slab of text where the table columns are interleaved into nonsense. That is exactly the difference between layout-aware parsing and plain text extraction, and it is where almost all of the quality is won or lost.

The second station is weeding out. Before filing anything, someone removes the duplicates (three copies of the same memo, two near-identical versions of a report) and cleans off the useless marks: “COPY” stamps, footers repeated on every sheet, coffee stains. If you do not weed, the archive fills up with copies that, when someone searches, return the same document six times and bury everything else. This is deduplication and cleaning.

The third station cuts it into records. A 400-page book is not catalogued as one giant record; it is broken down into manageable entries, by chapter, by section, of a size a reader can take in at a glance. Too big and the record mixes topics; too small and it loses context. This is chunking, and the size of the record is the decision that most shapes retrieval.

The fourth station labels. Each record carries a shelf mark, a date, its provenance, an access level (can anyone see this, or only the legal department?). Without those labels you cannot filter a search or say which document a statement came from. These are the metadata: source, page, section, timestamp, ACL.

The fifth and sixth stations place the record on the indexed shelf: they translate it into the catalogue’s language, a vector, and put it in the right drawer of the filing cabinet, so that a query finds the related records without walking through the whole thing. This is embedding and indexing in the vector store.

The moral runs through the whole post: RAG cannot retrieve better than what the cataloguing chain filed. If the first station chopped up a table badly, no reranker will fix it afterwards. Ingestion is the quality station for the entire system, and almost all of it, like the archive full of patient people working with nobody timing them, is background work that fits on CPU.

The six-stage pipeline

The temptation to treat ingestion as “read the PDF and chop it up” is the source of 80% of mediocre RAG systems. Each stage has a quality standard and a characteristic failure, and failures cascade downstream: if you parse badly, cleaning does not recover what was lost; if you chunk badly, embedding freezes the error into the vector; if you do not label, you will not be able to filter or cite. The rest of the post walks through the six stations and their decisions.

Stage 1 — Parsing: layout-aware vs plain text

Almost everything is won or lost here. A PDF is not text: it is a set of instructions for drawing glyphs on a canvas. “Extracting the text” from a PDF means reconstructing a reading order that the format does not guarantee, and tables, columns and figures break it systematically.

There are two philosophies, and the choice shapes the rest of the pipeline.

Plain text extraction. Tools such as PyMuPDF (fast, robust, no heavy dependencies) read the PDF’s text stream and dump it out. For single-column documents, running prose and no tables, it is perfect: extremely fast, faithful and cheap on CPU. Its limit shows up with structure: a two-column table comes out with the cells interleaved, a double-column document mixes the end of one with the start of the other, and a scanned invoice does not come out at all because there is no text layer. PyMuPDF can also segment by the table of contents (TOC) when the PDF carries one, which helps with chunking by section (Omdena, Document Parsing for RAG: A Complete Guide for 2026).

Layout-aware parsing. Tools such as Docling (an open source project driven by IBM Research) and unstructured.io first understand the layout, identifying titles, paragraphs, tables, figures, lists and formulas, and only then extract the content while respecting that structure. Docling captures table structure (rows, columns, multi-level headers) and, in its 2026 evolution, does so with a VLM: Granite-Docling-258M, released by IBM in January 2026 under Apache 2.0, a compact vision-language model (~258M parameters, Granite 3 backbone + SigLIP2 visual encoder) that converts pages, whether PDF, slides or scans, directly into a structured format called DocTags, preserving tables, code, inline and block mathematics, and the document’s hierarchy (IBM, Granite-Docling: End-to-end document understanding; model card on Hugging Face; Docling repo). IBM states that the VLM route avoids classic OCR and that this “reduces errors and speeds up the solution by up to 30×” against a traditional OCR pipeline. That is an IBM Research figure; I quote it and treat it as indicative, not as an independent benchmark reproduced here.

unstructured.io offers partitioning strategies graded by document complexity: fast (plain text, quick), hi_res (identifies the layout, recommended when classifying tables and elements properly matters), VLM and auto, balancing speed, cost and accuracy (Unstructured, PDF Parsing Strategies for RAG). The practical rule: fast for prose, hi_res or VLM for documents with tables and structure.

OCR for scans

When the document has no text layer (a scan, a photo, a digitised microfiche), you have to recognise the characters. Three routes:

Classic OCR (Tesseract, PaddleOCR, EasyOCR, which is what Docling integrates with when explicit OCR is needed). Mature, CPU-friendly, good with clean text; it struggles with tables, handwriting and complex layouts.
End-to-end VLM (Granite-Docling and similar). The model “looks” at the page and emits structure directly, with no separate OCR stage. Better with complex layouts; this is where the GPU does come in if the VLM is large or the volume is high.
Hybrid: OCR for character transcription, a layout model for the structure.

The honest criterion: for a corpus of native PDFs with text, PyMuPDF or unstructured fast handle it on CPU and cheaply. For a corpus with dense tables, forms or scans, layout-aware Docling/Granite-Docling earns its cost in chunk quality, and it is the only stage of the pipeline where the GPU can be justified.

Document case	Recommended tool	Silicon
Native PDF, single column, prose	PyMuPDF / unstructured `fast`	CPU
PDF with tables, double column, hierarchy	Docling / unstructured `hi_res`	CPU (layout models)
Scan, form, handwriting, complex layout	Granite-Docling (VLM) or OCR+layout	GPU if large VLM / high volume
HTML, DOCX, PPTX	Docling (multi-format) / native parsers	CPU

Stage 2 — Cleaning, normalising and deduplicating

Freshly parsed text comes in dirty. Cleaning means removing what adds nothing and normalising what is represented in a thousand ways:

Boilerplate: headers and footers repeated on every sheet, page numbers, watermarks, HTML navigation menus, cookie banners. If you do not remove them, they get embedded a thousand times over and contaminate both the index and the answers.
Unicode normalisation (NFC/NFKC), whitespace and dashes: the same character represented in several ways breaks exact matching and dirties the embeddings.
Paragraph reconstruction: undoing the hard line breaks the PDF inserted mid-sentence, without merging paragraphs that were meant to stay separate.

Deduplication: why it matters

RAG suffers two ills from duplicates. The exact kind, the same document uploaded three times, inflates the index and makes a search return the same answer repeatedly, wasting the top-k context slots. The near-duplicate kind, two near-identical versions of a report, a document and its draft, is worse: they look different to a hash but say the same thing, and they degrade retrieval recall and diversity. This is not theory: in the MS MARCO V2 collection a substantial overlap of near-duplicates has been documented which, left untreated, degrades retrieval precision and reduces the diversity of retrieved documents (Ragnarök / TREC RAG 2024).

Two levels of dedup, complementary:

Exact (hash). Compute a sha256 of the normalised content of each document (or chunk) and discard the matches. Cost $O(N)$, trivial. It catches byte-for-byte duplicates.
Near-dup (MinHash + LSH, or cosine of embeddings with a threshold). For those that differ slightly but mean the same thing. MinHash compresses each document into a compact signature such that the probability of two signatures matching at a position equals the Jaccard similarity of the original shingle sets; combined with Locality-Sensitive Hashing (LSH) it finds every near-duplicate pair without comparing all against all, reducing a quadratic problem to almost linear (Brenndoerfer, MinHash, Jaccard, LSH). It is the dominant technique in cleaning LLM training corpora (C4 and RefinedWeb use it) and applies just the same to a RAG corpus (Zilliz, Data Deduplication at Trillion Scale). The alternative, cosine of embeddings with a threshold (say > 0.97), catches semantic duplicates that MinHash cannot see (paraphrasing), but it requires you to already have the embeddings and is more expensive.

The practical rule: exact hash always (it is free); MinHash/LSH for large corpora with versions; cosine with a threshold if paraphrasing is a real problem. And deduplicate before embedding: re-embedding a duplicate means spending compute on rubbish you will then have to filter out.

Stage 3 — Chunking

Chunking is the decision that most shapes retrieval, and the one that carries the most myths. The trade-off is threefold: chunk size ↔ retrieval granularity ↔ context cost.

Large chunks: each one holds more context and there is less risk of cutting an idea in half, but the search is less precise (one vector represents too many topics) and, on retrieval, you put more tokens into the LLM’s prompt, which means more cost and more risk of diluting what is relevant.
Small chunks: very granular, precise retrieval, but each chunk loses context (a 43-token fragment may mean nothing outside its section) and you need to retrieve more of them to cover an answer.

The strategies, from least to most sophisticated:

Fixed size + overlap. Cut every $N$ tokens with an overlap of $k$ tokens between consecutive chunks so as not to slice a sentence dead. Simple, predictable, a reasonable baseline. The overlap is the insurance against cutting an idea right at the boundary.
Recursive (LangChain’s RecursiveCharacterTextSplitter). It tries to cut on separators in priority order (paragraph, then sentence, then word) to respect the natural structure as far as possible before falling back to a hard cut. It is the workhorse.
Semantic. Embed sentences and cut where the similarity between consecutive sentences falls below a threshold, grouping by coherence of meaning. It sounds better on paper; in 2026 practice it has disappointed (see below).
Structure/layout-aware (by headings). It exploits the hierarchy the layout-aware parser has already extracted: one chunk per section or subsection. unstructured offers the by_title strategy, which opens a new chunk when a title element appears, avoiding mixing text from different sections (Unstructured docs). If you parsed with Docling/hi_res, this strategy is almost free and is usually the best one for well-structured documents.
Late chunking. The twist of 2024–2026: instead of chunking and then embedding each chunk separately, embed the whole document first (with a long-context encoder) and then apply the chunk boundaries by mean-pooling the token embeddings inside each span. The result: every chunk keeps the document’s global context (a pronoun or a reference that only makes sense given the previous paragraph is encoded in the vector) and all of it with no extra storage cost, because you end up with one vector per chunk just as before (Jina AI, Late Chunking; arXiv:2409.04701).

What the 2026 benchmarks say (and why semantic disappoints)

It is worth being sceptical about fashion. A Vecta benchmark from February 2026 covering 7 strategies over 50 academic papers put recursive at 512 tokens in first place with 69% accuracy, while semantic chunking came in at 54%, partly because it produced tiny fragments, averaging 43 tokens, too small to mean anything (Firecrawl, Best Chunking Strategies for RAG in 2026). A systematic analysis from January 2026 also identified a “context cliff” around 2,500 tokens, where answer quality drops once you feed in contexts that are too long, an extra argument against giant chunks ([ibid.]). The honest reading: recursive at a moderate size with overlap remains the baseline that is hard to beat; late chunking is the improvement with the best cost/benefit ratio when the model supports it; semantic promises more than it delivers.

A worked numerical example of chunking

Take a technical document of 30 pages, with ~500 tokens of useful prose per page after cleaning (tables and figures are chunked separately). That is $30 \times 500 = 15{,}000$ tokens of text. We chunk with recursive at 512 tokens and 20% overlap ($0.20 \times 512 \approx 102$ tokens). The effective “step” between the start of one chunk and the next is:

$$\text{step} = \text{size} - \text{overlap} = 512 - 102 = 410 \text{ tokens}$$

The number of chunks in the document is then, approximately:

$$N_{chunks} \approx \left\lceil \frac{T_{doc} - \text{overlap}}{\text{step}} \right\rceil = \left\lceil \frac{15{,}000 - 102}{410} \right\rceil \approx \lceil 36.3 \rceil = 37 \text{ chunks}$$

Without overlap it would have been $\lceil 15{,}000 / 512 \rceil = 30$ chunks; the 20% overlap costs us 7 extra chunks (~23% more) in exchange for not splitting ideas at the boundaries. That is the concrete price of overlap: more vectors to embed, index and store, in exchange for robustness in retrieval. We will use this factor when sizing the full corpus.

Stage 4 — Enriching with metadata

A chunk with no metadata is a record with no shelf mark: it exists but it is no use. Each chunk gets a payload attached with, at a minimum:

Source and location: document_id, document name/URI, page number, section/heading (which the layout-aware parser already gave you). Essential for citation: being able to say “this comes from document X, page 12, section 3.2” is what separates an auditable RAG from one that hallucinates with no traceability.
Timestamp: when it was ingested and the document’s date. It lets you filter by freshness and detect stale content.
ACL / tenant: who is allowed to see this chunk. It is critical and it is applied as a filter at retrieval time: a user from department A must not retrieve chunks marked for B only. Without this, the RAG system is a data leak waiting to happen.
Embedding model version (model_version): so you know which embedder generated each vector and can migrate without mixing incompatible spaces.

These metadata are not decoration: they enable filtered retrieval (searching only in what the user can see, or only in documents after a given date), citation (reconstructing the origin of every statement) and auditability (knowing what was retrieved, from where and when). All of it lives in the point’s payload in the vector store.

Stages 5 and 6 — Embedding and indexing

The last two stations translate the chunk into a vector and place it on the shelf.

Embedding. The chunks are sent in batches to an embeddings server, typically Hugging Face’s TEI (Text Embeddings Inference), which exposes the OpenAI /v1/embeddings contract and runs on CPU or GPU. It is throughput-bound work with no latency SLA: its natural home is the CPU with a small encoder in int8 (the sibling piece in this batch, serving embeddings and rerankers with TEI, details the how). It is worth emitting dense + sparse at the same time: the dense vector captures semantics, the sparse one (SPLADE/BM25-like) exact lexical overlap, and together they make hybrid retrieval more robust.

Indexing. The vectors, with their payload, are upserted into the vector store. Two reference options, both current in 2026:

pgvector (a PostgreSQL extension). Its great virtue is living inside Postgres: ACID transactions, joins with the relational metadata, a single database to operate. Version 0.8 added halfvec (half-precision vectors, 2× less storage) and 0.9 (early 2026) added support for sparse vectors and speed improvements. Its known limit: it has no native int8 quantisation, so high-dimensional embeddings consume RAM linearly (Encore, pgvector vs Qdrant 2026; Katz, Scalar and binary quantization for pgvector).
Qdrant (a dedicated vector engine). It supports int8 scalar quantisation (float32 → int8, 4× less memory) and product quantisation, native sparse vectors and RRF fusion for hybrid search. It is more efficient in memory and in quantisation; the cost is operating one more system alongside Postgres (Markaicode, pgvector vs Qdrant 2026).

The practical rule: pgvector if you already have Postgres and the corpus fits comfortably in RAM (a single database to operate and back up); Qdrant if memory efficiency and int8 quantisation are critical because of the corpus size. Synchronisation between the relational truth (Postgres) and the index (Qdrant) when both are used is detailed in PostgreSQL + Qdrant in microservices.

A cross-cutting stage — Incremental ingestion and idempotency

A living corpus changes: documents are added, edited, deleted. Re-indexing everything each night is expensive and causes windows of unavailability. The alternative is incremental ingestion with two pillars:

Idempotency by doc-id + hash. Each chunk is identified deterministically ({doc_id}_{chunk_index}) and each document carries a content hash. On reprocessing, if the hash has not changed, nothing is re-embedded: the compute is saved. If it changed, the old chunks for that doc_id are deleted and the new ones are upserted. An upsert with a deterministic id is idempotent: reprocessing an event twice does not create duplicates.
CDC (Change Data Capture). Instead of polling, Debezium reads the PostgreSQL WAL and propagates additions, edits and deletions to the index in near real time. A deletion in Postgres triggers the deletion of that document’s chunks in the vector store, avoiding the “ghost documents” that contaminate retrieval. The deep dive is in PostgreSQL + Qdrant in microservices and in the one on Debezium and CDC.

The arithmetic: sizing the corpus and the index

Two calculations you need to be able to do before provisioning anything.

Corpus sizing and ingestion time

Suppose a corporate corpus of $N_{docs} = 50{,}000$ documents, averaging 20 pages and 500 useful tokens per page after cleaning. The total tokens in the corpus:

$$T_{corpus} = N_{docs} \times \text{pages} \times \text{tokens/page} = 50{,}000 \times 20 \times 500 = 5 \times 10^{8} \text{ tokens}$$

Five hundred million tokens. Applying the overlap factor from the chunking example (~1.23×, the 23% extra chunks from the 20% overlap) and an effective size of 512 tokens/chunk, the number of chunks is:

$$N_{chunks} \approx \frac{T_{corpus}}{\text{step}} = \frac{5 \times 10^{8}}{410} \approx 1.22 \times 10^{6} \text{ chunks}$$

That is, ~1.22 million chunks to embed. At a conservative CPU embedding throughput of 3,000 tok/s per Xeon server in int8, the same figure we use in the RAG-on-CPU piece, the ingestion time for the first full corpus on one box is:

$$t_{ingesta} = \frac{T_{corpus}}{\text{throughput}} = \frac{5 \times 10^{8}}{3{,}000} \approx 1.67 \times 10^{5} \text{ s} \approx 46 \text{ hours}$$

46 hours on a single box sounds bad, but ingestion is embarrassingly parallel: the corpus is shared out. With 8 CPU servers it drops to ~6 hours, comfortably within a weekend window for the initial load; and subsequent incremental ingestions (only what changed) take minutes. Layout-aware parsing adds its own cost, since Docling with a VLM is slower than PyMuPDF, but it is also batch and parallelises the same way.

Vector index size

Each vector has dimension $d = 1024$ (that of bge-m3). In float32 (4 bytes/dimension), each vector takes:

$$\text{bytes}_{fp32} = d \times 4 = 1024 \times 4 = 4096 \text{ B} = 4 \text{ KB}$$

For the 1.22 M chunks, the dense vectors alone in fp32:

$$\text{size}_{fp32} = N_{chunks} \times d \times 4 = 1.22 \times 10^{6} \times 4096 \text{ B} \approx 5.0 \text{ GB}$$

In int8 (1 byte/dimension), each vector takes 1 KB and the total falls 4×:

$$\text{size}_{int8} = N_{chunks} \times d \times 1 = 1.22 \times 10^{6} \times 1024 \text{ B} \approx 1.25 \text{ GB}$$

To this you must add the HNSW index (~1.2× the size of the vectors for $m=16$) and the payload (metadata + chunk text, ~500 B/chunk → ~0.6 GB). In round numbers:

Configuration	Vectors	HNSW (~1.2×)	Payload	Total
fp32	~5.0 GB	~6.0 GB	~0.6 GB	~11.6 GB
int8	~1.25 GB	~1.5 GB	~0.6 GB	~3.4 GB

The reading: a corpus of 50,000 documents fits in the RAM of a single node even in fp32, and in int8 (Qdrant) it fits with room to spare, which keeps search latency in single-digit milliseconds. int8 quantisation is almost always the balance point, saving 4× with a recall loss typically below 1%. (These numbers are order-of-magnitude, with the constants and assumptions stated; they are for sizing, not for pinning down an invoice.)

Applied to the generic 4×H100 cluster

Let us bring this down to the series’ cluster: 4×H100 SXM 80 GB plus a generic CPU fleet (Xeon with AMX, NUCs). The correct split for ingestion is almost all CPU, with the GPU as an occasional exception:

Cleaning, dedup, chunking, metadata, ingestion embedding and indexing → the CPU fleet. All of this is batch work, throughput-bound, with no latency SLA. It is exactly the “data plane” the RAG on CPU piece talks about: no H100 should spend a cycle chunking documents or building an HNSW index (which was always CPU by design). The nightly incremental re-indexing of a corpus that changes on the scale of hours is the textbook case of “CPU work in no hurry”.
The GPU only comes in at two points. First, in VLM parsing: if the corpus has scans, dense tables or forms and you choose Granite-Docling-258M or another vision-language model, that parsing can be accelerated on GPU, although at 258M parameters it is light and, at moderate volumes, runs on CPU without drama. Second, in the large embedder: if retrieval quality demands a 7B embedder (gte-Qwen2, NV-Embed) instead of bge-m3 (568M), that embedder is once again an LLM-class model and lives where the 7B models live, on the GPU.
The 4×H100 are reserved for generation. As throughout the series, the expensive and scarce silicon is kept for what is latency-bound, the LLM producing the answer, and at most for the peaks of VLM parsing or 7B embedding the CPU cannot absorb. To give a sense of the ceiling: a 4×H100 node serving bge-m3 through TEI runs at around ~2,000 chunks/s, against the thousands of tok/s of a Xeon in int8; but using the H100s for daily ingestion means spending the resource the whole organisation fights over on a job the CPU fleet does overnight without anyone missing it.

The sentence that sums up the split: ingestion is the archive’s cataloguing chain, and almost all of it is done by patient, cheap staff (CPU); the star writer (GPU) is only disturbed when there is a page no classic OCR can decipher.

Closing: quality is decided upstream

The recurring error in mediocre RAG is not in the reranker or the prompt: it is in an ingestion that parsed a table badly, did not weed out the duplicates, or chunked at a size that destroys context. Garbage-in, garbage-out: no downstream component fixes what ingestion spoiled upstream. Investing in the cataloguing chain, with layout-aware parsing where it is needed, real dedup, measured chunking, complete metadata, is what moves the needle on system quality the most, and almost all of it fits on the CPU fleet. The GPU, like the star writer, should only touch the corpus when its brain is genuinely required.

References

IBM — Granite-Docling: End-to-end document understanding (Granite-Docling-258M, Apache 2.0, DocTags, enero 2026). https://www.ibm.com/new/announcements/granite-docling-end-to-end-document-conversion
ibm-granite — granite-docling-258M model card (VLM ~258M, Granite 3 + SigLIP2). https://huggingface.co/ibm-granite/granite-docling-258M
Docling project — Docling: Get your documents ready for gen AI (layout, tablas, multi-formato). https://github.com/docling-project/docling
Unstructured — PDF Parsing Strategies for RAG (fast / hi_res / VLM / auto; chunking by_title). https://unstructured.io/blog/mastering-pdf-transformation-strategies-with-unstructured-part-2 · https://docs.unstructured.io/
Omdena — Document Parsing for RAG: A Complete Guide for 2026 (PyMuPDF, TOC, pipeline). https://www.omdena.com/blog/document-parsing-for-rag
Firecrawl — Best Chunking Strategies for RAG (and LLMs) in 2026 (benchmark Vecta feb-2026: recursive 512 → 69%, semántico 54%; context cliff ~2.500 tok). https://www.firecrawl.dev/blog/best-chunking-strategies-rag
Günther, M., et al. — Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models. arXiv 2409.04701. https://arxiv.org/abs/2409.04701 · Jina AI: https://jina.ai/news/late-chunking-in-long-context-embedding-models/
Brenndoerfer, M. — MinHash: Jaccard Similarity, LSH, and Near-Duplicate Detection. https://mbrenndoerfer.com/writing/minhash-algorithm-jaccard-similarity-lsh-deduplication
Zilliz — Data Deduplication at Trillion Scale (MinHash LSH dominante en limpieza de corpus). https://zilliz.com/blog/data-deduplication-at-trillion-scale-solve-the-biggest-bottleneck-of-llm-training
Ragnarök / TREC RAG 2024 — near-duplicados en MS MARCO V2 degradan recuperación y diversidad. https://arxiv.org/pdf/2406.16828
Encore — pgvector vs Qdrant in 2026 (halfvec, sparse, límites de cuantización de pgvector). https://encore.dev/articles/pgvector-vs-qdrant
Markaicode — pgvector vs Qdrant (2026 Benchmarks) (Qdrant int8 scalar quantization 4×). https://markaicode.com/vs/pgvector-vs-qdrant/
Katz, J. — Scalar and binary quantization for pgvector. https://jkatz05.com/post/postgres/pgvector-scalar-binary-quantization/
Hugging Face — Text Embeddings Inference (TEI), backends CPU/GPU, endpoints OpenAI-compatibles. https://github.com/huggingface/text-embeddings-inference

Taking RAG to the CPU: separating the data plane from the generation plane

Thu, 11 Jun 2026 03:20:00 +0000

Third piece in an operational series about squeezing a generic on-premise LLM cluster of 4×H100 SXM 80 GB. Its siblings: sharing one GPU between workloads (time-slicing, MPS, MIG) and serving several models on one GPU (swap + sleep) attack the split inside the GPU. This one attacks the split outside: which parts of RAG never need to touch the GPU at all. The closing piece of the series, the end-to-end sovereign assistant (fourth instalment, in preparation), assembles the complete system where these pieces fit together.

TL;DR

A RAG system is not one thing, it is three phases with opposing compute profiles, and putting them all on the GPU “because it is AI” is an allocation mistake. (1) Build/ingest, embedding the corpus and building the index, is batch, throughput-bound work with no latency SLA: its natural home is the CPU. (2) Query-time retrieval, embedding the query, HNSW search, RRF fusion, lightweight rerank, is mostly CPU, with caveats only for heavy reranking; vector search always ran on CPU, even in stacks sold as “GPU”. (3) Generation, the LLM producing the answer, is latency-bound, and there the GPU is irreplaceable: a 7B on CPU gives a time to first token of seconds, unacceptable for chat. The technical key to why (1) and (2) fit on CPU: the embedder is not an LLM. bge-m3 is around 568M parameters (an XLM-RoBERTa encoder), not 7B+; in int8 it occupies about 580 MB and activates fast integer compute paths (Intel AVX-512 + VNNI + AMX on 4th-gen Xeon and later; NEON SDOT/UDOT on ARM). The runtimes are ready: TEI with a CPU backend (same OpenAI API /v1/embeddings and /rerank), Qdrant’s fastembed (ONNX-CPU), bge-m3 in ONNX int8 with its three heads (dense/sparse/ColBERT). The Intel + Hugging Face blog post with Optimum Intel and fastRAG reports up to around 10× on indexing for BGE-large int8 on a 4th-gen Xeon (their benchmark figure, encoding-only; I quote it and qualify it below). The operational conclusion: separate the data plane (CPU) from the generation plane (GPU). In the 4×H100 cluster, no H100 should be spent re-indexing a corpus that changes once a day. That goes to the generic CPU fleet (Xeon AMX, NUCs), and the H100s are reserved for generation and, at most, for rerank peaks or large 7B embedders. What does not go down to CPU: interactive generation, massive reranking at high QPS, re-indexing with a strict real-time SLA, and 7B embedders (gte-Qwen2, NV-Embed).

The analogy: the library and the librarian

Picture a serious research library. There are three distinct jobs, done by different people, on different clocks.

The first is cataloguing. Boxes of new books arrive; someone opens them, classifies them, assigns a shelf mark, indexes them in the catalogue and puts them on the right shelf. It is patient background work, done at night or between other tasks. Nobody is standing over you with a stopwatch waiting for today’s batch to be catalogued: what matters is that tomorrow it is done and done well. It is pure throughput: how many books you catalogue per hour, not how long you take on one specific book. This is ingestion.

The second is handling an enquiry at the desk. A reader turns up and asks about a topic. The librarian goes to the catalogue, which already exists, finds half a dozen relevant shelf marks, fetches them and puts the books on the desk. It is fast, light, and consists of searching an index that already exists, not building it. This is retrieval.

The third is writing a reasoned report from those books. The reader, or an expert you commission, reads the six books, compares them, synthesises, and writes an argued answer with citations. This is slow, demands a very well-trained mind, and the reader is waiting: here there is a human stopwatch. This is generation, the LLM.

The moral is about staff allocation. You do not put your star writer, expensive, scarce, with a queue of people waiting for reports, to catalogue boxes of books at dawn. Cataloguing is done by a large, cheap team that works through the night without rushing. The star writer touches only what genuinely needs their mind: writing. In our system, the star writer is the H100, and cataloguing at dawn is corpus ingestion. Spending the H100 on re-indexing is exactly the error of putting the writer to work labelling boxes.

The rest of the post is, essentially, which parts of the library work the cheap CPU team can do (almost all of them) and which is irreducibly the GPU writer’s job (only the last one).

The three phases and their compute profiles

The confusion that GPU over-provisioning feeds on is treating “RAG” as a monolithic block that “uses AI, therefore goes on the GPU”. No. RAG is a data pipeline with a generative model plugged in at the end. The correct architectural boundary does not separate “what uses models” from “what does not”, because both sides use models, but throughput-bound from latency-bound, which is the same as separating the data plane from the generation plane.

Why ingestion fits on CPU: the embedder is not an LLM

The whole argument rests on a size asymmetry that gets overlooked. People hear “embeddings” and “generation” and put them in the same bag of “big models that need a GPU”. But the embedding encoder and the generative LLM are two orders of magnitude apart in parameters.

bge-m3, the reference multilingual embedder, is an XLM-RoBERTa of around 568M parameters (model card, paper arXiv:2402.03216). Its sibling reranker, bge-reranker-v2-m3, is built on the same base and lands at roughly the same 568M parameters (model card). Compare that with an entry-level generative LLM: a Llama 3.1 8B has around 14× more parameters, and the big production models sit at 70B+. A 568M encoder is, in compute budget terms, a different animal.

Two structural differences make that encoder comfortable on CPU:

It is an encoder, not an autoregressive decoder. It processes the whole sequence in a single forward pass and emits the vector. There is no token-by-token decode, no growing KV cache, none of the memory-bound generation phase that kills the CPU. It is a dense matrix pass and that is it.
It quantises to int8 with almost no loss. In int8, bge-m3 occupies on the order of 580 MB and, above all, activates the integer compute paths that a modern CPU executes quickly: matrix instructions such as Intel AMX (Advanced Matrix Extensions, 4th-generation Xeon and later), AVX-512 with VNNI (Vector Neural Network Instructions) on earlier Xeons, and NEON SDOT/UDOT on ARM. The quality loss from going FP32 to int8 on these models usually stays below 1% of retrieval recall, practically invisible (Intel + Hugging Face, CPU Optimized Embeddings).

Let us quantify the int8 size. For $P = 568 \times 10^6$ parameters at 1 byte each:

$$\text{size}_{\text{int8}} \approx 568 \times 10^6 \text{ params} \times 1 \text{ byte/param} \approx 568 \text{ MB}$$

In other words, the model fits in the cache and RAM of any server or NUC without blinking, and the bottleneck is integer compute, exactly what AMX/VNNI accelerate. There is nothing in this profile that asks for a GPU.

Runtimes that already do this effortlessly

Nothing needs inventing. The CPU ecosystem for the data plane is mature:

Text Embeddings Inference (TEI) from Hugging Face: a Rust server with CPU backends via ONNX Runtime (recommended) or Intel MKL, and OpenAI-compatible endpoints (/v1/embeddings) plus /rerank (TEI repo). That is, the data plane on CPU exposes exactly the same HTTP contract as a GPU server; the rest of the system never learns which silicon is behind it.
fastembed from Qdrant: a lightweight library that loads embedders on ONNX-CPU and generates dense, sparse and ColBERT vectors (fastembed repo). Designed from the start to run without a GPU.
bge-m3 in ONNX int8 with its three heads (dense / sparse-lexical / ColBERT multivector) exported and quantised, ready for ONNX Runtime CPU.

The Intel and Hugging Face figure that anchors viability: in their benchmark with Optimum Intel + fastRAG on a 4th-generation Xeon (8480+, 56 cores, 1 socket), the int8 variant of BGE-large reaches up to around 10× indexing throughput against FP32 (HF blog, Haystack/deepset). The small print needs reading, so I read it: that ~10× is encoding-only (tokenisation excluded), at sequence length 256, comparing int8 against FP32 on the same CPU. It is not “CPU 10× faster than GPU”, it is “int8 10× faster than FP32 on CPU”. It is still the relevant figure: it tells you that with quantisation the CPU goes from unworkable to perfectly useful for batch ingestion.

Viability table: CPU for each component?

This is the operational table. The column that matters is the caveat, because a bare “yes” or “no” lies.

Component	CPU viable?	Caveat
Chunking (slicing the corpus)	Yes, always	It is regex, parsing and windows; it never had anything to do with the GPU.
Ingest embedding `bge-m3` dense	Yes, its best case	Overnight batch, int8 + AMX/VNNI. This is exactly what the CPU shines at.
Sparse / SPLADE / BM25 head	Yes, CPU-native	Lexical work is pure inverted index; the GPU adds nothing here.
Building the HNSW index (Qdrant, pgvector)	Yes, always CPU	The HNSW graph build is CPU by design in these engines.
Query embedding (online)	Yes	A single short text; tens of ms on CPU, more than enough for chat.
Dense + sparse search + RRF	Yes	Vector search always ran on CPU, even in “GPU” stacks. RRF is sorting lists.
Cross-encoder reranker `bge-reranker-v2-m3` top-20/50	Yes, with care	A cross-encoder evaluates $k$ query-doc pairs: cost $\propto k$. Over 20-50 candidates it works; over hundreds at high QPS, no.
ColBERT late interaction	Marginal on CPU	The token-by-token matrix product of late interaction is heavy; viable at low volumes, suffers with QPS.
LLM generation	No, in practice	A 7B on CPU gives TTFT in seconds. Interactive latency = GPU.

Two rows deserve underlining because they dismantle myths.

“Vector search needs a GPU.” False from the start. The HNSW index, the navigable small-world graph used by Qdrant, pgvector with vector/halfvec, Milvus in CPU mode and almost everything else, was always built and traversed on CPU. Even the stacks advertised as “GPU-accelerated RAG” do the embedding on GPU but the ANN search stays on CPU in the vast majority of deployments; the GPU variants of the index (CAGRA and similar) are the expensive exception, not the norm, and are justified only with billions of vectors and extreme QPS. For a corporate corpus of millions of chunks, HNSW on CPU resolves in single-digit milliseconds.

“The reranker is a model, therefore GPU.” The bge-reranker-v2-m3 reranker is a cross-encoder of around 568M: it runs on CPU. The caveat is the number of pairs. A cross-encoder does not produce a reusable vector; it evaluates the (query, document) pair jointly, so its cost grows linearly with the number of candidates $k$:

$$\text{cost}_{\text{rerank}} \propto k \times \text{forward}(\text{query} + \text{doc})$$

Reranking the top-20 or top-50 coming out of hybrid retrieval is perfectly affordable on CPU. Reranking hundreds of candidates at high QPS is not: there the linear cost explodes and the GPU wins. The rule of thumb: broad cheap recall in the retriever, precision rerank over few candidates. (The detail of hybrid retrieval and reranking is in the fundamentals piece linked below.)

The numbers, with honest methodology

Here comes the part where a lot of people lie by omission. I am going to give throughput ranges, but they are literature ranges and orders of magnitude, not my own measurements on this hardware. Take them as such: the right decision does not depend on nailing the number, it depends on understanding the allocation.

For bge-m3 dense, sequence ≈256 tokens, embedding throughput moves roughly like this:

Platform	Dense throughput (order of magnitude)	Reading
High-end GPU (5090 fp16, TEI)	~12k tok/s+ (indicative)	The ceiling; expensive and scarce.
Large server CPU (Xeon ~56 cores, int8 ONNX)	low band of thousands tok/s	~1/5–1/10 of the GPU, but horizontally scalable and cheap.
Edge CPU / NUC (4-8 cores, int8)	tens to low hundreds tok/s	Enough for overnight ingestion of a local corpus.

The temptation is to read the second row as “CPU is 5-10× slower, discarded”. That is the wrong reading for ingestion. For batch work with no SLA, what rules is not absolute tok/s but throughput per euro and throughput per watt, and there the arithmetic changes sign.

Take a numerical example of allocation. Suppose a corpus of 2 million chunks of around 256 tokens that has to be re-indexed once a day (the corpus changes, the embeddings have to be redone). That is:

$$2 \times 10^6 \text{ chunks} \times 256 \text{ tok/chunk} \approx 5.1 \times 10^8 \text{ tokens}$$

At a conservative CPU throughput of, say, 3,000 tok/s per int8 Xeon server:

$$t_{\text{ingest}} \approx \frac{5.1 \times 10^8 \text{ tok}}{3000 \text{ tok/s}} \approx 1.7 \times 10^5 \text{ s} \approx 47 \text{ hours on a single server}$$

47 hours on one box sounds bad until you remember two things. First, this is embarrassingly parallel: the corpus is sliced and distributed; with 8 CPU servers it drops to around 6 hours, with 16 to around 3 hours, comfortably inside the overnight window. Second, and more importantly: that same work on the GPU blocks the GPU. If the H100 does 12k tok/s, it takes around 12 hours, but they are 12 hours of the H100, the resource the whole organisation fights over in order to generate. Spending the scarce, expensive resource on re-indexing a corpus that changes once a day is bad allocation, even if it is “faster”: you are optimising the wrong tok/s.

The mental rule: for ingestion, optimise throughput/€ and throughput/W; absolute tok/s belongs to the generation plane, where the human stopwatch really is running.

Decision tree: CPU or GPU for this piece?

Reference architecture (a): CPU-only

The first case is a node with no GPU: a NUC, an office Xeon, a sovereign edge server in a branch or in an air-gapped environment. The whole data plane lives there; generation is delegated to a remote GPU endpoint or done in batch with an SLM when latency is not pressing.

The stack:

TEI-CPU serving bge-m3 int8 with dense + sparse (same OpenAI contract /v1/embeddings, plus /rerank for the reranker).
Qdrant with a dense HNSW index plus sparse vectors, native RRF fusion.
Reranker bge-reranker-v2-m3 over the top-k (via TEI’s /rerank).
Gateway that orchestrates and, for generation, calls an external endpoint.

# docker-compose: full RAG data plane on CPU (no GPU)
services:
 tei-embed:
 image: ghcr.io/huggingface/text-embeddings-inference:cpu-latest
 command: ["--model-id", "BAAI/bge-m3", "--pooling", "cls", "--dtype", "int8"]
 ports: ["8081:80"]
 # ONNX/MKL backend: exploits AVX-512+VNNI / AMX if the Xeon supports it

 tei-rerank:
 image: ghcr.io/huggingface/text-embeddings-inference:cpu-latest
 command: ["--model-id", "BAAI/bge-reranker-v2-m3", "--dtype", "int8"]
 ports: ["8082:80"]
 # exposes /rerank — invoked ONLY over top-20/50, never over hundreds

 qdrant:
 image: qdrant/qdrant:latest
 ports: ["6333:6333"]
 volumes: ["./qdrant_storage:/qdrant/storage"]
 # HNSW dense + sparse vectors + RRF, all CPU

Hybrid search with RRF fusion in Qdrant (dense + sparse in a single query):

from qdrant_client import QdrantClient, models

client = QdrantClient(url="http://qdrant:6333")

# query embedding: dense and sparse from TEI-CPU (HTTP wiring omitted)
hits = client.query_points(
 collection_name="corpus",
 prefetch=[
 models.Prefetch(query=dense_vec, using="dense", limit=50),
 models.Prefetch(query=sparse_vec, using="sparse", limit=50),
 ],
 query=models.FusionQuery(fusion=models.Fusion.RRF), # native RRF
 limit=20,
).points
# -> then: POST /rerank (TEI) over these 20, keep the top-5
# -> then: the gateway sends query + top-5 to the GENERATION endpoint (GPU)

Generation, on this CPU-only node, leaves the node: the gateway builds the augmented prompt and sends it to a vLLM endpoint on the GPU cluster (or, if there is no interactive SLA, to an SLM on CPU in batch mode, accepting a TTFT of seconds). The entire data plane, everything above, runs without a single GPU.

Reference architecture (b): the recommended hybrid

This is the one I recommend for the general case with a GPU cluster available: data plane on CPU, generation plane on GPU, connected by OpenAI-compatible HTTP contracts so that each side is replaceable.

Generation server, minimal, on GPU:

# vLLM on the GPU cluster — generation ONLY
services:
 vllm-gen:
 image: vllm/vllm-openai:latest
 command: >
 --model meta-llama/Llama-3.1-8B-Instruct
 --dtype bfloat16 --max-model-len 8192
 --gpu-memory-utilization 0.85
 # exposes /v1/chat/completions — the gateway sends it query + the top-5 already retrieved
 deploy:
 resources:
 reservations:
 devices: [{driver: nvidia, count: 1, capabilities: [gpu]}]

The virtue of the design: because both sides speak the OpenAI contract over HTTP, the CPU data plane and the GPU generation plane scale separately and are replaceable. If tomorrow you want to move the rerank to GPU because QPS went up, you change one URL. If you want to add more CPU ingest nodes, you add them without touching generation. The whole stack is OSS and licence-clean: bge-m3 and bge-reranker-v2-m3 are MIT (bge-m3, reranker), Qdrant is Apache-2.0, TEI and vLLM are OSS.

Applied to the generic 4×H100 cluster

Let us bring this down to the cluster in the series: 4×H100 SXM 80 GB plus a generic CPU fleet (Xeon with AMX, NUCs). The correct allocation:

Build and indexing → CPU fleet. No H100 should spend a cycle re-embedding the corpus. That goes to the Xeon AMX machines (large servers, throughput in the thousands of tok/s in int8) or, for small local corpora, to the NUCs overnight. Overnight re-indexing of a corpus that changes once a day is the textbook case of “unhurried CPU work”.
The H100s → generation. All four cards are reserved for what only they do well: producing tokens at interactive latency. This is what the sibling pieces in the series, sharing a GPU and several models on one GPU, help to squeeze: once ingestion stops competing for the GPU, all the expensive silicon is free to generate and is shared better between models and tenants.
The H100s, at most, → rerank peaks or large embedders. If at some point you need a 7B embedder (gte-Qwen2, NV-Embed) for a domain where bge-m3 does not reach, or a massive rerank at a QPS the CPU cannot absorb, those peaks can indeed visit the GPU. But they are the occasional exception, not the base load.

The auditability angle: ENS / NIS2

There is a compliance argument that rarely gets mentioned and that the CPU/GPU split gives you almost for free.

A CPU-only node with no proprietary driver is easier to audit. There is no closed NVIDIA kernel stack, no CUDA and firmware versions to reconcile with the supply chain, no proprietary driver surface to document for an ENS or a NIS2 assessment. The whole data plane, chunking, embeddings, index, search, runs on OSS software on generic CPU with standard instructions. For a sovereign or classified environment, being able to say “the plane that touches the corpus depends on no proprietary binary” is a real argument, not marketing.

And there is a second auditability angle intrinsic to RAG done well: source traceability. A RAG system that retrieves identifiable chunks and cites them is auditable, since you can reconstruct which document each claim came from, as opposed to context-stuffing or the model’s opaque parametric knowledge, where there is no way to know where a fact came from. That traceability lives in the data plane (what was retrieved, from which source, with what score), precisely the plane we are putting on auditable CPU. The two arguments reinforce each other: the auditable silicon and the auditable chain of evidence are the same plane.

When NOT to take it to CPU

For honesty, and to avoid falling into the mirror image of the hype, the cases where CPU is not the answer:

Generation at interactive latency. The obvious case. A 7B on CPU gives TTFT in seconds: unacceptable for chat. If the user is waiting, generation goes to GPU. With no practical exceptions today.
Massive reranking at high QPS. A cross-encoder or ColBERT over hundreds of candidates, multiplied by many requests per second, saturates the CPU. The cost $\propto k \times \text{QPS}$ crosses the threshold where the GPU pays off. Keep CPU rerank bounded to top-20/50; if you need more breadth at higher QPS, move up to GPU.
Real-time re-indexing with a strict SLA. If the corpus changes continuously and freshness is measured in seconds (not hours), CPU throughput may not fit the window. There, ingest embedding may need a GPU, but note that this is rare: most corporate corpora change at the pace of hours or days, not seconds.
Large embedders (7B). bge-m3 (568M) is comfortable on CPU; a gte-Qwen2 or NV-Embed at 7B is LLM-class again and drags the same cost profile as generation. If your retrieval quality demands a 7B embedder, that embedder lives where 7Bs live: on the GPU.

The sentence that sums it all up: the CPU is the default home of the data plane; the GPU is the justified exception for what is latency-bound and massively online. Start by putting everything on CPU and move up to GPU only what proves it does not fit, not the other way round.

References

Chen, J., et al. BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings. arXiv 2402.03216. https://arxiv.org/abs/2402.03216
BAAI — BGE-M3 model card (568M params, XLM-RoBERTa, 8192 tokens, MIT). https://huggingface.co/BAAI/bge-m3
BAAI — bge-reranker-v2-m3 model card (cross-encoder on top of bge-m3, ~568M). https://huggingface.co/BAAI/bge-reranker-v2-m3
Intel + Hugging Face — CPU Optimized Embeddings with Optimum Intel and fastRAG (~10× indexing for BGE-large int8, 4th-gen Xeon). https://huggingface.co/blog/intel-fast-embedding
deepset / Haystack — CPU-Optimized Embedding Models with fastRAG and Haystack. https://haystack.deepset.ai/blog/cpu-optimized-models-with-fastrag
Hugging Face — Text Embeddings Inference (TEI), CPU ONNX/MKL backends, OpenAI-compatible endpoints. https://github.com/huggingface/text-embeddings-inference
Qdrant — fastembed (ONNX-CPU, dense/sparse/ColBERT) and Hybrid Search with RRF. https://github.com/qdrant/fastembed · https://qdrant.tech/documentation/beginner-tutorials/hybrid-search-fastembed/

Semantic cache in RAG: the receptionist with a photographic memory

Thu, 04 Jun 2026 08:00:00 +0200

TL;DR

In a RAG system with real traffic, 30–70% of queries are semantically equivalent to some earlier one even when the text differs. The semantic cache intercepts those queries ahead of the retriever and the LLM, returning the answer already computed if the cosine similarity with a previous query exceeds a threshold θ (typically 0.92–0.95). At 10,000 requests/day and a 45% hit rate, that amounts to not running 4,500 LLM generations: roughly 0.62 GPU hours saved every day on a cluster running Llama-3.1-70B. The fundamental trade-off is that a high θ gives more accurate answers but less saving; a low θ maximises saving but can return incorrect answers for subtly different queries.

The analogy: the receptionist with a notebook

Picture the front desk of a 400-room hotel. Over the course of the day, the receptionist takes hundreds of questions. But if you look through the log book, you will see that 60% of those questions are variants of the same ten:

“Where is the gym?”
“What time is breakfast?”
“Do you have parking?”
“How do I connect to the WiFi?”

By the third day, the receptionist has built a mental notebook of answers. When someone asks “where can I go to exercise?”, they do not call the concierge (retrieval) or open the hotel’s 300-page internal manual (LLM): they look at the notebook, work out that the question is the same as “where is the gym?”, and answer in two seconds.

But when someone asks “what time does the gym close today?”, the receptionist knows the notebook cannot be trusted: the schedule may have changed because of a private event. They have to call the concierge.

That is exactly the mechanism of a semantic cache:

The notebook is the cache store (Redis with a vector index, or a Qdrant collection).
Working out that “exercise” ≈ “gym” is the cosine similarity search with threshold θ.
Calling the concierge is retrieval over the corpus.
Consulting the manual is LLM generation.
“Today” is the temporal query signal that invalidates the cache.

The threshold θ is exactly what separates “where is it” (semantically the same) from “what time is it open today” (semantically different). It is not magic: it is vector arithmetic over learned representations.

The problem in production

A typical RAG pipeline has three layers of latency and compute: embedding the query, vector search over the corpus, and generation with the LLM. In development, that cost is irrelevant. In production with 50 concurrent users, each of those layers scales linearly with the number of requests.

The problem is that users ask the same questions over and over, with slightly different wording:

Original query	Equivalent query
“How do I configure the agent?”	“What is the process for configuring the agent?”
“error installing the dependency”	“the dependency installation fails”
“what is an embedding?”	“explain to me what embeddings are”

Running the full pipeline for each of these variants is pure waste. Empirical studies in technical support and corporate Q&A systems report that between 30% and 70% of a day’s queries are semantically redundant with respect to earlier queries from the same week.

The query distribution in real systems follows a power law similar to the Zipf distribution: the 100 most frequent topics account for roughly 60% of total traffic. A well-calibrated cache captures exactly that concentration.

How the semantic cache works

The full flow is shown in the diagram below. Let us describe it in prose first.

When a new query $q$ arrives:

Embedding the query: $q$ is embedded with the same model used to index the corpus. This is critical: if the corpus was indexed with text-embedding-3-large and the cache uses a different embedder, the vector spaces are not comparable.
Search in the cache store: an ANN (Approximate Nearest Neighbor) search is run over the vectors of previously cached queries. The most similar query $q^*$ and its cosine similarity $s$ are retrieved.
Threshold decision:
$$ \text{answer} = \begin{cases} r^* & \text{if } s(q, q^*) \geq \theta \\ \text{pipeline}(q) & \text{if } s(q, q^*) < \theta \end{cases} $$
where $r^*$ is the cached answer associated with $q^*$.
On a miss: the full pipeline runs (retrieval + LLM). The generated answer is stored in the cache with a configurable TTL for future similar queries.

The cache store is not an ordinary key-value database. It is a vector index over the query embeddings, with the values being the generated answers. Each entry has the structure:

{vector: embed(q), response: r, ttl: T, metadata: {...}}

Flow diagram

The semantic cache flow as middleware between the gateway and the retriever. Cache hits skip the vector search over the corpus and the LLM generation entirely.

The threshold θ and its trade-off

The threshold θ is the most sensitive parameter in the system. It works exactly like the receptionist’s recognition threshold: if it is too demanding, it will only identify textually identical questions and the notebook will not be much use. If it is too lax, it will return the answer to “where is the gym?” to someone who asked “what time does the gym close?”.

The cosine similarity between two vectors $\mathbf{a}$ and $\mathbf{b}$ is:

$$ s(\mathbf{a}, \mathbf{b}) = \frac{\mathbf{a} \cdot \mathbf{b}}{\|\mathbf{a}\| \cdot \|\mathbf{b}\|} $$

For prose text (Spanish or English), modern embedders such as text-embedding-3-large or nomic-embed-text assign cosine similarities of around 0.90–0.96 to semantically equivalent paraphrases and similarities of 0.75–0.88 to related but non-equivalent queries.

The cache quality metric is not just hit rate: it is precision@cache, defined as the fraction of cached answers that remain correct for the new query. A cached answer is “correct” if an evaluator (another LLM, or metrics such as BERTScore) judges it equivalent to the one the full pipeline would have generated for that specific query.

θ	Estimated hit rate	Estimated precision@cache	Effective saving
0.85	~65%	~72%	~47%
0.90	~55%	~85%	~47%
0.92	~48%	~91%	~44%
0.93	~45%	~94%	~42%
0.95	~35%	~98%	~34%
0.97	~18%	~99.5%	~18%

The effective saving is defined as $\text{hit rate} \times \text{precision@cache}$, since a hit with an incorrect answer is not a saving: it is an error that can cost more in lost trust than what was saved in GPU.

The empirical sweet spot for most corporate Q&A applications in Spanish or English lies between θ = 0.92 and θ = 0.95. In highly specialised domains where small nuances change the answer (medicine, law, finance), θ ≥ 0.95 is advisable.

The arithmetic of the saving

Let us put concrete numbers on a real system.

Base configuration:

10,000 requests/day
Technical corpus of 1 million chunks in a Qdrant index
LLM: Llama-3.1-70B on 4×H100 SXM (320 GB, NVLink)
Average answer: 200 output tokens
LLM throughput on this hardware: ~400 tokens/s/GPU with batching (continuous batching enabled, see continuous-batching-fundamentos)

Cost of a request without cache:

Embedding the query takes ~2 ms on a GPU. The ANN vector search over 1 M chunks in Qdrant takes ~5 ms (measured empirically with HNSW, ef=128). Generating 200 tokens at 400 tok/s in total (4 GPUs) comes to:

$$ t_{\text{LLM}} = \frac{200 \text{ tokens}}{400 \text{ tok/s}} = 0.5 \text{ s per request} $$

If 10,000 requests reach the LLM in a day, the total GPU time spent on generation is:

$$ T_{\text{GPU}} = 10{,}000 \times 0.5 \text{ s} = 5{,}000 \text{ s} \approx 1.38 \text{ GPU hours per day} $$

With a semantic cache at θ = 0.93 and a hit rate of ~45%:

Only 5,500 requests (55%) reach the LLM:

$$ T_{\text{GPU,cache}} = 5{,}500 \times 0.5 \text{ s} = 2{,}750 \text{ s} \approx 0.76 \text{ GPU hours per day} $$

Saving:

$$ \Delta T_{\text{GPU}} = 1.38 - 0.76 = 0.62 \text{ GPU hours/day} $$

In inference compute, this amounts roughly to being able to serve 45% more users without adding hardware, or cutting inference costs by 45% if you work with external APIs billed per token.

The cost of the semantic cache itself (embedding the query + ANN search over the cache store) is ~7 ms per request, negligible against the 500 ms of generation avoided on hits.

Zipf distribution of topics:

The reason it works is the Zipf distribution of traffic. If we number topics by frequency (topic 1 = most frequent), the frequency of topic $k$ is proportional to $1/k$. With 1,000 distinct topics:

$$ \text{fraction of traffic covered by top-}N = \frac{\sum_{k=1}^{N} 1/k}{\sum_{k=1}^{1000} 1/k} \approx \frac{\ln N}{\ln 1000} = \frac{\ln N}{6.9} $$

For the top-100 topics: $\ln(100)/6.9 \approx 4.6/6.9 \approx 67\%$ of traffic. The cache does not need to cover every topic: it captures 67% of traffic while covering only 10% of the topics.

OSS stack 2026

GPTCache

GPTCache is the reference library for a standalone semantic cache. Its architecture is modular:

Embedder: ONNX Runtime with converted models (onnx/all-MiniLM-L6-v2 by default), with no GPU dependency for the cache layer.
Vector store: Faiss (local), Milvus, or Qdrant.
Scalar store: SQLite (development) or Redis (production) for metadata, TTL, and answers.
Similarity evaluation: cosine by default, configurable.

Minimal configuration in Python:

from gptcache import cache
from gptcache.adapter import openai
from gptcache.embedding import Onnx
from gptcache.manager import CacheBase, VectorBase, get_data_manager
from gptcache.similarity_evaluation.distance import SearchDistanceEvaluation

onnx = Onnx()
data_manager = get_data_manager(
 CacheBase("redis", url="redis://localhost:6379"),
 VectorBase("qdrant", host="localhost", collection_name="query_cache")
)
cache.init(
 embedding_func=onnx.to_embeddings,
 data_manager=data_manager,
 similarity_evaluation=SearchDistanceEvaluation(),
)
cache.set_openai_key()

GPTCache intercepts calls to the OpenAI API (or to compatible proxies) transparently. The TTL is configured at the data_manager level.

MeanCache

MeanCache (2024) extends GPTCache to multi-turn conversations. The problem with standard GPTCache is that in dialogue the “relevant” query is not just the last message but the whole context window. MeanCache computes the query embedding as the weighted mean of the embeddings of the last $k$ turns:

$$ \mathbf{e}_{\text{query}} = \frac{\sum_{i=1}^{k} w_i \cdot \mathbf{e}_{q_i}}{\sum_{i=1}^{k} w_i} $$

where $w_i$ decreases with the age of the turn. This reduces false positives in dialogues where the topic keeps shifting.

Qdrant as a dual cache store

If the RAG corpus is already in Qdrant, you can use the same instance with a separate collection for the cache. The advantages are operational: a single service to manage, the same backup and monitoring infrastructure.

The cache collection uses payload filters to implement TTL:

from qdrant_client import QdrantClient
from qdrant_client.models import Filter, FieldCondition, Range
import time

client = QdrantClient("localhost", port=6333)

# Search the cache with a TTL filter
hits = client.search(
 collection_name="semantic_cache",
 query_vector=query_embedding,
 query_filter=Filter(
 must=[FieldCondition(
 key="expires_at",
 range=Range(gt=time.time())
 )]
 ),
 limit=1,
 score_threshold=0.93
)

You have to run a periodic cleanup of expired entries, since Qdrant has no native TTL (unlike Redis).

Langfuse for traceability

Langfuse is the OSS standard for LLM pipeline observability (see tracing-llm-otel-genai). Every request should be tagged with whether it was a cache hit or a miss:

from langfuse import Langfuse
from langfuse.decorators import observe

langfuse = Langfuse()

@observe()
def process_query(query: str) -> dict:
 cache_result = semantic_cache.lookup(query)
 if cache_result:
 langfuse.update_current_observation(
 metadata={"cache_hit": True, "cache_score": cache_result.score}
 )
 return cache_result.response
 # full pipeline...
 langfuse.update_current_observation(
 metadata={"cache_hit": False}
 )

With this metadata, Langfuse lets you compute the real hit rate, the distribution of similarity scores, and detect whether the threshold θ needs adjusting.

On-premise hardware: a reference configuration

For an on-premise deployment with this stack, a suitable configuration for RAG with a semantic cache is:

Inference node: 4×H100 SXM (320 GB NVLink in total) for Llama-3.1-70B in FP8. Throughput ~400 tok/s in generation with continuous batching (vLLM or TGI).

Vector services node: CPU with 256 GB RAM. Qdrant for the corpus (1–10 M chunks) and for the cache store (up to 500K entries in memory). Redis 7.x for metadata and an exact-match cache as a first layer.

Embedding node: CPU or a mid-range GPU (A10G). The cache embedder can run on ONNX Runtime on CPU with no perceptible impact on latency (~2 ms per embedding).

Separating the cache store from the corpus matters: the corpus holds millions of chunks with large HNSW indexes; the cache store holds at most tens of thousands of queries with a much smaller index and search times of 1–2 ms.

Cases where the cache fails

The receptionist with a notebook fails in three well-defined scenarios:

1. Queries with a temporal context

“What is the current status of the incident?” or “What changed in the latest release?” are questions whose correct answer changes over time. A cache with a 24-hour TTL could return stale information.

The solution is to detect temporal markers in the query (regular expressions over “today”, “now”, “current”, “latest”, “yesterday”, and their Spanish equivalents) and force a cache miss for these queries, whatever the similarity score.

2. Personalised queries over private data

If the RAG has access to user data (account history, private documents), two different users asking the same question must get different answers. A shared cache that ignores user context is a privacy risk.

The solution is a cache partitioned by user_id or tenant_id. This lowers the hit rate (each user’s cache is smaller) but it is the only safe option in multi-tenant architectures.

3. TTL and a stale corpus

When the corpus is updated (new documents are ingested, errors are corrected), cached answers can go out of date. A fixed TTL (24–48 hours) mitigates the problem but does not remove it.

For corpora with frequent updates, the solution is an active invalidation mechanism: when the corpus is updated in Qdrant, a job runs that identifies which cache entries might be affected (by semantic overlap with the updated chunks) and deletes them. This is the “selective cache invalidation” mentioned in the section on topics not covered.

Integration into the pipeline as middleware

The semantic cache is implemented as middleware between the API gateway and the retriever. It does not change the API contract: the client still sends queries and receives answers in the same format.

class SemanticCacheMiddleware:
 def __init__(self, cache_store, retriever, llm, threshold=0.93):
 self.cache = cache_store
 self.retriever = retriever
 self.llm = llm
 self.threshold = threshold

 async def process(self, query: str, context: dict) -> dict:
 # First layer: exact-match cache (Redis GET, O(1))
 exact = await self.cache.exact_lookup(query)
 if exact:
 return {**exact, "cache_type": "exact"}

 # Second layer: semantic cache (ANN search)
 query_embedding = await self.embed(query)
 semantic = await self.cache.semantic_lookup(
 query_embedding, threshold=self.threshold
 )
 if semantic:
 return {**semantic, "cache_type": "semantic"}

 # Miss: full pipeline
 chunks = await self.retriever.retrieve(query_embedding)
 response = await self.llm.generate(query, chunks)

 # Store for future queries
 await self.cache.store(
 embedding=query_embedding,
 query=query,
 response=response,
 ttl=context.get("ttl", 86400)
 )
 return {**response, "cache_type": "miss"}

The first exact-match layer (Redis GET) is an extra optimisation: for textually identical queries, the embedding is not even computed. The cost is a Redis operation taking microseconds. Only if there is no exact match does it fall through to the semantic lookup.

What we have not covered

Selective cache invalidation: when a subset of the corpus is updated (for example, the documents for a specific product are reindexed), you would need to identify which cache entries overlap semantically with the updated chunks and mark them stale. The mechanism involves computing similarity between the embeddings of the updated chunks and the embeddings of the cached queries, which is expensive at scale.
Multi-tenant cache: isolation vs sharing: in a SaaS product with multiple customers, a shared cache maximises the hit rate but can expose one tenant’s answers to another if filtering is not done correctly. A cache partitioned by tenant is safe but has much lower hit rates. The middle ground is a shared cache with ACL filtering applied over Qdrant’s payload filters.
Semantic cache for streaming responses: when the LLM emits tokens in streaming mode (SSE), the cache cannot easily intercept the complete answer. The options are: cache on the first miss and return the whole answer at once on hits (breaking the streaming experience), or implement a “fake streaming” that emits the tokens of the cached answer at a controlled rate.
Exact-match cache as a first layer: ahead of the semantic cache, an O(1) lookup in Redis with the query as the key can catch textually identical queries at negligible cost. The code in the previous section already shows this two-layer architecture.

References

Bang, J. et al. (2024). MeanCache: User-Centric Semantic Cache for Large Language Model Based Web Applications. arXiv:2403.02694.
Zilliz. (2023). GPTCache: A Library for Creating Semantic Cache for LLM Queries. GitHub: zilliztech/GPTCache.
Qdrant Team. (2024). Qdrant Documentation: Filtering with payload. qdrant.tech/documentation.
Manning, C. D., Raghavan, P., Schütze, H. (2008). Introduction to Information Retrieval. Cambridge University Press. Cap. 19: Web search (distribución Zipf).
Langfuse. (2024). Observability for LLM Applications. langfuse.com/docs.
Meta AI. (2024). Llama 3.1 Model Card. ai.meta.com.
Guo, Y. et al. (2023). Evaluating the Factual Consistency of Large Language Models Through Summarization. Referencia para BERTScore como métrica de evaluación de respuestas cacheadas.

PostgreSQL + Qdrant in RAG ingestion: the postman who keeps two worlds in sync

Thu, 04 Jun 2026 07:00:00 +0200

TL;DR — In a production RAG system, PostgreSQL holds the official truth about documents and Qdrant holds the vectors for search. Keeping them in sync is not trivial: if you delete a document from Postgres and do not invalidate its chunks in Qdrant, the system returns answers from ghost documents. There are two patterns to avoid this: the outbox pattern (atomic transaction plus asynchronous worker, at-least-once) and CDC with Debezium (reading the Postgres WAL directly, low latency, higher complexity). This article explains when to use each one, how to orchestrate them as microservices and what numbers to expect with bge-m3 on on-premise hardware.

The postman and the civil registry

Picture a town with two complementary offices.

The first is the civil registry: it keeps the official census. Every time someone is born, dies or moves house, the registry is the first to know. It is slow, structured, transactional. If the registry says someone exists, they exist. If it says they died, they are dead. PostgreSQL is the civil registry of your documents.

The second is the postman’s notebook: a copy optimised for finding any resident in seconds, organised by area, phonetic names and habitual routes. The postman cannot update the registry, but can search at speeds the registry would never reach. Qdrant is the postman’s notebook.

The problem is synchronisation. If the registry records a death but nobody tells the postman, he will keep trying to deliver letters to an address that no longer exists. In RAG, that translates into indexed chunks of documents that have already been deleted, edited or replaced: ghost documents that pollute the results.

How does the registry tell the postman?

Outbox pattern: every time the registry updates its ledger, it notes the change on an outbox sheet. A messenger clerk reads that sheet periodically and updates the postman’s notebook. Guaranteed, asynchronous, fault tolerant.
CDC with Debezium: the postman has a direct phone line to the registry. Every time the clerk writes something new, the phone rings and the postman updates his notebook in near real time.

The consistency gap problem

In a naive RAG architecture, the flow is:

The user uploads a document → it is inserted into Postgres with metadata.
A worker splits it into chunks, generates embeddings and upserts into Qdrant.
Retrieval uses Qdrant to find relevant chunks and Postgres to hydrate metadata.

So far so good. The problem shows up in updates and deletes:

The user edits a document → Postgres updates the row, but the vectors of the old chunks are still in Qdrant. Retrieval returns stale context.
The user deletes a document → Postgres removes the row, but the chunks remain in Qdrant. Retrieval returns chunks from a document that should no longer exist.
The permission system revokes a tenant’s access → Qdrant has no way of knowing unless there is explicit synchronisation.

This is not a theoretical problem. In living corpora (corporate wikis, knowledge bases updated daily), the consistency gap accumulates noise progressively. An internal study on production pipelines shows that without active reconciliation, 3-8% of indexed chunks correspond to documents that no longer exist in the source of truth after 30 days of operation.

The answer is not “reindex everything every night”. With 10M chunks and a non-trivial embedding model, that costs hours of compute and causes unavailability windows. The answer is change propagation with guarantees.

Outbox pattern: the outbox sheet

Mechanism

The outbox pattern solves the problem of “writing to two systems in the same operation” without distributed transactions (which are expensive and fragile).

The idea is simple: PostgreSQL is the single coordinator. When the ingestion microservice processes a document, it performs two writes in the same local transaction:

It inserts or updates the document in the documents table.
It inserts an event into the outbox_events table.

If the transaction fails, both writes are rolled back. If it succeeds, both are committed atomically. There is no inconsistent intermediate state.

-- Relevant tables
CREATE TABLE documents (
 id UUID PRIMARY KEY DEFAULT gen_random_uuid(),
 tenant_id TEXT NOT NULL,
 title TEXT NOT NULL,
 content TEXT NOT NULL,
 checksum TEXT NOT NULL,
 updated_at TIMESTAMPTZ DEFAULT now()
);

CREATE TABLE outbox_events (
 id BIGSERIAL PRIMARY KEY,
 aggregate_id UUID NOT NULL, -- document id
 event_type TEXT NOT NULL, -- 'document.created' | 'document.updated' | 'document.deleted'
 payload JSONB NOT NULL,
 created_at TIMESTAMPTZ DEFAULT now(),
 processed_at TIMESTAMPTZ -- NULL = pending
);

-- Example of an atomic insert
BEGIN;
 INSERT INTO documents (tenant_id, title, content, checksum)
 VALUES ($1, $2, $3, $4)
 RETURNING id INTO _doc_id;

 INSERT INTO outbox_events (aggregate_id, event_type, payload)
 VALUES (_doc_id, 'document.created', jsonb_build_object(
 'tenant_id', $1,
 'title', $2,
 'checksum', $4
 ));
COMMIT;

The outbox worker

A separate process (the outbox worker) polls outbox_events where processed_at IS NULL, processes each event (chunking, embedding, upsert into Qdrant) and marks the row as processed:

UPDATE outbox_events
 SET processed_at = now()
 WHERE id = $1;

Guarantee: at-least-once. If the worker fails between the Qdrant upsert and the UPDATE, the event will be reprocessed. Qdrant tolerates idempotent upserts (same point id = overwrite), so reprocessing does not create duplicates.

Latency: it depends on the polling interval. With polling every 500ms, p50 latency is ~250ms; p99, ~500ms. Acceptable for most RAG cases, where the user does not expect to see a document indexed in under a second.

CDC with Debezium: the direct phone line

Mechanism

Change Data Capture (CDC) reads PostgreSQL’s Write-Ahead Log (WAL) directly. Postgres writes every change to the WAL before applying it to the tables; it is the mechanism it uses for replication and recovery. Debezium subscribes to a logical replication slot and turns those events into structured messages.

-- Enable logical replication in postgresql.conf
-- wal_level = logical

-- Create a replication slot for Debezium
SELECT pg_create_logical_replication_slot('debezium_slot', 'pgoutput');

The full flow:

Postgres WAL → Debezium connector → Kafka/NATS → Indexer consumer → Qdrant

Debezium emits events with the record’s before/after structure:

{
 "op": "d",
 "before": {
 "id": "550e8400-e29b-41d4-a716-446655440000",
 "tenant_id": "acme",
 "checksum": "sha256:abc123"
 },
 "after": null
}

With "op": "d" (delete), the consumer knows it must delete every point in Qdrant whose payload contains that document_id.

# Consumer: delete by payload filter
qdrant_client.delete(
 collection_name="corpus",
 points_selector=FilterSelector(
 filter=Filter(
 must=[
 FieldCondition(
 key="document_id",
 match=MatchValue(value=event["before"]["id"])
 )
 ]
 )
 )
)

Upsides and downsides

CDC removes polling and cuts latency to tens of milliseconds (WAL propagation time plus consumer processing). But it adds operational complexity: you need to manage the replication slot (unconsumed slots retain WAL indefinitely, which can fill the disk), the message broker and the consumer offset state.

Comparison: outbox vs CDC

Criterion	Outbox pattern	CDC with Debezium
Typical latency	250ms – 2s	20ms – 200ms
Delivery guarantee	At-least-once	At-least-once
Operational complexity	Low (Postgres only)	High (Debezium + broker)
WAL retention risk	None	High if the slot stalls
Idempotency required	Yes (in the indexer)	Yes (in the consumer)
Multi-table support	Manual	Automatic (any table)
Backpressure	Natural (polling)	Needs explicit design
When to choose it	Corpus < 100k docs/day, small team	Corpus > 1M docs/day, low latency critical

Rule of thumb: start with outbox. Move to CDC when the change volume goes past ~50k events/hour or when latency measured in seconds becomes unacceptable for the use case (for example, real-time news indexing).

Microservice architecture

The ingestion pipeline is made up of three microservices with cleanly separated responsibilities:

Microservice 1: Ingestor

Responsibilities: receive documents, split them into chunks and request embeddings. It does not write to Qdrant directly.

# ingestor/main.py (simplified)
from langchain_text_splitters import RecursiveCharacterTextSplitter

splitter = RecursiveCharacterTextSplitter(
 chunk_size=512, # tokens, not characters
 chunk_overlap=64,
 length_function=token_count,
)

def ingest_document(doc: Document, db: Session) -> None:
 chunks = splitter.split_text(doc.content)
 with db.begin():
 db.execute(
 "UPDATE documents SET checksum=$1 WHERE id=$2",
 [doc.checksum, doc.id]
 )
 db.execute(
 """INSERT INTO outbox_events (aggregate_id, event_type, payload)
 VALUES ($1, 'document.updated', $2)""",
 [doc.id, {"chunks": chunks, "tenant_id": doc.tenant_id,
 "model": "bge-m3", "model_version": "1.0.0"}]
 )

Microservice 2: Indexer

It reads the outbox, generates embeddings by calling the inference server (vLLM or TEI) and upserts into Qdrant.

# indexer/worker.py
import asyncio
from qdrant_client import QdrantClient
from qdrant_client.models import PointStruct, VectorParams, Distance

qdrant = QdrantClient(host="qdrant-service", port=6333)

async def process_event(event: dict) -> None:
 payload = event["payload"]
 chunks = payload["chunks"]
 doc_id = str(event["aggregate_id"])

 # Batch embedding on TEI (Text Embeddings Inference)
 embeddings = await embed_batch(chunks, model="bge-m3")

 points = [
 PointStruct(
 id=f"{doc_id}_{i}",
 vector=emb,
 payload={
 "document_id": doc_id,
 "tenant_id": payload["tenant_id"],
 "chunk_index": i,
 "text": chunks[i],
 "model_version": payload["model_version"],
 }
 )
 for i, emb in enumerate(embeddings)
 ]

 # If it is an update, delete the previous chunks first
 if event["event_type"] in ("document.updated", "document.deleted"):
 qdrant.delete(
 collection_name="corpus",
 points_selector=filter_by_doc_id(doc_id)
 )

 if event["event_type"] != "document.deleted":
 qdrant.upsert(collection_name="corpus", points=points)

Microservice 3: Reconciler

The reconciler is the safety net. Periodically (every hour, for example) it compares the set of document_id values in Postgres with the set of document_id values in Qdrant. IDs present in Qdrant but absent from Postgres are ghosts: they get deleted.

# reconciler/diff.py
async def reconcile(tenant_id: str) -> int:
 pg_ids = set(await fetch_all_doc_ids(tenant_id))
 qdrant_ids = set(await scroll_all_doc_ids(tenant_id)) # paginated scroll

 orphans = qdrant_ids - pg_ids
 if orphans:
 logger.warning("Orphan chunks for %d documents", len(orphans))
 for doc_id in orphans:
 qdrant.delete("corpus", filter_by_doc_id(doc_id))

 return len(orphans)

Throughput and ingestion maths

Embedding throughput

The bge-m3 model (1024 dimensions, dense + sparse + colbert support) on a node with 4×H100 SXM (320 GB NVLink) running via vLLM or HuggingFace TEI reaches roughly 2,000 chunks/second with batch size = 256 and 512-token sequences.

$$\text{throughput} = 4 \times 500 \text{ chunks/s/GPU} = 2{,}000 \text{ chunks/s}$$

The figure of 500 chunks/s per GPU comes from public TEI benchmarks with bge-m3 on H100 SXM5, batch=256, seq_len=512 ¹.

Total re-ingestion time

For a corpus of 10M chunks:

$$t = \frac{10{,}000{,}000 \text{ chunks}}{2{,}000 \text{ chunks/s}} = 5{,}000 \text{ s} \approx 83 \text{ minutes}$$

That is pure embedding time. Adding Qdrant write latency (~0.5ms per upsert batch of 100 points):

$$t_{\text{qdrant}} = \frac{10{,}000{,}000}{100} \times 0.5\text{ ms} = 50{,}000 \text{ ms} = 50 \text{ s}$$

Estimated total for a full re-ingestion: ~85-90 minutes on a 4×H100 node.

Storage cost in Qdrant

Each bge-m3 vector has 1024 dimensions in float32 (4 bytes):

$$\text{size per vector} = 1{,}024 \times 4 \text{ B} = 4{,}096 \text{ B} = 4 \text{ KB}$$

For 10M chunks (dense vectors only):

$$\text{total vectors} = 10^7 \times 4{,}096 \text{ B} = 40.96 \text{ GB}$$

Adding the JSON payload (estimated at ~500 bytes/chunk):

$$\text{payload} = 10^7 \times 500 \text{ B} = 5 \text{ GB}$$

HNSW index (roughly 1.2× the vector size for $m=16$):

$$\text{HNSW} \approx 40.96 \text{ GB} \times 1.2 = 49.15 \text{ GB}$$

Estimated total on disk: ~95 GB for 10M chunks with dense bge-m3.

With scalar quantization (int8), the vector size shrinks 4×:

$$\text{with int8 quantization} \approx \frac{40.96}{4} + 5 + \frac{49.15}{4} \approx 27.5 \text{ GB}$$

Configuration	Vectors	HNSW	Payload	Total
float32, no quantization	40.96 GB	49.15 GB	5 GB	~95 GB
int8 scalar quantization	10.24 GB	12.29 GB	5 GB	~28 GB
binary quantization	1.28 GB	1.54 GB	5 GB	~8 GB

Binary quantization loses recall precision (~2-5% in NDCG@10), but it lets you host much larger corpora in RAM. For production where recall is critical, int8 is the usual balance point.

Recommended on-premise hardware

For a continuous ingestion pipeline in production:

Embedding node: 4×H100 SXM (320 GB, NVLink), 2× 64-core CPU (EPYC 9654), 1 TB DDR5 RAM, 100 GbE. Runs vLLM or TEI serving bge-m3. Sustained throughput: ~2,000 chunks/s with an asynchronous batch pipeline.

Qdrant node: 32-core CPU, 256 GB RAM (to keep the HNSW index in memory with 10M chunks and no quantization), 2 TB NVMe (Qdrant snapshot and WAL writes). Qdrant recommends that the HNSW index fit in RAM for p99 latency < 5ms.

PostgreSQL node: 16-core CPU, 128 GB RAM, 4 TB NVMe for the WAL (especially relevant if you use CDC with a logical replication slot; the slot retains WAL until Debezium consumes it).

Broker (if CDC): 3-broker Kafka with 500 GB NVMe per node, or NATS JetStream with 3 nodes for more modest loads.

Kubernetes manifests

Indexer Deployment

apiVersion: apps/v1
kind: Deployment
metadata:
 name: rag-indexer
 namespace: rag-pipeline
spec:
 replicas: 3
 selector:
 matchLabels:
 app: rag-indexer
 template:
 metadata:
 labels:
 app: rag-indexer
 spec:
 containers:
 - name: indexer
 image: registry.example.com/rag-indexer:1.0.0
 env:
 - name: POSTGRES_DSN
 valueFrom:
 secretKeyRef:
 name: pg-credentials
 key: dsn
 - name: QDRANT_HOST
 value: "qdrant-service.qdrant.svc.cluster.local"
 - name: EMBEDDING_ENDPOINT
 value: "http://tei-service.embeddings.svc.cluster.local:8080"
 - name: POLL_INTERVAL_MS
 value: "500"
 resources:
 requests:
 memory: "512Mi"
 cpu: "500m"
 limits:
 memory: "2Gi"
 cpu: "2"

Reconciler CronJob

apiVersion: batch/v1
kind: CronJob
metadata:
 name: rag-reconciler
 namespace: rag-pipeline
spec:
 schedule: "0 * * * *" # hourly
 concurrencyPolicy: Forbid
 jobTemplate:
 spec:
 template:
 spec:
 restartPolicy: OnFailure
 containers:
 - name: reconciler
 image: registry.example.com/rag-reconciler:1.0.0
 env:
 - name: POSTGRES_DSN
 valueFrom:
 secretKeyRef:
 name: pg-credentials
 key: dsn
 - name: QDRANT_HOST
 value: "qdrant-service.qdrant.svc.cluster.local"
 - name: SCROLL_PAGE_SIZE
 value: "1000"
 resources:
 requests:
 memory: "256Mi"
 cpu: "250m"
 limits:
 memory: "1Gi"
 cpu: "1"

Production gotchas

Reindexing when the embedding model changes

This is the most painful problem. If you move from bge-m3 to nomic-embed-text-v2, the vectors are incompatible: they live in different embedding spaces and the cosine distances between them are meaningless.

The answer is dual-index aliasing:

Create a new collection in Qdrant: corpus_v2.
Re-embed the whole corpus with the new model and load corpus_v2.
When the new collection is complete and validated (recall test), switch the corpus_prod alias from corpus_v1 to corpus_v2.
Delete corpus_v1 once traffic has migrated.

During the migration the two indexes coexist. The retriever uses the alias, not the collection’s direct name.

Index versioning

Store model_version in the payload of every point in Qdrant. This lets you:

Filter by version during retrieval (useful in model A/B testing).
Have the reconciler detect points with an old version and reprocess them selectively.
Audit: know which model generated each embedding.

# Filter by model version at retrieval time
results = qdrant.search(
 collection_name="corpus",
 query_vector=query_embedding,
 query_filter=Filter(
 must=[
 FieldCondition(key="model_version", match=MatchValue(value="1.0.0")),
 FieldCondition(key="tenant_id", match=MatchValue(value=tenant_id)),
 ]
 ),
 limit=10
)

Namespace per tenant (multi-tenancy)

There are two strategies in Qdrant:

Strategy	Pros	Cons
Collection per tenant	Total isolation, no extra filter	N collections = N HNSW indexes in RAM
Payload filter per tenant	A single collection, less RAM	The filter adds ~10-15% search latency

For fewer than 100 tenants with large corpora (> 1M chunks/tenant), use a collection per tenant. For hundreds or thousands of tenants with small corpora, use a payload filter with tenant_id indexed:

qdrant.create_payload_index(
 collection_name="corpus",
 field_name="tenant_id",
 field_schema=PayloadSchemaType.KEYWORD
)

What we have not covered

Streaming corpus updates with near-real-time CDC: selective chunk invalidation when only one section of a document changes (incremental chunking based on content diff, not full re-chunking).
Multi-tenant corpus isolation with per-chunk ACLs: going beyond the tenant_id filter towards group, role or even individual document permissions applied at retrieval time.
Federated corpus: corpora distributed across cross-border silos where regulation (GDPR, CCPA) prevents centralising the embeddings; federated search patterns without moving data.
Zero-downtime incremental reindexing using dual-index aliasing: the full model migration protocol with rollback, recall regression testing and progressive traffic splitting.

References

HuggingFace Text Embeddings Inference — benchmarks oficiales con modelos de la familia bge en hardware A100/H100. https://github.com/huggingface/text-embeddings-inference ↩︎

Embeddings in 2026: the three families (dense, sparse, multi-vector), the model zoo and the decision that matters in production

Wed, 03 Jun 2026 03:00:00 +0200

This post opens the data sub-saga inside the six-stage LLMOps pipeline by going into the piece that holds up three-layer retrieval: the embedder. If the librarian of curation decided what enters the index and the faculty committee decided what reaches the model’s face, this post looks at the one in the middle: the cartographer who draws the map you search over.

TL;DR

The conversation about embeddings has been simplified in production to the point where “which embedder do you use” always gets the same answer: OpenAI text-embedding-3-large in the demo, bge-m3 in the “sovereign-ready” version. That simplification hides the fact that an embedder is three different models at once, dense single-vector, learned sparse (SPLADE) and multi-vector late-interaction (ColBERT), and that in 2026 the leading models do not compete within the same family: gte-Qwen2-7B-instruct and NV-Embed-v2 break MTEB in dense single-vector, SPLADE-v3 and the sparse head of bge-m3 dominate the learned lexical descriptor, Jina-ColBERT-v2 and ColNomic-7B are the strongest in multilingual multi-vector, and Snowflake Arctic Embed L 2.0 has slipped in as the small multilingual favourite with decent Matryoshka. This post takes apart the three families with their maths (InfoNCE with τ, MaxSim, SPLADE’s FLOPS regularization, Matryoshka Representation Learning), goes through the open source model zoo with the dimension, licence and niche of each one, poses the specific problem of multilingual Spanish, which cuts the list of viable models to fewer than six, gives the real storage cost per million chunks with int8 / binary / TurboQuant, describes how they are served on-premise with TEI, Infinity and vLLM --task embed, sets the minimum hardware at an RTX 4090 and the good hardware at a 4×H100 cluster, lists the seven operational traps that sink quality without warning (corpus drift, forgotten normalisation, badly chosen Matryoshka dimension, missing hard negatives, chat template slipped into the embedder, tokenizer drift, MTEB overfit) and closes with a license-clean stack for sovereign production.

The analogy: three librarians cataloguing the same book

A technical library receives a new book and, before putting it on the shelves, has to generate a searchable identifier for it. Three librarians with three different trades live in the library, and all three catalogue the same book at once:

Librarian A, the thematic one. He reads the whole book and gives it a single rich RFID tag. That tag is a vector of 1,024 numbers where each coordinate encodes a latent semantic axis (one the librarian never verbalises: he has learned it by reading a hundred million previous books). Two books about Kubernetes in production will end up with RFID tags very close together in the space even if one talks about Linkerd and the other about Cilium, because they share thematic axes. To search, you compare the question’s tag with the book’s and return the nearest ones by cosine. It is fast, it scales to millions, and it loses fine nuance. This is the dense single-vector embedder: bge-m3 in dense mode, gte-Qwen2-7B-instruct, Snowflake Arctic Embed L 2.0, multilingual-e5-large-instruct.

Librarian B, the lexical one. He summarises nothing. What he does is write on a card the weighted list of relevant terms from the book, expanded with synonyms from the field. The Kubernetes book carries on its card “kubernetes 4.2, linkerd 3.8, cilium 3.7, service-mesh 4.1, sidecar 3.2, mtls 2.9, ebpf 2.6, k8s 4.0…”, each term with a weight. The trick is that the model does the expansion: if your question says “service mesh” and the original book only said “Linkerd”, librarian B’s card did record “service-mesh 4.1” because he understood the relationship. To search, you intersect the question’s card with the book’s the old-fashioned way: inverted index, posting lists. It is decisive when the reader writes few, very specific words (product names, errors, jargon). This is the learned sparse embedder: SPLADE-v3 or the sparse head of bge-m3. It is the modern successor to BM25, not its rival; we will see why.

Librarian C, the copyist. He gives up on summarising. He takes every word on every page of the book and gives it a 128-dimension mini-RFID. He ends up with a 30,000-token book turned into 30,000 mini-RFIDs. When you search for something, the librarian compares every word of your question with every word of the book and keeps the maximum for each word of the question, adding them up. It captures nuance the other two lose by construction (proper nouns, numbers, specific phrasings), but its filing system is an order of magnitude larger in disk space. This is ColBERT-v2 / Jina-ColBERT-v2 / ColNomic-7B: late interaction, MaxSim.

And then there is the all-rounder librarian who does all three jobs in a single pass: bge-m3. A single 568 M parameter model that simultaneously returns the thematic RFID tag (1,024-d), the weighted list of terms (sparse) and the set of per-token mini-RFIDs (128-d). That is why it has settled in as the standard embedder for multilingual on-premise RAG: a single model.encode(chunk) produces the three outputs that feed hybrid retrieval without orchestrating three different models.

The same chunk indexed by three librarians. The all-rounder bge-m3 runs all three in a single 568 M parameter pass.

What an embedding really is

A text embedding is a function f : text → ℝᵈ trained so that two “semantically similar” texts produce nearby vectors in that space. The important part is not “vector”, any fixed-length hash is one, but what closeness means. Closeness is defined implicitly by the loss the model is trained with.

Almost every modern embedder is trained with InfoNCE (also called Multiple-Negatives Ranking Loss in sentence-transformers):

\mathcal{L}_{\text{InfoNCE}} = -\log \frac{\exp(\text{sim}(q, d^+)/\tau)}{\sum_{d \in \mathcal{B}} \exp(\text{sim}(q, d)/\tau)}

For each (query, doc⁺) pair the model has to assign the positive more similarity than all the other documents in the batch B, where the other documents act as free in-batch negatives. The temperature τ (typically 0.02–0.07, almost always 0.05) controls how sharp the distribution gets: lower τ means the model becomes more demanding about the positive but less stable. A large batch size |B| means many more negatives per gradient, so a better informed model. That is why embedders are trained with batch ≥ 1,024 on H100 clusters with AllGather across nodes to stack every negative in the cluster into a single effective batch.

On top of the in-batch negatives comes hard negatives mining: documents deliberately picked because they are “almost in the answer” (typically the next 10-100 neighbours from a previous BM25 / dense retrieval). Without hard negatives the model learns to discriminate the trivial and real quality on BEIR / MTEB drops 5-10 points.

This matters because the embedder family depends on what you feed into sim(·,·):

Family	`sim(q, d)`	Model output
Dense single-vector	dot product of two normalised 1024-d vectors	`f(q), f(d) ∈ ℝ^d`
Learned sparse	dot product of two 30,522-d vectors (≈80 non-zero each)	`f(q), f(d) ∈ ℝ^V`, `V =
Multi-vector	`Σᵢ maxⱼ ⟨qᵢ, dⱼ⟩` (MaxSim)	`f(q) ∈ ℝ^{

The loss is the same in all three cases, InfoNCE, but the geometry of the space changes, and with it quality, storage cost and search latency.

The three families in detail

Dense single-vector, the cartographer

The embedder reads the full chunk, passes it through an encoder transformer (XLM-RoBERTa, BERT, a Mistral decoder with a prompt) and aggregates the token representations into a single vector by:

CLS pooling: uses the embedding of the [CLS] token. Standard in BERT-base.
Mean pooling: plain average of the embeddings of all tokens. Standard in multilingual-e5-large-instruct, bge-m3 dense, Snowflake Arctic Embed L 2.0.
Last-token pooling: for decoder-LLM based embedders (e5-mistral-7b-instruct, gte-Qwen2-7B-instruct, NV-Embed-v2) that take the final token as the aggregate. It works because the model is causal and the last token “has seen” the whole context.
Latent-attention pooling: new in NV-Embed-v2. A learnable attention layer that weights tokens instead of averaging them. +2-3 MTEB points over mean pooling.

After pooling the vector is normalised to norm 1: v ← v / ‖v‖₂. With normalised vectors, cosine and dot product coincide:

\cos(q, d) = \frac{q \cdot d}{\|q\|\|d\|} = q \cdot d, \qquad \|q - d\|^2 = 2 - 2\,(q \cdot d)

That is why almost every vector DB indexes by inner product and leaves it to the user to normalise beforehand (Qdrant, Faiss IP, Milvus). If you forget to normalise the query vector but the corpus vectors are normalised, retrieval degrades silently: the query’s magnitude distorts the ranking. It is the number one bug in RAG in production.

A minimal numerical example, two normalised 4-d vectors:

q = [0.5, 0.5, 0.5, 0.5] ‖q‖ = 1
d₁ = [0.6, 0.4, 0.5, 0.5] ‖d₁‖ = 1.005, normalised [0.597, 0.398, 0.498, 0.498]
q · d₁ = 0.5·0.597 + 0.5·0.398 + 0.5·0.498 + 0.5·0.498 = 0.995

Closeness of nearly 1, as expected. If d₁ had not been normalised, q · d₁ = 1.0 and it would appear closer than any d that is perfectly aligned but with a norm < 1.005. Normalisation is not a detail: it is the contract of the space.

Learned sparse, the lexical descriptor

SPLADE-v3 (Naver, March 2024) has consolidated the modern version of the lexical librarian. Internally it is a small BERT (~110 M parameters, DistilBERT/BERT base) that produces, for each input token, a distribution over the whole vocabulary (30,522 dimensions in BERT WordPiece), and then does max-pool over the tokens:

w_j = \max_{i \in \text{seq}} \log\bigl(1 + \text{ReLU}(W_{ij})\bigr)

where Wᵢⱼ is the logit of input token i for vocabulary term j. The log(1+ReLU) saturates high logits (it stops a single word from dominating the vector) and the ReLU cuts the negatives. The result is a 30,522-dimension vector of which typically 50-200 entries stay non-zero.

The key part is the FLOPS regularization added to the loss during training:

\mathcal{L}_{\text{FLOPS}} = \lambda \cdot \sum_{j=1}^{V} \bar{w}_j^2, \qquad \bar{w}_j = \frac{1}{|B|}\sum_{i \in B} w_{ij}

It penalises the expected cost of the posting lists: if a vocabulary word appears on average in many documents, adding it to a new document costs double. The model learns to generate sparse vectors by construction.

What does this do to the text “Linkerd 3.8 brings mTLS by default”? The model does not only write the literal terms, it also writes, with a smaller but non-zero weight, “service-mesh”, “kubernetes”, “tls”, “sidecar”, “envoy”, “istio” (its competitor, also semantically related), “encryption”, “k8s”. That semantic expansion of the document is what sets SPLADE apart from BM25. BM25 only knows what was literally in the text; SPLADE knows what an expert would add as a descriptor.

In practice SPLADE-v3 beats BM25 by 3-6 MRR@10 points on MS MARCO and dominates BEIR zero-shot. The cost is ~2-4× the query latency of BM25 over the same inverted index, which static pruning can mitigate.

For the multilingual case, bge-m3 with its sparse head is the only maintainable option: SPLADE-v3 is trained in English and the multilingual ports are in an experimental state.

Multi-vector, the copyist

ColBERT-v2 (Stanford, NAACL 2022) introduced the late interaction paradigm. Instead of compressing the document to a single vector, it leaves it as a matrix (|d|, k) with a k-dimension vector per token. The similarity between query and document is computed token by token and aggregated with MaxSim:

s(q, d) = \sum_{i \in q} \max_{j \in d} \langle q_i, d_j \rangle

What is being computed: for each word of the query, find the document token that fits it best and add that similarity. The sum is over the query, not over the document. This lets a 30,000-token document compete fairly with a 200-token one, because the query always adds |q| terms.

Why does it give more quality than dense single-vector? Because compressing to one vector loses information about where each idea was. If the query is “which version brought mTLS by default in Linkerd”, the dense summary of the document only knows that the chunk is about “Linkerd and mTLS”; ColBERT’s copyist can match “which version” with “3.8” because it stores the embedding of the 3.8 token separately. On BEIR / out-of-domain, late interaction beats single-vector by between +2 and +6 nDCG@10 with the same backbone.

The price is storage. Per document, a 1024-dimension dense single-vector in fp16 takes 1024 × 2 = 2 KB. ColBERT-v2 with 128-dimension tokens for a 256-token chunk takes 256 × 128 × 2 = 65,536 B ≈ 64 KB: 32× more space. With ColBERT-v2’s nbits=2 residual compression it drops to ~16 KB (8×). Jina-ColBERT-v2 adds Matryoshka over the token dimensions (truncatable to 128 / 96 / 64), cutting another 50%.

For 1 million chunks:

Family	Per doc	Total
Dense fp32 (1024-d)	4,096 B	4.0 GB
Dense fp16 / halfvec	2,048 B	2.0 GB
Dense int8 (SQ)	1,024 B	1.0 GB
Dense binary (1 bit/d)	128 B	128 MB
SPLADE (≈ 80 terms × 8 B)	~640 B	~640 MB
ColBERT fp16 (256 tok × 128 d)	65,536 B	64 GB
ColBERT residual `nbits=2`	~16,000 B	~16 GB
Jina-ColBERT-v2 (MRL token 64)	~8,000 B	~8 GB

ColBERT in on-premise production is reserved for corpora of up to a few million chunks, or applied only as a reranker over the top-100 from the first committee (dense + sparse), as described in the reranker post.

Matryoshka, the truncatable dimension

An operational lever that changed the conversation about embeddings between 2024 and 2026 is Matryoshka Representation Learning (Kusupati et al., NeurIPS 2022). The trick: during training, on top of the loss over the full D-dimension vector, the same loss is computed over prefixes of the vector:

\mathcal{L}_{\text{MRL}} = \sum_{k \in \{64, 128, 256, 512, 1024\}} \alpha_k \cdot \mathcal{L}_{\text{InfoNCE}}\bigl(\text{emb}[:k]\bigr)

The first k dimensions of the embedding are trained to be, on their own, a valid embedding. At inference time, if you want a cheaper embedding, you truncate the vector: the first quarter is already a usable embedding. Without Matryoshka, truncating wrecks the geometry: the first 256 dimensions of an embedding trained only at 1024-d encode nothing coherent.

Typical degradation in MTEB nDCG@10 when truncating an MRL embedder:

Truncation	Average loss
1024 → 512	-1 to -2 points
1024 → 256	-3 to -5 points
1024 → 128	-5 to -8 points
1024 → 64	-8 to -12 points

Native MRL models in 2026 (the ones that let you pick the dimension at runtime without retraining):

jina-embeddings-v3 (1024 → 32, fine steps, CC-BY-NC-4.0)
jina-embeddings-v4 (2048 → 128, multimodal text+image, CC-BY-NC-4.0)
nomic-embed-text-v2-moe (768 → 256, Apache 2.0)
Snowflake-arctic-embed-l-v2.0 (1024 → 256, Apache 2.0)
mxbai-embed-large-v1 and mxbai-embed-2d-large-v1 (the latter also truncatable in layer depth)
Stella_en_1.5B_v5 (multiple steps 512 / 768 / 1024 / 2048 / 4096 / 6144 / 8192, MIT, English only)
text-embedding-3-large (OpenAI, 3072 → 256, API only)
voyage-3 family (1024 → 256 / 512 / 1024 / 2048, API only)

Production recommendation: always use an MRL model even if you do not truncate at the start, because it simplifies the future quantisation decision. And above all, evaluate truncation on your real corpus: the average MTEB degradation of “-3 points” becomes “-12 points” in a niche domain.

The 2026 open source model zoo

What follows is a spec sheet per model. Verified as of 2026-06: HuggingFace cards + reference papers + MTEB / MMTEB leaderboard.

Dense single-vector

Model	Params	Dim	Tokens	Languages	Licence	Distinctive
`BAAI/bge-m3` (dense)	568 M	1024	8192	100+	MIT	Tri-mode (dense + sparse + colbert) in one forward. De facto standard for multilingual on-prem.
`Snowflake/snowflake-arctic-embed-l-v2.0`	568 M	1024 (MRL → 256)	8192	~100	Apache 2.0	Trained for multilingual + English without degrading either. MIRACL 55.8.
`intfloat/multilingual-e5-large-instruct`	560 M	1024	512	~100	MIT	Veteran multilingual baseline. Short window.
`intfloat/e5-mistral-7b-instruct`	7.1 B	4096	4096	English	MIT	First decoder-as-embedder to break MTEB. English.
`Alibaba-NLP/gte-Qwen2-7B-instruct`	7 B	3584	32,768	100+	Apache 2.0	The only one with 32k context. Top MTEB-en, strong multilingual.
`nvidia/NV-Embed-v2`	7.85 B	4096	32,768	English	CC-BY-NC-4.0	Latent-attention pooling. Top quality. Licence blocker in prod.
`Linq-AI-Research/Linq-Embed-Mistral`	7 B	4096	4096	English	CC-BY-NC-4.0	Top MTEB retrieval Aug 2024. Non-commercial.
`NovaSearch/stella_en_1.5B_v5`	1.54 B	8192 (multiple MRL)	8192	English	MIT	Small + rich MRL. English.
`BAAI/bge-multilingual-gemma2`	9 B	3584	8192	100+	Gemma	High quality, the Gemma licence restricts redistribution.
`BAAI/bge-en-icl`	7 B	4096	8192	English	MIT-style	In-context learning from examples in the prompt.
`mixedbread-ai/mxbai-embed-large-v1`	335 M	1024 (MRL)	512	English	Apache 2.0	MRL + native binary. Short window.
`jinaai/jina-embeddings-v3`	570 M	1024 (MRL → 32)	8192	89	CC-BY-NC-4.0	Per-task LoRA. Non-commercial without a licence.
`nomic-ai/nomic-embed-text-v2-moe`	475 M / 305 M active	768 (MRL → 256)	512	~100	Apache 2.0	First general-purpose MoE in embeddings.

Learned sparse

Model	Params	Vocab	Tokens	Languages	Licence	Distinctive
`naver/splade-v3`	110 M	30,522	512	English	CC-BY-NC-SA-4.0	SOTA learned sparse. Non-commercial.
`BAAI/bge-m3` (sparse head)	568 M	XLM-R vocab	8192	100+	MIT	The only license-clean multilingual option.

Multi-vector (late interaction)

Model	Params	Dim/token	Tokens	Languages	Licence	Distinctive
`colbert-ir/colbertv2.0`	110 M	128	512	English	MIT	The original paper, the basis for everything.
`jinaai/jina-colbert-v2`	560 M	128 / 96 / 64 (MRL)	8192	89	Apache 2.0	The license-clean multilingual multi-vector.
`nomic-ai/colnomic-embed-multimodal-7b`	7 B	3584 (Qwen2-VL)	—	~100	Apache 2.0	Multimodal text+image multi-vector. Vidore-v2 open SOTA.

The leaderboard, with caution

MTEB / MMTEB (Massive Multilingual Text Embedding Benchmark, Enevoldsen et al., arxiv 2502.13595) is the standard thermometer. Top retrieval on MMTEB in mid-2026 is dominated by Qwen3-Embedding-8B (~70.6 multilingual avg) and Llama-Embed-Nemotron-8B. Below them, the 7B models (gte-Qwen2-7B, NV-Embed-v2) and the 568M ones (bge-m3, Snowflake-Arctic-L-2.0) compete task by task.

Trap: MTEB has started to saturate through dataset contamination. The higher the ranking, the more likely it is that the model saw subsets of the evaluation datasets during training. The rule in production: the leaderboard is for discarding bad models, not for picking the best one. The final decision is made on your own domain eval set, generated with the LLM-as-judge or evals recipe.

The specific problem of multilingual Spanish

For a Spanish customer serving corporate, legal or support documentation in Spanish (and often with Catalan / Portuguese / English mixed in), the embedder zoo narrows to fewer than six viable models. The operational exclusions:

English-only models: e5-mistral-7b-instruct, stella-en-1.5B-v5, Linq-Embed-Mistral, mxbai-embed-large-v1, NV-Embed-v2, SPLADE-v3. They cut Spanish performance below an acceptable level: translating the query into English before searching is one route, but it introduces latency, tokenisation drift and another model dependency.
Models with a non-commercial licence: jina-embeddings-v3, jina-embeddings-v4, NV-Embed-v2, Linq-Embed-Mistral. They work for a PoC, but commercial production needs an explicit agreement with the vendor. Unless you have the licence signed, they have to be excluded.
Models with a Gemma licence: bge-multilingual-gemma2. Allowed for internal use, awkward to redistribute weights to a customer.

The ones left, ordered by practical choice in sovereign production:

BAAI/bge-m3 — MIT, 568 M, 100+ languages (including Spanish and Catalan, explicitly trained), 8,192 tokens, tri-mode dense+sparse+colbert. A reasonable default. Fits in an RTX 4090. TEI and Infinity serve it natively.
Snowflake/snowflake-arctic-embed-l-v2.0 — Apache 2.0, same size, explicit Matryoshka, better MIRACL/CLEF than bge-m3 on some multilingual tasks, no sparse head. If the priority is pure MMTEB in Spanish.
intfloat/multilingual-e5-large-instruct — MIT, 560 M, veteran baseline. Its 512-token window is the big limitation: long documents have to be split first. If what you already have in production works, do not migrate out of fashion.
Alibaba-NLP/gte-Qwen2-7B-instruct — Apache 2.0, 32k context, high quality in Spanish (Qwen2 is well trained on Spanish). If the chunks are long (more than 4k tokens) and you have the GPU to serve it (it does not fit in a 4090; it does in an H100). It fits alongside an LLM on an H100 80GB with care.
nomic-ai/nomic-embed-text-v2-moe — Apache 2.0, 305 M active, MRL, ~100 languages. If latency and cost per token rule: the MoE gives it disproportionate throughput for its quality.
jinaai/jina-colbert-v2 — Apache 2.0, multilingual multi-vector, as a reranker or as the main retrieval on a small corpus (< 1 M chunks). The only license-clean multi-vector in Spanish.

The rule of thumb: bge-m3 as dense+sparse on the front line, jina-colbert-v2 as a third reranking layer when the use case warrants it, and Snowflake Arctic L 2.0 as the alternative if the corpus-specific eval prefers its geometry.

Serving embeddings on-premise

Three engines split the on-prem embedding serving landscape in 2026, with different profiles.

Text Embeddings Inference (TEI), the standard

huggingface/text-embeddings-inference is a server written in Rust with a Candle / ONNX backend, integrated FlashAttention and dynamic batching by tokens. It exposes an OpenAI-compatible /v1/embeddings API and supports the three bge-m3 modes simultaneously from version 1.5 onwards. For multilingual production it is the obvious default.

# values.yaml — TEI serving multilingual bge-m3 on an RTX 4090
image: ghcr.io/huggingface/text-embeddings-inference:1.5
args:
 - --model-id=BAAI/bge-m3
 - --pooling=cls
 - --max-batch-tokens=16384
 - --max-concurrent-requests=512
 - --dtype=float16
resources:
 limits:
 nvidia.com/gpu: 1

Indicative throughput with bge-m3, fp16, 512-token sequence, batch 32:

RTX 4090 (24 GB): ~8–15 k tokens/s
A100 80 GB: ~60 k tokens/s sustained
H100 80 GB: ~40–80 k tokens/s, with fp8 about 50% more

(The ranges are approximate and depend on the real batch, the average sequence length and compilation with FA2/FA3.)

Infinity, the flexible one

michaelfeil/infinity (MIT) is a multi-model FastAPI server able to load bge-m3, Snowflake Arctic, Jina-v3, Nomic, ColPali, CLAP and rerankers simultaneously behind the same OpenAI-style API. Backend PyTorch + Optimum (ONNX/TensorRT) or CTranslate2. Useful when you need to serve several different embedders (one for text, another for code, another for images) behind a single endpoint, or when the model does not yet have TEI support.

vLLM `--task embed`, for the 7B embedders

When the embedder is really an LLM decoder turned into an embedder (e5-mistral-7b-instruct, gte-Qwen2-7B-instruct, NV-Embed-v2, Stella-1.5B), the natural place to serve it is vLLM, which already has the PagedAttention and continuous batching stack in production:

vllm serve Alibaba-NLP/gte-Qwen2-7B-instruct \
 --task embed \
 --dtype bfloat16 \
 --max-model-len 32768 \
 --trust-remote-code

vLLM detects the correct pooling (last-token on the Qwen / Mistral based ones) and exposes an OpenAI-compatible /v1/embeddings. For inference clusters already running vLLM with an LLM on another port, it is the natural way to serve the embedder without standing up another stack.

fastembed, the lightweight one

qdrant/fastembed loads bge-small, MiniLM, ColBERT and BM25/SPLADE sparse in ONNX-CPU. It is not competitive on throughput against TEI/Infinity with a GPU, but it is the right option when you have to serve embeddings on a NUC node without a GPU (see mixed NVIDIA + Intel environments) or when the embedder is part of the client (a preview in a UI, prior scoring at the edge).

Storage, quantisation and the corpus arithmetic

The embedding does not stay in the embedder’s memory: it lives in the vector DB index and is materialised every time you ingest a new chunk. The storage cost calculation is what decides the final dimension, not MTEB quality. For 1 million chunks with a dense embedder at 1024-d:

fp32 : 1,024 dims × 4 B × 1 M = 4,096 MB ≈ 4.0 GB
fp16/halfvec: 1,024 dims × 2 B × 1 M = 2,048 MB ≈ 2.0 GB
int8 (SQ) : 1,024 dims × 1 B × 1 M = 1,024 MB ≈ 1.0 GB
binary : 128 B × 1 M ≈ 128 MB

The quantisation options in order of real use (mid-2026):

halfvec (fp16): the default in pgvector 0.7+ and in any serious vector DB. Zero MTEB loss, 2× compression. Always turn it on.
Scalar Quantization int8 (SQ): each vector component is mapped to int8 with a global min/max. Typical recall@10 loss: 0–1%. 4× compression. The Qdrant default, supported in Milvus and Weaviate.
Binary quantisation: bit = sign(v_i - μ_i). 32× raw compression. Cold loss 5–15%. Mitigated with a Hadamard / TurboQuant pre-rotation (Qdrant 1.18, Dec 2025): it pre-multiplies by a random orthogonal matrix that spreads the energy across dimensions before binarising. After TurboQuant the loss drops to 1–3%. It also combines with rescoring over the original fp16 for the top-100 candidates.
Product Quantization (PQ): the FAISS classic. Up to 64× compression, 2–5% loss. More complex to operate (it needs a trained codebook); in 2026 it has given ground to binary + rescoring.

A corpus of 100 million chunks (a real figure for a large corporate RAG) with dense bge-m3:

Format	Total
fp32	400 GB
fp16	200 GB
int8	100 GB
binary + Hadamard	12.5 GB

The difference between 200 GB and 12.5 GB is the difference between needing a dedicated vector DB node with 8 NVMe in RAID and fitting in the RAM of a single node. For large corpora, quantisation is no longer an optimisation: it is the only way to operate.

The vector DB integration

The vector DBs of 2026 have become hybrid DBs that index all three types at once. The quick map:

Vector DB	Native hybrid	Multi-vector / ColBERT	Quantisation
Qdrant ≥1.10	RRF/DBSF in `query_points` with dense + sparse + colbert in one collection	Yes, native (one-shot MaxSim)	SQ int8, binary, TurboQuant 1.18
Weaviate	`hybrid(alpha=0.75)` BM25 + dense, multi-target named vectors	Yes, as a multi-vector named vector	PQ, SQ, rotational 8-bit BBQ
Milvus ≥2.4	Multi-vector + sparse in the schema; 2.5 adds native BM25 full-text	Multi-vector field, MaxSim orchestrated from the client	SQ, PQ, CAGRA GPU
pgvector 0.7+/0.8	`halfvec`, `sparsevec`, `bit`; HNSW for all three	Not native (separate-table workaround)	`binary_quantize()`, halfvec, rescoring with exact `<#>`
Elasticsearch / OpenSearch	`sparse_vector` (ELSER, SPLADE) + `dense_vector` HNSW; RRF	OpenSearch 3.x yes	ES 9: `int8_hnsw` by default, BBQ binary quantization

For sovereign on-prem production in Spanish, the easiest combination to operate in 2026 is Qdrant + bge-m3: a single collection indexes the three modes of the same model, the hybrid query with RRF is done in one call, TurboQuant quantisation brings the corpus down to manageable levels, and the operator is a Go binary with simple backups to S3/MinIO. pgvector + bge-m3 is the other reasonable option when you already have Postgres with HA and do not want to add a second DB to the operational inventory; you lose native multi-vector, but you gain cross-cutting SQL over the chunks.

The HNSW parameters you absolutely have to touch:

M: connections per node in the graph. 16–32 typical. Higher means more recall, more RAM. For small corpora (<1 M) M=16; for medium corpora (10 M) M=24; for large corpora M=32+IVF-PQ or M=32+binary.
ef_construction: search width during construction. 100–400. Higher means a better graph, slower construction. Build with ef_construction=400 even if it is slow; you pay it once.
ef_search: width during the query. 50–200. The main knob for the recall/latency trade-off at runtime. Start at 64 and measure.

Implications for on-premise inference

The embedder does not share hardware with the LLM as comfortably as it might seem. The numbers:

bge-m3 (568 M) takes about 568 × 2 = 1,136 MB in fp16 for the weights, plus the batch KV cache, plus temporary activations. In practice it is served comfortably in 6–8 GB of VRAM even at a high batch. It fits alongside a 7B-Q4 LLM on an RTX 4090.
gte-Qwen2-7B-instruct needs ~14 GB fp16 for weights alone. It does not fit next to a 7B LLM on a 4090; on an H100 80 GB it does, with care over simultaneous batching.
jina-colbert-v2 (560 M) takes ~1.1 GB of weights, but the storage of the multi-vector index is the real cost: 8 GB per million chunks even with Matryoshka and compression.

On the RTX 4090 (24 GB)

A realistic minimum stack for a Spanish RAG with a corpus under 1 M chunks:

GPU 24 GB ┐
 ├─ TEI bge-m3 (dense + sparse + colbert) │ ~6 GB VRAM, ~12 k tok/s
 └─ vLLM Qwen2.5-7B-Instruct AWQ Q4 │ ~8 GB VRAM, ~80 tok/s
CPU/RAM ┐
 ├─ Qdrant with bge-m3 dense + sparse + colbert │ ~3 GB RAM per M chunks
 └─ FastAPI gateway (LiteLLM)

It serves a few tens of RAG QPS with decent multilingual quality. It is the PoC configuration and the deployment for a small site.

On the 4×H100 80 GB cluster

For the production case with several million chunks and an SLO of p99 < 500 ms:

H100 #1 (80 GB) ── vLLM Qwen3-72B-Instruct AWQ + Qwen2.5-7B speculative ┐
H100 #2 (80 GB) ── vLLM gte-Qwen2-7B-instruct (embedding 32k ctx) │ LLM + large embed
H100 #3 (80 GB) ── TEI bge-m3 multi-tenant + jina-colbert-v2 reranker │ medium embed
H100 #4 (80 GB) ── Hold-out for canary / shadow │ see canary post
 ┘
Qdrant cluster (3 CPU nodes + NVMe) ── 100 M chunks indexed (binary + TurboQuant + rescoring)

This configuration separates the large LLM from the large embedder (they share the Qwen2 architecture but compete for VRAM if put on the same GPU) and leaves a whole H100 for variants in canary. bge-m3 fits with room to spare alongside the reranker on a single H100, serving tens of thousands of requests/min.

The seven operational traps of the embedder

Not normalising the query vector. Cosine and dot product coincide only when both vectors are unit vectors. If you forget v ← v / ‖v‖₂ in the client, the results are “almost right”, the top-1 is still correct on trivial queries, the top-10 no longer is, and nobody notices until RAG quality drops 8 points. Fix: bake normalisation into the embedder adapter, not into the client.
Chat template slipped into the embedder. Some LLM-based embedders (e5-mistral-7b-instruct, gte-Qwen2-7B) expect a specific instruction prompt before the text to embed ("Instruct: Retrieve relevant passages\nQuery: ..."). Forgetting it leaves performance ~5 MTEB points lower. Fix: read the usage_template in the model card and put it into the embedder wrapper.
Badly chosen Matryoshka dimension. The default in many Qdrant / pgvector clients is dim=768. If your embedder is natively MRL at 1024 → 768, fine. If it is 1024 without MRL, truncating to 768 wrecks the space (typical loss -8 MTEB points). Fix: use the model’s native dim and truncate only when storage demands it, and only on MRL models.
Missing hard negatives in fine-tuning. When the embedder is fine-tuned with your own data (which should be standard practice for corporate RAG), if the mini-batch only carries positives and in-batch negatives from the same domain, the model learns that anything outside the domain is negative, but inside the domain it does not discriminate. Fix: mine hard negatives with BM25 / dense from your own corpus before fine-tuning.
Corpus drift without reindexing. When you retrain or replace the embedder but only apply the new model to new chunks, you end up with the index mixing two incompatible geometries. Chunks from the old and the new model are not comparable by cosine. Fix: every change of embedder is a full reindexing of the corpus, planned as an operational retrain.
Tokenizer drift between client and model. The Python client that prepares the queries uses its own tokenizer (sometimes tiktoken by default) and truncates at 8,192 tokens. The embedder uses XLM-R with sentencepiece and truncates at 8,192 of its own tokenizer. Long queries are truncated differently; the corpus embeddings are consistent but the query ones are not. Fix: use the model’s tokenizer in the client or in the wrapper.
MTEB overfit as a selection guide. The MTEB leaderboard has become a contaminated metric: there is evidence that leading models saw subsets of the evaluation datasets during training. The model that is +0.5 points above the second is not necessarily better for your domain. Fix: your own domain eval set (100-300 labelled query-doc pairs) run with the evals recipe decides.

License-clean stack for sovereign production

Let us write down the recommended ending. For a Spanish organisation serving corporate RAG on-prem under ENS / ISO 42001 / EU AI Act, with a corpus of 1-50 M chunks in Spanish + English + Catalan:

Layer	Component	Licence	Rationale
Dense embedder	`BAAI/bge-m3`	MIT	Robust multilingual, 8k tokens, license-clean, served by TEI
Sparse embedder	`bge-m3` sparse head	MIT	Same pass as the dense one, no second model needed
Layer 2 reranker	`BAAI/bge-reranker-v2-m3`	MIT	Multilingual cross-encoder from the same team
Layer 3 reranker (optional)	`jinaai/jina-colbert-v2`	Apache 2.0	License-clean multilingual multi-vector
Embed server	`TEI` + `Infinity` for multi-model	Apache 2.0 / MIT	Supported stack
Vector DB	`Qdrant` (preferred) or `pgvector 0.8`	Apache 2.0 / PostgreSQL	Native hybrid + quantisation
Quantisation	`int8 SQ` + `binary + TurboQuant + rescoring`	Apache 2.0	Cuts the corpus 16×–32× with < 3% loss
Minimum hardware	RTX 4090 24 GB	—	For PoC and small sites
Production hardware	4×H100 80 GB cluster	—	For RAG with an SLO of p99 < 500 ms

The alternative stack, if explicit MMTEB in Spanish weighs more than the tri-modality of bge-m3: replace bge-m3 with Snowflake/snowflake-arctic-embed-l-v2.0 (Apache 2.0, MRL → 256) and explicitly add SPLADE-v3 or plain BM25 for the sparse layer. It loses the elegance of the single forward and gains 1-2 points on Spanish MIRACL.

Conclusion

The embedder is the easiest piece of a RAG to oversimplify and the one that decides real quality the most once everything else is in place. The three families (dense, sparse, multi-vector) are not three options to choose between but three trades that bge-m3 performs in a single pass and that hybrid retrieval consumes in parallel. The maths that matters is modest, InfoNCE with τ, MaxSim, FLOPS regularization, MRL, but the operational traps are many and quiet: forgotten normalisation, missing chat template, badly chosen Matryoshka dimension, tokenizer drift. For sovereign production in Spanish the list of viable models fits in fewer than ten, and the real decision comes down to “bge-m3 or Snowflake Arctic L 2.0”, with jina-colbert-v2 added as layer three when the fine-grained quality justifies the cost. The license-clean stack fits on an RTX 4090 for a PoC and scales to a 4×H100 cluster for real production.

References

Chen et al. M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. arXiv:2402.03216. https://arxiv.org/abs/2402.03216
Sturua et al. Jina Embeddings v3: Multilingual Embeddings With Task LoRA. arXiv:2409.10173. https://arxiv.org/abs/2409.10173
Günther et al. Jina Embeddings v4: Universal Embeddings for Multimodal Multilingual Retrieval. arXiv:2506.18902. https://arxiv.org/abs/2506.18902
Nussbaum et al. Nomic Embed v2: Multilingual Mixture of Experts. arXiv:2502.07972. https://arxiv.org/abs/2502.07972
Wang et al. Improving Text Embeddings with Large Language Models (E5-Mistral). arXiv:2401.00368.
Yu et al. Arctic-Embed 2.0: Multilingual Retrieval Without Compromise. Snowflake, 2024-12. https://www.snowflake.com/blog/arctic-embed-2-multilingual/
Lee et al. NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models. arXiv:2405.17428.
Khattab y Zaharia. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. SIGIR 2020.
Santhanam et al. ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction. arXiv:2112.01488.
Jha et al. Jina-ColBERT-v2: A General-Purpose Multilingual Late Interaction Retriever. arXiv:2408.16672.
Lassance et al. SPLADE-v3. arXiv:2403.06789.
Kusupati et al. Matryoshka Representation Learning. NeurIPS 2022, arXiv:2205.13147.
Enevoldsen et al. MMTEB: Massive Multilingual Text Embedding Benchmark. arXiv:2502.13595.
Hugging Face Text Embeddings Inference. https://github.com/huggingface/text-embeddings-inference
Michael Feil. Infinity. https://github.com/michaelfeil/infinity
Qdrant. TurboQuant 1.18 release notes. https://qdrant.tech/articles/turboquant-quantization/
pgvector. Release notes 0.7 / 0.8. https://github.com/pgvector/pgvector
Hugging Face. Embedding Quantization. https://huggingface.co/blog/embedding-quantization

Anatomy of an on-premise LLM inference stack: the seven layers that have to hold each other up

Sat, 30 May 2026 16:00:00 +0200

TL;DR

The on-premise LLM inference stack is not a model server: it is a building of seven layers that hold each other up. The inference layer (vLLM / SGLang) serves tokens, but without a dedicated embeddings layer RAG does not work; without the gateway layer the client couples its SDK to the specific engine; without an LLM-aware observability layer (Langfuse + OpenTelemetry GenAI) any quality degradation goes unnoticed; without a GitOps control plane layer (Flux + Forgejo) any manual change leaves invisible debt; and without a dependency tracking layer (Hubble + intent-based policies) decommissioning a Service silently breaks applications nobody remembers were using it. This post is born from a concrete incident: a pipeline that reported status: completed and matched_jobs: 0 for days because it kept calling an Ollama that had already been scaled to zero, while a badly written except labelled the generic ConnectError as “ChromaDB indexing error” and the vector store, innocent, took the blame. Each of the three symptoms was the cry of a layer that was missing or badly designed. The body of the post walks the seven layers with their canonical OSS piece, the design decisions that break them, the sizing maths on a generic 4×H100 SXM cluster (320 GB of VRAM, NVLink) and a final diagram of the connected stack. The thesis: a stack that passes the incident test is not measured by its peak throughput, it is measured by how long it takes to shout when something drifts from the design.

You are here: the seven layers seen from above

Before going into detail, the map. The seven layers are not seven servers: they are seven responsibilities the stack has to cover. A layer can collapse into one pod (gateway) or spread across several components (observability = traces + metrics + flow logs). What it cannot do is be missing.

Layers 1–4 sit on the request path: if they fall, the client notices in seconds. Layers 5–7 sit on the path of design and operations: if they fall, there is no immediately visible error, and that is why they are the ones that produce silent incidents, which is how most serious incidents end up. This post argues that the quality of the stack is measured in layers 5, 6 and 7, because 1–4 are commodities where everyone picks roughly the same pieces.

The analogy: the office building with shared services

Picture a twelve-storey office building with several tenants. It has a concierge desk (gateway: filters who comes in), it has lifts (LLM inference: they move the heavy load), it has stairs and a goods lift (embeddings: light but constant traffic, hardly anyone notices them until they break), it has plumbing and water tanks (vector store: what actually holds the state), it has a distribution board and sensors (observability: what warns you when something is drawing more than planned), it has a building manager (GitOps control plane: the only legitimate authority to move anything), and it has a tenant register (dependency tracking: who is connected to which shared service).

When a new client moves into the third floor and asks to install a server that needs more amperage than planned, the problem is not electrical: it is administrative. If the client can go straight to the board and plug in whatever they like, the building survives for a while and then a breaker trips at three in the morning. If the client has to go through the manager, the manager consults the tenant register (is anyone else hanging off that same circuit?), reviews the electrical planning (are we at the limit?) and either authorises or redirects. The building stays standing not because of its electrical installation but because of the discipline of going through the manager.

The LLM inference stack works identically. The physical layers (1–4) are the visible ones, the ones marketing people put on the slide. The governance layers (5–7) are the ones that distinguish a platform from a pile of pods with good luck.

Now to the incident that motivates the whole post.

The hook: the log that lied for six days

The application is called jobhunter internally. It is a cron pipeline that every six hours sweeps public job-ad sources, filters by EU geography, embeds the new ads, indexes them in a vector store and matches them against search profiles. The last step fires notifications.

For six days the pipeline reported the same thing on every run:

status: completed
total_found: 756
new: 23
matched_jobs: 0 ← zero, run after run

And in the logs:

[INFO] httpx: POST chromadb:8000/api/v2/.../collections/<id>/delete → 200 OK
[INFO] matcher: Purged 12 expired jobs from ChromaDB
[ERROR] pipeline: ChromaDB indexing error: All connection attempts failed
[INFO] httpx: POST chromadb:8000/api/v2/.../collections/<id>/get → 200 OK
[ERROR] pipeline: Matching error: All connection attempts failed

This is a log that invites you to blame ChromaDB. And in fact the first post-mortem written internally pointed at a version incompatibility between the Python client and the v2 server. A reasonable hypothesis, technically plausible, completely false.

The real cause: weeks earlier the general LLM on the platform had been migrated from Ollama to vLLM. The migration was clean for the two applications that depended directly on the large model, their manifests were pointed at the new endpoint. What nobody did was look at who else was calling the Service ollama.ollama.svc:11434. jobhunter called it to generate the embeddings for the ads. When the Ollama deployment was scaled to zero, the Service was left empty, and any outbound connection received a generic ConnectError("All connection attempts failed") from httpcore. The try/except wrapping the whole matching stage caught the exception and labelled it “ChromaDB indexing error”, because that was the lexical wrapper of the block, not because ChromaDB had anything to do with it. ChromaDB was answering 200 to delete and get in those same logs.

Three factors kept the incident alive for six days:

The except named the wrapper rather than the stage that failed. The log said “ChromaDB indexing error” when the error was an embeddings call against a nonexistent Service.
The pipeline returned status: completed even when there were errors. No alerting based on status fired. The metric that would have fired (matched_jobs stuck at zero) was not instrumented.
The pipeline image ran as :latest with no versioning, no SBOM, no reproducibility. When it started failing, it was impossible to know which Ollama client it had been built with.

Each of the three symptoms is the cry of a layer that was missing. The first asks for LLM-aware observability that distinguishes stages (layer 5). The second asks for the pipeline logic and the Prometheus metrics to behave like contracts (layer 5, SLI/SLO dimension). The third asks for GitOps with pinned images and an auditable SBOM (layer 6). And the whole incident, decommissioning a Service without knowing who consumes it, shouts dependency tracking (layer 7).

The rest of the post walks the seven layers through that lens: what each one solves, which OSS piece implements it in 2026, and which typical design decision breaks it.

Layer 1 — Gateway: the client SDK must not couple to the engine

What it has to solve. That the client sends POST /v1/chat/completions with the standard OpenAI SDK and never learns which engine (vLLM, SGLang, TensorRT-LLM), which specific model, which LoRA adapter or even which GPU pool is serving the request. Authentication, rate limit, routing by body.model and by tenant, header injection for tracing.

Canonical OSS piece in 2026. Envoy AI Gateway (Envoy with GenAI extensions: routing by model, token-based rate limit, fallback chains) or Cilium Gateway API with your own filters. For cases where the client wants multi-provider without distinguishing on-prem from SaaS, LiteLLM Proxy is the lightweight equivalent.

What breaks the layer. Exposing the inference engine endpoint directly. If clients call http://vllm-prod.svc:8000, any change of engine, model or pool forces you to touch the code of every app. The rule: the engine changes, the contract does not. The OpenAI SDK is standard; the routing behind the gateway is where design freedom lives.

Where jobhunter failed. There was no gateway. The app called ollama.ollama.svc:11434 directly. When Ollama died, there was no intermediate layer that could answer with a fallback, a retry against another pool, or at least a 503 error with a descriptive body.

Layer 2 — LLM inference: vLLM as the default choice, SGLang when prefix caching rules

What it has to solve. Serving tokens with throughput and latency under control: continuous batching, PagedAttention so the KV cache does not fragment the VRAM, FP8 so a 32B model fits with room to spare, multi-LoRA for per-tenant customisation without replicating the base, structured output for function calling with schema guarantees.

Canonical OSS piece in 2026. vLLM is the default choice: it covers the state of the art (PagedAttention, continuous batching, FlashAttention, FP8 quantisation, multi-LoRA, structured output, speculative decoding). SGLang comes in when the workload has high prefix caching (long chat with large system prompts, agents with repeated instructions), because its RadixAttention composes better than standard prefix caching. For very specialised inference with proprietary NVIDIA kernels, TensorRT-LLM, accepting the hardware lock-in.

What breaks the layer. Assuming a single model “does everything”. A 32B chat LLM does not serve /v1/embeddings; if you try, vLLM answers BadRequestError: "The model does not support Embeddings API". Assuming it did was one of the concrete wounds of the incident: the app expected an endpoint the model did not implement.

The design decision that costs more later. Serving the LLM with aggressive quantisation (INT4) without a quality eval calibrated for your corpus. INT4 with AWQ or GPTQ saves VRAM, but it degrades answers in technical or legal Spanish in a measurable way. The rule: any quantisation change goes through the same golden eval as a model change.

Layer 3 — Embeddings: separate from the LLM, fixed dimension, a life of its own

What it has to solve. Generating dense vectors for the RAG corpus and for retrieval queries. The embeddings model is a different thing from the chat LLM: different architecture (encoder, not decoder), different size (hundreds of millions of parameters, not tens of thousands of millions), different API (an /embeddings endpoint that takes text and returns a fixed-dimension vector).

Canonical OSS piece in 2026. Infinity or Hugging Face Text Embeddings Inference (TEI) to serve models of the bge-*, multilingual-e5-*, nomic-embed-* families with high throughput and multi-model support. OpenVINO Model Server when Intel hardware is available. sentence-transformers as a fallback embedded in the application itself when the corpus is small and the deployment is constrained.

What breaks the layer. Treating it as “the LLM does that too”. It does not if it is chat-only; and even when it does (some models have a dual endpoint), mixing chat and embeddings serving in the same process punishes the throughput of both. The physical separation is the design.

The technical fact people forget. The vector store and the embeddings model form an indivisible unit. Changing the model from multilingual-e5-large (1024 dim) to multilingual-e5-small (384 dim) is not a substitution: it is creating a new collection (my_corpus_v2) and re-embedding the whole corpus. If you upsert into the old collection, you hit a dim mismatch at runtime that takes the pod down. This seems obvious and is violated constantly, because the embeddings model is chosen once and then forgotten.

Layer 4 — Vector store + relational data + storage: what actually holds the state

What it has to solve. Persisting the vectors with efficient filters (tenant_id, created_at, source), persisting the relational metadata (users, configs, versioned prompts, traces), and persisting the model weights, the LoRA adapters, the datasets and the original corpora.

Canonical OSS pieces in 2026.

Qdrant for large collections (>200 k vectors), payload-aware filters, multi-tenancy via collection or via field.
pgvector for small collections with mandatory relational joins (wanting to do WHERE doc.author = ... AND vector <=> $1 in the same SQL).
PostgreSQL operated by CloudNativePG (CNPG) for the relational side: Barman Cloud backups, replication, connection via pooler.
MinIO for S3-compatible objects: bucket per tenant, cross-site replication, weights and adapters versioned by sha256.
Redis for queues, rate-limit counters and a cache of frequent retrievals.

What breaks the layer. Assuming the vector store is stateless ephemeral. It is exactly the opposite: it is the component where losing state costs the most. Without verified backups of the vector store, an index corruption forces you to re-embed the whole corpus, and on a corpus of millions of documents that is hours or days of GPU.

The design decision that pays later. Forgetting to version the collection by dimension and embeddings model. Suggested convention: my_corpus__embed-multie5l__1024d__v3. The name carries metadata; any change in any of the three attributes forces a new collection. It is ugly but it protects against an accidental upsert with the wrong dim.

Layer 5 — Observability: LLM-aware traces + infrastructure metrics + flow logs

What it has to solve. That any inference can be recovered from its trace_id, with all the gen_ai.* attributes (semantic conventions) plus your own attributes (tenant_id, adapter_id, priority_tier), a breakdown of latency (queue → prefill → decode → network), tokens consumed, tools invoked, exact model and adapter. And in parallel: Prometheus metrics from vLLM (vllm:num_requests_running, vllm:gpu_cache_usage_perc, vllm:prefix_cache_hit_rate), from the GPU (DCGM Exporter), from the network (Hubble flow logs with drops and NetworkPolicy enforcement).

Canonical OSS pieces in 2026.

OpenTelemetry Collector as the single transport for traces, metrics and logs, with OTLP receivers and separate exporters per destination.
Langfuse self-hosted for the LLM-aware side: tracing, prompt versioning, evals with LLM-as-judge (post and post).
VictoriaMetrics + Grafana for high-throughput TSDB metrics with long retention.
Hubble (Cilium) for L3/L4/L7 flow logs and NetworkPolicy visualisation.
DCGM Exporter for GPU metrics.

What breaks the layer. Wrapping the whole pipeline stage in a single try/except that labels the exception with the wrapper’s name. The operational rule: one try/except per stage, with the stage label in the message and a Prometheus metric with labels={"stage": "<name>"}. That way “ChromaDB indexing error” would never have been the log for an embeddings failure; it would have been “Embeddings call failed: ConnectError(ollama:11434)”.

Complementary rule. The pipeline returns status: completed if and only if there were no errors. With errors it returns completed_with_errors or failed, and the metric pipeline_errors_total{stage} is incremented. An alert based on increase(pipeline_errors_total[1h]) > 0 fires before the second failed run. Without this discipline, observability exists but does not warn.

Layer 6 — GitOps control plane: the only legitimate authority

What it has to solve. That the cluster state is the state declared in git. That any divergence between git and the cluster is visible and, in critical components, auto-reconciled or auto-alerted. That every deployed image has an immutable tag (sha digest or semver pin), an SBOM (Trivy) and traceability back to the commit that introduced it.

Canonical OSS pieces in 2026.

Flux (or ArgoCD) as the reconciler.
Forgejo (or Gitea, GitLab CE) as the self-hosted OSS forge.
cert-manager + Trust Manager for internal PKI.
External Secrets Operator + SOPS for encrypted versioned secrets.
Kyverno (or OPA Gatekeeper) for binding policies: deny :latest images, deny pods without a NetworkPolicy, deny Services without an owner label.
Trivy for SBOM and vulnerability scanning of images in the CI pipeline.

What breaks the layer. Images with a mutable tag (:latest, :main). Any kubectl edit in production that is not reflected in git. main branches with write permissions for humans without review. The rule: if a human can mutate the cluster without going through a signed commit, you do not have GitOps, you have somebody’s whiteboard.

How it applies to the incident. If the pipeline image had been pinned to a sha digest (registry.interno.local/jobhunter@sha256:9af2...), the team could have audited immediately which Ollama client it carried. With :latest, not even that.

Layer 7 — Dependency tracking: the layer the incident exposed

What it has to solve. Knowing who calls which Service, both declaratively (what the gitops repo says) and observed (what has been seen crossing the network over the last N days). And, on mature platforms, propagating that information as policy: if nobody declares and nobody observes traffic to the Service ollama.ollama.svc, decommissioning it is safe; if somebody declares it or uses it, decommissioning it opens a ticket.

Canonical OSS pieces in 2026.

Hubble (Cilium) for observed flow logs: hubble observe --to-namespace ollama --since 14d gives the list of source namespaces that have talked to Ollama in the last two weeks.
Otterize for intent-based policy: each Deployment declares “I need to talk to ollama-svc”, and the operator generates the corresponding NetworkPolicy and maintains a browsable catalogue of who is trying to talk to what.
manual kubectl-grep as a fallback: kubectl get deployments,cronjobs,statefulsets -A -o yaml | grep -E 'ollama[.-]' produces the declarative list.
NetworkPolicy as code reviewed in CI: every PR that touches a Service requires the associated policy to be kept or explicitly updated.

A pre-decom checklist that the incident suggests codifying as a CI hook:

SVC="ollama.ollama.svc.cluster.local"

# (a) declarative grep in the gitops repo
git -C $GITOPS_REPO grep -l "$SVC" || echo "OK declarative"

# (b) observed grep in Hubble (last 14 days)
hubble observe --to-fqdn "$SVC" --since 336h --output json \
 | jq -r '.source.namespace' | sort -u || echo "OK observed"

# (c) live grep in the cluster
kubectl get all -A -o yaml | grep -E "$SVC" || echo "OK live"

If all three return empty, the decom is safe. If any of them has content, there is downstream debt still open. The jobhunter incident is exactly what happens when this check does not exist: the team that decommissioned Ollama looked at the list of applications it knew depended on it directly; nobody looked at the list of those that depended on it silently.

The maths that matter: sizing the stack on 4×H100 SXM (320 GB)

The generic reference cluster for everything that follows: 4×H100 SXM 80 GB, NVLink between the four, 640 GB of system RAM, 2×NVMe NVMe-oF for local storage of weights and caches, redundant 25/100 GbE to the switch. Aggregate VRAM: 320 GB.

The VRAM budget is not free. A first reasonable split for a stack that serves one large general LLM, a medium code-specialised model, embeddings and a reranker, with room for multi-LoRA and for the KV cache:

Component	Reference model	Quant	Model weights	Reserved KV cache	Total VRAM
General LLM (TP=4)	70B-instruct	FP8 W8A8	70 GB	60 GB	130 GB
Code LLM (TP=2)	32B-coder	FP8 W8A8	32 GB	28 GB	60 GB
Embeddings	multilingual-e5-large	FP16	1.3 GB	n/a	8 GB (×2 replicas)
Reranker	bge-reranker-v2-m3	FP16	0.6 GB	n/a	4 GB
Multi-LoRA pool (over the general LLM)	up to 16 adapters	bf16	16 × 0.4 GB ≈ 6 GB	reuses the LLM’s KV	6 GB
Reserved for fragmentation + overhead					~30 GB
Total committed					~238 GB / 320 GB
Free headroom					~82 GB

The 26 % of free headroom is not waste: it is what lets the vLLM scheduler avoid preempting requests under moderate pressure, lets continuous batching group large batches without aborting, and lets a failover from the other site promote a standby without OOM.

Expected throughput, with the 70B general LLM in FP8 and tensor parallel 4, on an H100 SXM with continuous batching active and prefix caching at 35–55 % (typical in multi-turn chat with shared system prompts):

$$ \text{aggregate tokens/second} \approx 1500 \text{ to } 2500 $$

for concurrency between 32 and 64 requests, with sub-second TTFT P95 on short prompts (<2k tokens) and TPOT P95 around 40–60 ms/token as perceived by the client. These numbers are reasonable orders of magnitude, not guarantees: real throughput depends on the prompt mix, on whether speculative decoding (EAGLE-3) is active and burnt-in, and on the network cost between gateway and inference pods.

Embeddings throughput on two replicas of multilingual-e5-large with dynamic batching:

$$ \text{embeddings/second} \approx 3000 \text{ to } 6000 \quad (\text{optimal batch} \sim 64) $$

Enough to reindex a corpus of 1 million documents in roughly an hour, assuming chunks of 512 tokens and two chunks per document on average. For corpora of tens of millions of documents, re-embedding is done by delta via CDC over the source (covered in RAG corpus curation), not by full sweep.

Retrieval latency on Qdrant with HNSW (M=16, ef_construct=200) over a collection of 5 million 1024-dim vectors filtered by tenant_id:

$$ \text{P95 latency retrieve top-50} \approx 8 \text{ to } 25 \text{ ms} $$

Below the cost of cross-encoder reranking (bge-reranker-v2-m3 over top-50 = roughly 30–60 ms more), and below any call to the LLM. The bottleneck in a well sized RAG pipeline is never the vector store: it is the LLM’s decoding.

Final diagram: the complete connected stack

The solid lines are the request path: client → gateway → inference engine → vector store/embeddings → response. The dashed blue lines are telemetry: each component emits OTel to the collector, which routes traces to Langfuse, metrics to VictoriaMetrics and logs to Loki. The dashed red lines are reconciliation: the GitOps control plane keeps any layer in its declared state and warns of divergence.

The diagram is not decorative: every arrow is a stable contract between two layers. If one layer changes (vLLM → SGLang, multilingual-e5 → bge-m3, Qdrant → pgvector), the arrows hold. That stability of contracts is the architectural property that lets a team migrate components without breaking downstream apps.

Typical design decisions that break the stack

A short list of mistakes seen repeatedly in stacks that looked well designed on paper:

1. Coupling the client SDK to the inference engine. Dropping the gateway because “vLLM already speaks OpenAI-compatible” works on day one and hurts the day you have to add a fallback, a canary or a second model.

2. Sharing the LLM and embeddings endpoint. A qwen2.5-32b-Instruct is chat-only; BadRequestError: "The model does not support Embeddings API" is the cry of a design that confused the two layers.

3. Reusing the vector store collection when changing the embeddings model. Different dimensions do not allow upsert. Versioning the collection by (model, dim, version) is ugly but it saves the day of the change.

4. A try/except wrapping an entire pipeline with the wrapper’s label. The log lies because the label is lexical, not causal. Every stage in its own try/except with its own label and its own metric.

5. status: completed with errors. The pipeline has to distinguish completed, completed_with_errors and failed, and the alerting has to fire on the last two. Without this, observability exists in theory and does not warn in practice.

6. Images with a mutable tag. :latest and :main are not tags, they are aliases. Without a sha digest there is no reproducibility and no auditable SBOM.

7. Decommissioning a Service without a pre-decom check. The three-grep check (declarative + observed + live) takes two minutes and costs six days of incident when it is skipped.

8. Default limits.memory on pods that load models. A sidecar loading sentence-transformers + torch + tokenizer needs 2–4 GB; with limits.memory: 1Gi you hit OOM on the first pod restart, and sometimes with no alert because the liveness probe answers through another route.

All of them are variants of the same principle: the stack does not fail in its most expensive layer (inference, where nobody underestimates the cost), it fails in the cheap, boring layers (gateway, observability, GitOps, dependency tracking) where it is tempting to save.

Applied to typical on-premise hardware: a 4×H100 SXM cluster

On the generic reference cluster (4×H100 SXM 80 GB, NVLink, 640 GB RAM), the suggested split into pods:

gpu-node-01 (4×H100 SXM, intra-node NVLink)
├── vllm-llm-general (TP=4) ~130 GB VRAM (4 GPUs)
└── (shares GPUs with the multi-LoRA pool on the same deployment)

gpu-node-02 (4×H100 SXM, second node)
├── vllm-llm-code (TP=2) ~60 GB VRAM (2 GPUs)
├── infinity-embeddings (×2) ~16 GB VRAM (shared on 1 GPU with optional MIG)
├── tei-reranker ~4 GB VRAM (cohabitant)
└── failover reserve ~120 GB VRAM free for canary / standby

Not everything fits comfortably on a single node; two nodes with two H100 SXM each would be enough for the conservative setup, and the rest of the cluster (CPU-bound: gateway, vector store, observability, control plane) runs on GPU-free nodes.

The operational rule: inference concentrates, the rest of the stack distributes. Concentrating inference maximises the use of NVLink (cross-GPU tensor parallel without going through PCIe); distributing the rest prevents an event on the GPU node from taking the control plane down with it.

An even more conservative configuration, for SMEs with a single 4×H100 SXM node as a starting point, serves the general LLM (TP=4) and embeddings/reranker cohabiting with MIG (Multi-Instance GPU to partition an H100 into hardware-isolated slices). The code LLM is deferred to a second phase. It is viable and cost-aware; what is not viable is doing without layers 5, 6 and 7.

What we have not covered (upcoming posts)

This post focuses on the static design of the stack. Some pieces still deserve an article of their own:

The multi-site active/standby plane: Cilium Cluster Mesh, cross-cluster Qdrant replication, realistic RTO/RPO, when active-active pays off and when it does not.
The continuous fine-tuning plane: how the LoRA pipeline closes the loop from production feedback to a promoted adapter, in the spirit of the retrain post.
The safety/guardrails plane: where Llama Guard fits, Presidio for PII and XGrammar for guaranteed structured output, connected to layer 1, the gateway.
The cost plane: gen_ai.usage.* instrumentation at tenant and model level, tokens/euro dashboards, GPU elasticity decisions via KEDA.
The compliance plane: how the stack maps to ENS High, NIS2 and ISO/IEC 42001 without turning the deployment into a compliance exercise that paralyses delivery.
Below the engine: the mini-series that opens the stack’s basement — the interconnect (NVLink/NCCL), the host (NUMA, hugepages, CPU isolation) and the RKE2 resource managers that pin each pod to the right NUMA node.

Each one falls into a different series on the blog and will be covered with the same discipline: a concrete piece, a justified decision, the typical mistake seen in practice.

References

The six-stage LLMOps pipeline — the general framework this stack operates in.
The OSS catalogue for LLMOps in six stages — a card per tool in the stack.
Anatomy of an LLM request in production — the same architecture seen from the perspective of an individual request.
OSS vs hyperscalers in LLMOps — a comparison with the AWS/Azure/GCP stacks.
LLM tracing with OpenTelemetry GenAI — the backbone of layer 5.
Multi-LoRA serving — the piece that gives per-tenant customisation without replicating the base.
RAG corpus curation and Reranker and hybrid retrieval — everything that goes into layers 3-4 so that RAG does not degrade.
Quantisation for LLM inference — the reason for the FP8 W8A8 in the sizing.

Relevant official documentation

vLLM Production Stack — docs.vllm.ai
SGLang RadixAttention — github.com/sgl-project/sglang
Envoy AI Gateway — aigateway.envoyproxy.io
Langfuse self-hosted — langfuse.com/docs/self-hosting
OpenTelemetry Semantic Conventions for GenAI — opentelemetry.io/docs/specs/semconv/gen-ai
Hubble flow observability — docs.cilium.io/en/stable/observability/hubble
Otterize intent-based access — docs.otterize.com
Flux GitOps toolkit — fluxcd.io

Embeddings on lo0 — Tech Blog

Serving embeddings and rerankers with TEI in production

TL;DR

The analogy: the stamping press that only stamps vectors

What TEI is, exactly

The endpoints

The backends

Why a dedicated server and not calling the model inline

Dynamic batching: the mechanics of the press

The number: throughput with batching vs without batching

Deployment: compose for embedder and reranker

Embedder (bge-m3) on GPU

Reranker (bge-reranker-v2-m3) on GPU

CPU for batch ingestion

Integration: how ingestion and the gateway consume it

Sizing: memory, replicas and latency budget

Footprint of a 568M encoder

Latency budget: short query vs top-50 rerank

Applied to the generic 4×H100 cluster

Conclusion

See also

References

End-to-end document ingestion: from PDF to indexed chunk

TL;DR

The analogy: an archive’s cataloguing chain

The six-stage pipeline

Stage 1 — Parsing: layout-aware vs plain text

OCR for scans

Stage 2 — Cleaning, normalising and deduplicating

Deduplication: why it matters

Stage 3 — Chunking

What the 2026 benchmarks say (and why semantic disappoints)

A worked numerical example of chunking

Stage 4 — Enriching with metadata

Stages 5 and 6 — Embedding and indexing

A cross-cutting stage — Incremental ingestion and idempotency

The arithmetic: sizing the corpus and the index

Corpus sizing and ingestion time

Vector index size

Applied to the generic 4×H100 cluster

Closing: quality is decided upstream

See also

References

Taking RAG to the CPU: separating the data plane from the generation plane

TL;DR

The analogy: the library and the librarian

The three phases and their compute profiles

Why ingestion fits on CPU: the embedder is not an LLM

Runtimes that already do this effortlessly

Viability table: CPU for each component?

The numbers, with honest methodology

Decision tree: CPU or GPU for this piece?

Reference architecture (a): CPU-only

Reference architecture (b): the recommended hybrid

Applied to the generic 4×H100 cluster

The auditability angle: ENS / NIS2

When NOT to take it to CPU

See also

References

Semantic cache in RAG: the receptionist with a photographic memory

TL;DR

The analogy: the receptionist with a notebook

The problem in production

How the semantic cache works

Flow diagram

The threshold θ and its trade-off

The arithmetic of the saving

OSS stack 2026

GPTCache

MeanCache

Qdrant as a dual cache store

Langfuse for traceability

On-premise hardware: a reference configuration

Cases where the cache fails

1. Queries with a temporal context

2. Personalised queries over private data

3. TTL and a stale corpus

Integration into the pipeline as middleware

What we have not covered

See also

Embedder (`bge-m3`) on GPU

Reranker (`bge-reranker-v2-m3`) on GPU

vLLM `--task embed`, for the 7B embedders