Reranker on lo0 — Tech Blog

Serving embeddings and rerankers with TEI in production

Thu, 11 Jun 2026 09:40:00 +0000

Sixth piece in an operational series about squeezing a generic on-premise LLM cluster of 4×H100 SXM 80 GB. If RAG on the CPU argued where the data plane runs and the embedder zoo decided which model you serve, this post looks at the engine that serves them: the embeddings and rerankers server. The sisters in this batch, document ingestion from PDF to indexed chunk, GitOps for the inference stack with Flux and hardening and secrets for the sovereign stack, build the rest of the system around this piece. The closer of the series, the end-to-end sovereign assistant with LibreChat + LiteLLM + RAG, is still in draft.

TL;DR

Serving embeddings is not model.encode(text) scattered through the ingestion code. Just as you do not put a transformers.generate() inside your API and call it vLLM, you do not put the embedder inline either: you put it behind a dedicated server that does batching, controls concurrency, emits metrics and exposes a single HTTP contract that both ingestion and query-time reuse. In the 2026 open source ecosystem that server is TEI, Text Embeddings Inference from Hugging Face: a Rust binary that serves embedding, reranker (cross-encoder) and sequence classification models, with token-based dynamic batching, and that exposes OpenAI-compatible endpoints (/v1/embeddings) on top of the native /embed, /embed_sparse, /rerank and /predict (TEI repo, docs). It runs on CPU (ONNX Runtime, Intel MKL) and on CUDA (with FlashAttention and cuda graphs), with the same contract on both sides, so the rest of the system does not know, or care, what silicon sits behind it. The technical piece that gives almost all the performance is dynamic batching: TEI groups requests that arrive separately into a batch that shares the fixed cost of the forward, controlled by --max-batch-tokens (how many tokens fit per batch) and --max-concurrent-requests (backpressure). Grouping raises throughput, going from serving 1 sequence to 32 multiplies tokens/s at almost constant cost until the hardware saturates, in exchange for some latency while the tray fills. The numbers: a 568M encoder (bge-m3, bge-reranker-v2-m3) takes ~1.1 GB in fp16 and ~0.57 GB in int8, so several replicas fit even in a small H100 slice; embedding a short query online costs tens of ms, but reranking the top-50 costs linearly in candidates because a cross-encoder evaluates each (query, doc) pair separately. The split across the 4×H100 cluster: TEI-CPU on the fleet for batch ingestion (no SLA, throughput/€), TEI-GPU in a MIG slice of one H100 for high-QPS online embeddings and rerank, talking to ingestion and to the LiteLLM gateway through the OpenAI contract.

The analogy: the stamping press that only stamps vectors

Picture a workshop with a specialised stamping press. It does not make varied parts; it does one single thing: it takes flat material and, in one stroke, stamps a shape into it. In our case it takes text and stamps a vector. And like every industrial press, it has one property that defines all its behaviour: the cost of the stroke is almost fixed, no matter how much material you put on the tray.

Lowering the press, heating the die, aligning, pressing and lifting costs, say, one second. If you put a single sheet in, you spend that whole second stamping one part: a waste. If you fill the tray with thirty sheets and lower the press once, you spend nearly the same second and come out with thirty stamped parts. The cost per part collapses. The press performs far better when it fills the tray before pressing.

This is exactly the dynamic batching of an embeddings server. The “stroke” is the model’s forward pass on the GPU or the CPU: an operation that loads the weights, multiplies them by the activations and emits the vectors, with a large fixed cost that does not scale with how many sequences you process at once, until you saturate memory bandwidth or the compute units. Serving requests one at a time is stamping one sheet per stroke. Grouping them into a batch is filling the tray.

But the press has an operational dilemma, and here is the heart of the post. If you wait until the tray is completely full before pressing, the first sheets that arrived sit waiting for the rest: latency. If you press as soon as one sheet drops so nobody has to wait, you go back to the waste of one stroke per part: low throughput. The embeddings server solves this with a bounded wait window: it gathers whatever arrives within a window of microseconds to milliseconds, or until the batch token budget is full, and then it presses. That is the policy that turns a pile of independent requests into a few efficient strokes without anyone waiting too long.

The rest of the post is, at bottom, about configuring that press properly: how many sheets fit on the tray (max-batch-tokens), how much queue is tolerated before work is rejected (max-concurrent-requests), whether the press lives on the cheap CPU fleet (nightly ingestion, pure throughput) or in a slice of the expensive GPU (online, bounded latency), and why the reranker is a different press whose cost does grow with the number of parts.

What TEI is, exactly

Text Embeddings Inference (TEI) is Hugging Face’s inference server for the family of models that does not generate tokens: embedders, cross-encoder rerankers and sequence classifiers. It is to encoders what vLLM is to generative LLMs: a high-performance server, written in Rust, that handles tokenisation, batching, concurrency, metrics and the HTTP contract, leaving the application code to only make calls (TEI repo).

Three model types, three jobs:

Embedding (bi-encoder): turns a text into a reusable vector. bge-m3, Snowflake Arctic Embed, multilingual-e5. The vector is stored in the index and reused on every search.
Reranker (cross-encoder): takes a pair (query, document) and emits a relevance score, not a vector. bge-reranker-v2-m3. It produces nothing reusable: every pair is evaluated again.
Sequence classification: emits labels/probabilities over a text. Useful for lightweight guardrails, routing, language or toxicity detection.

The endpoints

TEI exposes both an OpenAI-compatible endpoint and its native endpoints, verified against the current documentation (Quick Tour, DeepWiki):

Endpoint	What for	Model type
`/v1/embeddings`	Embeddings with the OpenAI contract (drop-in for clients that already speak OpenAI)	embedder
`/embed`	Dense embeddings, TEI’s native API	embedder
`/embed_sparse`	Sparse embeddings (lexical head of models like `bge-m3`/SPLADE)	embedder
`/rerank`	Orders a list of texts by relevance against a query	reranker (cross-encoder)
`/predict`	Sequence classification (labels/scores)	classifier
`/embed_all`	Per-token embeddings (no pooling)	embedder
`/similarity`	Direct similarity between texts	embedder
`/metrics`, `/health`, `/info`, `/tokenize`, `/decode`	Operation, observability and utilities	all

The operational key is in the first row: TEI speaking the OpenAI contract on /v1/embeddings means the embedder is swappable without touching client code. The same code that called OpenAI’s text-embedding-3-large points at your on-prem TEI by changing the base_url, and the gateway (LiteLLM) routes it as one more provider. /rerank, by contrast, is not part of the OpenAI standard, no such endpoint exists in their API, so the gateway treats it as a reranking-specific endpoint (LiteLLM TEI rerank).

The backends

TEI runs on CPU and on GPU with the same HTTP contract, which is exactly the property the CPU/GPU split of RAG needs (TEI docs, hardware):

CPU: cpu-* image. ONNX Runtime (recommended) or Intel MKL backend. It exploits the integer compute paths (AVX-512+VNNI, AMX on 4th-gen Xeon) when the model is in int8. It is the batch ingestion engine.
CUDA: architecture-specific images (cuda-*, with variants for different compute capabilities). It uses FlashAttention and cuda graphs to squeeze the GPU. It requires compute capability ≥ 7.5 (Volta is out) and CUDA 12.2+ compatible drivers. It is the high-QPS online engine.
There is also Metal support (Apple Silicon) and experimental ROCm (AMD Instinct), less relevant to the on-prem case of this post.

The golden rule: the same bge-m3 served by TEI-CPU and by TEI-GPU exposes an identical /v1/embeddings. The client does not notice the silicon. That is what lets you put ingestion on CPU and online on GPU without rewriting anything.

Why a dedicated server and not calling the model inline

The temptation, especially in the prototype, is to load the embedder into the application process and call model.encode() directly. It works in the demo and breaks in production for five reasons, all solved by the dedicated server:

Batching. Your ingestion code processes documents; your query code embeds one query at a time. Inline, each encode() is an independent forward, one press stroke per sheet. A dedicated server joins requests from different origins (ingestion + several concurrent queries) into a single batch and amortises the fixed cost. This is what really multiplies throughput, and you cannot do it if the model lives inside each isolated process.
Concurrency and backpressure. Under load, what happens when 10,000 requests arrive at once? Inline, you run out of memory or queue without control. TEI has --max-concurrent-requests: above that limit it rejects instead of degrading everyone, which is the correct way to manage overload (TEI CLI args).
Metrics. TEI exposes /metrics in Prometheus format: latencies, batch size, tokens/s, queued requests. Inline you have none of this without hand-instrumenting every encode(). To observe the data plane you need that endpoint.
A reusable HTTP contract. The same server is consumed by batch ingestion, by online query-time and by any other service that needs vectors. One copy of the model in memory, one version point, one place to change the model. Inline, each service loads its own copy and the version drifts on its own.
Lifecycle separation. The embedder is updated, restarted or scaled without touching the application. If tomorrow you swap bge-m3 for Snowflake Arctic, you change the server container, not the code of N services. (Remember the embedder drift trap: changing embedder forces a reindex; having a single server makes that change atomic and auditable.)

It is the same argument for why nobody serves an LLM with transformers.generate() in production: you serve it with a dedicated engine. The embedder is no different.

Dynamic batching: the mechanics of the press

Here is the performance engine. TEI does token-based dynamic batching: instead of processing requests one at a time or grouping them by a fixed number of requests, it groups them by token budget (TEI README, Discussion #367).

Two knobs rule:

--max-batch-tokens (the default depends on the build, typically 16384): the total tokens that fit in a batch. With max-batch-tokens=1000 you get 10 requests of 100 tokens, or one of 1000. The docs say it clearly: this number should be as large as possible until the model becomes compute-bound. It is the size of the press tray, measured in tokens, not in parts, because the real cost scales with tokens, not with number of texts.
--max-concurrent-requests (default 512): how many requests it admits in flight before rejecting. It is the backpressure control, not the batching one: it protects against surges by returning an error instead of queuing indefinitely.

The policy: TEI keeps an internal queue. When there is work, it fills a batch up to the max-batch-tokens budget (or until there are no more requests waiting) and sends it to the model in one stroke. Requests that arrive while the previous batch is being processed wait in the queue and go into the next one. Watch out for a real operational detail: if max-batch-tokens is smaller than the model’s maximum input length, you can trigger a loop where a long request never fits into a batch (Issue #723); the token budget has to be at least as large as the longest sequence you accept.

The number: throughput with batching vs without batching

Let us model the press. Let $C$ be the fixed cost of a forward pass (loading weights, launching kernels) and $c$ the marginal cost per sequence inside the batch. For a batch of size $B$, the stroke time is roughly:

$$t(B) \approx C + c \cdot B$$

as long as the batch fits in the hardware (memory-bandwidth-bound, which is the normal regime of a small encoder on GPU). The throughput, sequences per second, is:

$$\text{throughput}(B) = \frac{B}{t(B)} = \frac{B}{C + c \cdot B}$$

Let us put in concrete numbers, indicative but of the right order for bge-m3 (568M) on a GPU serving ~256-token sequences. Say $C = 4$ ms of fixed cost per stroke and $c = 0.5$ ms per marginal sequence.

Without batching ($B=1$):

$$t(1) = 4 + 0.5 = 4.5 \text{ ms} \quad\Rightarrow\quad \text{throughput} = \frac{1}{4.5\text{ms}} \approx 222 \text{ sec/s}$$

With batching ($B=32$):

$$t(32) = 4 + 0.5 \times 32 = 20 \text{ ms} \quad\Rightarrow\quad \text{throughput} = \frac{32}{20\text{ms}} = 1600 \text{ sec/s}$$

The batch of 32 multiplies throughput by ~7.2× compared with serving one at a time, because the fixed cost $C$, which at $B=1$ was 89% of the time, is spread across 32. And the asymptotic limit, as $B \to \infty$, is $1/c = 2000$ sec/s: the press cannot go faster than its marginal cost, and we are already at 80% of that ceiling with $B=32$. Going up to $B=128$ would give ~1969 sec/s, a marginal improvement in exchange for a good deal more latency and memory. Batching has diminishing returns: most of the gain is in going from 1 to a few dozen.

The latency trade-off is the other face. The request that arrived first in the batch of 32 does not see its response until the whole stroke finishes: it waits for the tray to fill plus the 20 ms of the forward. If the 32 requests arrived within a window of, say, 5 ms, the first request suffers ~5 ms of batching wait + 20 ms of compute = 25 ms of latency, against the 4.5 ms it would have had being served alone. The p99 latency gets worse, that is the price of the full tray, but in exchange the system absorbs 7× more load with the same silicon. That is why the correct configuration depends on the use case: in batch ingestion (no user waiting) you maximise max-batch-tokens and latency does not matter; online (user waiting) you bound the wait window to keep p99 under control even at the cost of some throughput.

Deployment: compose for embedder and reranker

TEI is deployed as a container. The key arguments: --model-id (the model from the Hub), --pooling (how it aggregates tokens into the vector: cls, mean, splade, last-token), --dtype (precision), and the batching knobs already seen. Two separate services, since the embedder and the reranker are different models and different endpoints, each with its own container.

Embedder (`bge-m3`) on GPU

# docker-compose: TEI serving bge-m3 as an embedder, on GPU
services:
 tei-embed:
 image: ghcr.io/huggingface/text-embeddings-inference:cuda-1.9
 command:
 - --model-id=BAAI/bge-m3
 - --pooling=cls
 - --dtype=float16
 - --max-batch-tokens=16384 # big tray: the press performs
 - --max-concurrent-requests=512 # backpressure: reject above this
 ports: ["8081:80"]
 volumes: ["./hf-cache:/data"]
 deploy:
 resources:
 reservations:
 devices: [{driver: nvidia, count: 1, capabilities: [gpu]}]
 # exposes /v1/embeddings (OpenAI), /embed, /embed_sparse, /metrics

Reranker (`bge-reranker-v2-m3`) on GPU

 tei-rerank:
 image: ghcr.io/huggingface/text-embeddings-inference:cuda-1.9
 command:
 - --model-id=BAAI/bge-reranker-v2-m3  # cross-encoder, ~568M
 - --dtype=float16
 - --max-batch-tokens=16384
 - --max-concurrent-requests=256
 ports: ["8082:80"]
 volumes: ["./hf-cache:/data"]
 deploy:
 resources:
 reservations:
 devices: [{driver: nvidia, count: 1, capabilities: [gpu]}]
 # exposes /rerank — invoked ONLY over top-k (20/50), not over hundreds

bge-reranker-v2-m3 is a ~568M cross-encoder built on bge-m3, multilingual, that takes (query, document) and emits a relevance score directly, without producing a reusable vector (model card, BGE docs). It carries no --pooling because it emits no embedding: it emits a scalar.

CPU for batch ingestion

For the CPU fleet, it is enough to change the image and the dtype; the HTTP contract is identical:

 tei-embed-cpu:
 image: ghcr.io/huggingface/text-embeddings-inference:cpu-1.6
 command:
 - --model-id=BAAI/bge-m3
 - --pooling=cls
 - --dtype=float16  # on CPU, ONNX int8 if the export supports it
 - --max-batch-tokens=8192 # smaller tray: the CPU saturates sooner
 ports: ["8083:80"]
 volumes: ["./hf-cache:/data"]
 # ONNX/MKL backend: exploits AVX-512+VNNI / AMX on 4th-gen Xeon

Any OpenAI client embeds against the endpoint without knowing what is behind it:

from openai import OpenAI
client = OpenAI(base_url="http://tei-embed:80/v1", api_key="-") # api_key ignored
r = client.embeddings.create(model="BAAI/bge-m3", input=["user query"])
vec = r.data[0].embedding # same contract as OpenAI; the silicon is invisible

And the rerank, through its native (non-OpenAI) endpoint:

import requests
docs = ["retrieved chunk 1", "retrieved chunk 2", "...top-20 from retrieval..."]
r = requests.post("http://tei-rerank:80/rerank",
 json={"query": "user query", "texts": docs})
ranked = r.json() # [{index, score}, ...] ordered by relevance

Integration: how ingestion and the gateway consume it

The TEI server has two clients with opposite profiles, and that is precisely the reason to centralise it:

Ingestion (batch, no SLA) calls /v1/embeddings or /embed with large lots of chunks. Here throughput is what matters: ingestion pushes thousands of texts and TEI groups them into large batches against the high max-batch-tokens. It is the case of the press with a full tray. Ingestion then writes the vectors (dense + sparse) into the vector DB; the specific PDF-to-indexed-chunk pipeline is the sister piece of this series in preparation, and meanwhile the PostgreSQL + Qdrant schema in microservices covers the structure of the pipeline.

The gateway (query-time, online) consumes TEI through two different routes:

Embedding the query: a single call to /v1/embeddings with a short text. The gateway, LiteLLM or another L7 router, treats it as one more OpenAI embeddings provider, routed by model. That embedded query goes to the vector DB for the hybrid search.
Reranking the top-k: after retrieving 20-50 candidates from the retriever, the gateway calls /rerank on the reranker server with the query and the candidate list, and keeps the best ones. Since /rerank is not an OpenAI standard, LiteLLM exposes it through its own reranking contract (LiteLLM rerank with TEI).

The full query-time pattern: embed query (TEI dense) → search (vector DB, hybrid retrieval) → rerank top-k (TEI rerank) → send query + top-5 to the LLM. Two different TEI endpoints, one for dense and another for rerank, both behind the gateway, both swappable.

Sizing: memory, replicas and latency budget

Footprint of a 568M encoder

The weight footprint of bge-m3 (568M params) depends only on precision:

$$\text{fp16: } 568 \times 10^6 \times 2 \text{ B} \approx 1.14 \text{ GB} \qquad \text{int8: } 568 \times 10^6 \times 1 \text{ B} \approx 0.57 \text{ GB}$$

On top of this comes the space for the batch’s temporary activations (proportional to max-batch-tokens × hidden dimension) and, in TEI, the FlashAttention/cuda graphs structures. In practice, a bge-m3 in fp16 is served comfortably in ~6 GB of VRAM even at high batch, and in int8 on CPU the whole model (0.57 GB) fits in the cache and RAM of any server without blinking, activating the AMX/VNNI paths that make it viable, which is exactly the argument of RAG on the CPU.

How many replicas fit? If we reserve a 10 GB slice of an H100 for serving embeddings online, with an effective footprint of ~2.5 GB per replica (fp16 weights + activations + overhead), around 3-4 replicas of bge-m3 fit in that slice, splitting the QPS between them. In int8 the footprint drops and more fit, at the cost of a full point of quality. For the reranker, the weight footprint is the same (568M), but its batch pattern is different (query-doc pairs), so it is worth sizing it separately.

Latency budget: short query vs top-50 rerank

Here is the fundamental asymmetry between embedder and reranker, and it is what decides the sizing.

Embedding a short query (online): it is one forward over a text of tens of tokens. On GPU, inside a dynamic batch, it is tens of ms including the batching wait, the order of the earlier numerical example, ~5-25 ms. It is cheap and constant: a query is a query, no matter how many documents are in the corpus. That is why embedding online fits easily into an interactive latency budget.

Reranking the top-50 (online): a cross-encoder does not produce a reusable vector; it evaluates every pair (query, document) in a forward. The cost is linear in the number of candidates:

$$\text{cost}_{\text{rerank}} \propto k \times \text{forward}(\text{query} + \text{doc})$$

Reranking a top-50 is 50 forwards of (query + doc) pairs, where each pair is longer than the query alone (query + ~256-token chunk). TEI groups them into batches, and batching helps again there, but the total work is ~50× that of embedding one query. If embedding a query costs of the order of 10-20 ms, reranking 50 candidates costs of the order of hundreds of ms depending on the batch and the hardware. Hence the operational rule repeated throughout the series: broad, cheap recall in the retriever, precision rerank over FEW candidates. Reranking a top-20/50 fits in the online budget; reranking hundreds at high QPS does not, and that is where the reranker demands a dedicated GPU or a larger slice. The detail of why the cross-encoder is expensive but precise is in reranker and hybrid retrieval.

The query-time latency budget, added up:

Stage	Typical cost (order)	Scales with
Embed query (TEI dense)	~10-20 ms	constant (1 short text)
Hybrid search (vector DB, HNSW+sparse)	single-digit ms	index size (sublinear)
Rerank top-k (TEI cross-encoder)	~100-300 ms	linear in k
Generation (LLM, outside TEI)	seconds	output tokens

The reranker is the most expensive stage of the data plane, and the only one that scales badly with the width of retrieval. Size it as the critical component.

Applied to the generic 4×H100 cluster

Let us come down to the cluster of the series: 4×H100 SXM 80 GB plus a generic CPU fleet (Xeon with AMX, NUCs). TEI lives in two places depending on the load profile, and the identical OpenAI contract is what makes each place swappable.

TEI-CPU on the fleet → batch ingestion. Re-embedding the corpus is throughput-bound work with no latency SLA: its place is the CPU fleet with bge-m3 in int8 (ONNX/AMX). Here you maximise max-batch-tokens because nobody is waiting, and the press performs with the tray brimming. No H100 should spend a cycle reindexing a corpus that changes once a day, which is exactly the bad split that RAG on the CPU takes apart. The CPU fleet scales horizontally and cheaply; the corpus is chopped up and spread across nodes.

TEI-GPU in a MIG slice → high-QPS online embeddings and rerank. Query-time has a latency SLA and, if the system serves many requests per second, it needs the GPU’s throughput and low latency. But a 568M encoder does not deserve a whole H100: it takes ~2.5 GB effective, and leaving it 80 GB wastes 97% of the card. The right answer is a MIG slice: partition an H100 into hardware-isolated instances and give TEI one of the small slices (a 1g.10gb, for example), leaving the large slices for generation or for other tenants. The online embedder and reranker live there, with memory and compute isolation guaranteed by the partitioning. The how of the partitioning, MIG, MPS, time-slicing and when to use each, is in sharing a GPU; the reading for TEI is direct: MIG gives the hard isolation that an online service with bounded latency wants, while batch ingestion on CPU does not even touch the GPU.

The split, in one sentence: ingestion squeezes throughput/€ on CPU with a full tray; online squeezes latency/QPS in a MIG slice of an H100; both speak the same /v1/embeddings and /rerank, so moving load from one side to the other is changing a URL in the gateway.

If at some point you need a 7B embedder (gte-Qwen2, NV-Embed) whose quality bge-m3 cannot reach, that is no longer small-encoder work: it drags an LLM’s cost profile with it and is served where 7Bs are served, with vLLM --task embed in a large GPU slice, not with TEI-CPU. But it is the occasional exception, not the base load.

Conclusion

TEI is the stamping press of the RAG data plane: a specialised server that only stamps vectors and scores, and that performs far better when it fills the tray before pressing. The technical piece that gives the performance is token-based dynamic batching, controlled by max-batch-tokens and max-concurrent-requests, with a clear trade-off between throughput and p99 latency that is resolved differently in ingestion (tray brimming, no SLA) and online (bounded window, p99 protected). The architectural virtue is the OpenAI-compatible contract on /v1/embeddings, which makes the embedder swappable without touching code, plus the native /embed_sparse and /rerank endpoints for the sparse head of bge-m3 and for the bge-reranker-v2-m3 cross-encoder. The numbers drive the sizing: a 568M encoder fits in gigabytes, not tens of gigabytes, so several replicas live in a MIG slice; embedding a query is cheap and constant, but reranking is linear in candidates and is the critical stage of the latency budget. On the 4×H100 cluster, the right split is TEI-CPU for batch ingestion and TEI-GPU in a MIG slice for high-QPS online: two presses, two silicons, one contract.

References

Hugging Face — Text Embeddings Inference (TEI), repo y docs (Rust, backends CPU ONNX/MKL y CUDA, endpoints OpenAI-compatibles). https://github.com/huggingface/text-embeddings-inference · https://huggingface.co/docs/text-embeddings-inference
Hugging Face — TEI Quick Tour (endpoints /embed, /embed_sparse, /rerank, /predict, /v1/embeddings). https://huggingface.co/docs/text-embeddings-inference/quick_tour
Hugging Face — TEI CLI arguments (--model-id, --pooling, --dtype, --max-batch-tokens, --max-concurrent-requests). https://github.com/huggingface/text-embeddings-inference/blob/main/docs/source/en/cli_arguments.md
Hugging Face — TEI README / Discussion #367 (token-based dynamic batching; semántica de max-batch-tokens). https://github.com/huggingface/text-embeddings-inference/blob/main/README.md · https://github.com/huggingface/text-embeddings-inference/discussions/367
BAAI — BGE-M3 model card (568M, XLM-RoBERTa, dense+sparse+colbert, MIT). https://huggingface.co/BAAI/bge-m3
BAAI — bge-reranker-v2-m3 model card (cross-encoder ~568M sobre bge-m3, multilingüe). https://huggingface.co/BAAI/bge-reranker-v2-m3 · https://bge-model.com/bge/bge_reranker_v2.html
Chen, J., et al. BGE M3-Embedding. arXiv:2402.03216. https://arxiv.org/abs/2402.03216
BerriAI — LiteLLM: proveedor TEI para /rerank. https://github.com/BerriAI/litellm/issues/8372

Taking RAG to the CPU: separating the data plane from the generation plane

Thu, 11 Jun 2026 03:20:00 +0000

Third piece in an operational series about squeezing a generic on-premise LLM cluster of 4×H100 SXM 80 GB. Its siblings: sharing one GPU between workloads (time-slicing, MPS, MIG) and serving several models on one GPU (swap + sleep) attack the split inside the GPU. This one attacks the split outside: which parts of RAG never need to touch the GPU at all. The closing piece of the series, the end-to-end sovereign assistant (fourth instalment, in preparation), assembles the complete system where these pieces fit together.

TL;DR

A RAG system is not one thing, it is three phases with opposing compute profiles, and putting them all on the GPU “because it is AI” is an allocation mistake. (1) Build/ingest, embedding the corpus and building the index, is batch, throughput-bound work with no latency SLA: its natural home is the CPU. (2) Query-time retrieval, embedding the query, HNSW search, RRF fusion, lightweight rerank, is mostly CPU, with caveats only for heavy reranking; vector search always ran on CPU, even in stacks sold as “GPU”. (3) Generation, the LLM producing the answer, is latency-bound, and there the GPU is irreplaceable: a 7B on CPU gives a time to first token of seconds, unacceptable for chat. The technical key to why (1) and (2) fit on CPU: the embedder is not an LLM. bge-m3 is around 568M parameters (an XLM-RoBERTa encoder), not 7B+; in int8 it occupies about 580 MB and activates fast integer compute paths (Intel AVX-512 + VNNI + AMX on 4th-gen Xeon and later; NEON SDOT/UDOT on ARM). The runtimes are ready: TEI with a CPU backend (same OpenAI API /v1/embeddings and /rerank), Qdrant’s fastembed (ONNX-CPU), bge-m3 in ONNX int8 with its three heads (dense/sparse/ColBERT). The Intel + Hugging Face blog post with Optimum Intel and fastRAG reports up to around 10× on indexing for BGE-large int8 on a 4th-gen Xeon (their benchmark figure, encoding-only; I quote it and qualify it below). The operational conclusion: separate the data plane (CPU) from the generation plane (GPU). In the 4×H100 cluster, no H100 should be spent re-indexing a corpus that changes once a day. That goes to the generic CPU fleet (Xeon AMX, NUCs), and the H100s are reserved for generation and, at most, for rerank peaks or large 7B embedders. What does not go down to CPU: interactive generation, massive reranking at high QPS, re-indexing with a strict real-time SLA, and 7B embedders (gte-Qwen2, NV-Embed).

The analogy: the library and the librarian

Picture a serious research library. There are three distinct jobs, done by different people, on different clocks.

The first is cataloguing. Boxes of new books arrive; someone opens them, classifies them, assigns a shelf mark, indexes them in the catalogue and puts them on the right shelf. It is patient background work, done at night or between other tasks. Nobody is standing over you with a stopwatch waiting for today’s batch to be catalogued: what matters is that tomorrow it is done and done well. It is pure throughput: how many books you catalogue per hour, not how long you take on one specific book. This is ingestion.

The second is handling an enquiry at the desk. A reader turns up and asks about a topic. The librarian goes to the catalogue, which already exists, finds half a dozen relevant shelf marks, fetches them and puts the books on the desk. It is fast, light, and consists of searching an index that already exists, not building it. This is retrieval.

The third is writing a reasoned report from those books. The reader, or an expert you commission, reads the six books, compares them, synthesises, and writes an argued answer with citations. This is slow, demands a very well-trained mind, and the reader is waiting: here there is a human stopwatch. This is generation, the LLM.

The moral is about staff allocation. You do not put your star writer, expensive, scarce, with a queue of people waiting for reports, to catalogue boxes of books at dawn. Cataloguing is done by a large, cheap team that works through the night without rushing. The star writer touches only what genuinely needs their mind: writing. In our system, the star writer is the H100, and cataloguing at dawn is corpus ingestion. Spending the H100 on re-indexing is exactly the error of putting the writer to work labelling boxes.

The rest of the post is, essentially, which parts of the library work the cheap CPU team can do (almost all of them) and which is irreducibly the GPU writer’s job (only the last one).

The three phases and their compute profiles

The confusion that GPU over-provisioning feeds on is treating “RAG” as a monolithic block that “uses AI, therefore goes on the GPU”. No. RAG is a data pipeline with a generative model plugged in at the end. The correct architectural boundary does not separate “what uses models” from “what does not”, because both sides use models, but throughput-bound from latency-bound, which is the same as separating the data plane from the generation plane.

Why ingestion fits on CPU: the embedder is not an LLM

The whole argument rests on a size asymmetry that gets overlooked. People hear “embeddings” and “generation” and put them in the same bag of “big models that need a GPU”. But the embedding encoder and the generative LLM are two orders of magnitude apart in parameters.

bge-m3, the reference multilingual embedder, is an XLM-RoBERTa of around 568M parameters (model card, paper arXiv:2402.03216). Its sibling reranker, bge-reranker-v2-m3, is built on the same base and lands at roughly the same 568M parameters (model card). Compare that with an entry-level generative LLM: a Llama 3.1 8B has around 14× more parameters, and the big production models sit at 70B+. A 568M encoder is, in compute budget terms, a different animal.

Two structural differences make that encoder comfortable on CPU:

It is an encoder, not an autoregressive decoder. It processes the whole sequence in a single forward pass and emits the vector. There is no token-by-token decode, no growing KV cache, none of the memory-bound generation phase that kills the CPU. It is a dense matrix pass and that is it.
It quantises to int8 with almost no loss. In int8, bge-m3 occupies on the order of 580 MB and, above all, activates the integer compute paths that a modern CPU executes quickly: matrix instructions such as Intel AMX (Advanced Matrix Extensions, 4th-generation Xeon and later), AVX-512 with VNNI (Vector Neural Network Instructions) on earlier Xeons, and NEON SDOT/UDOT on ARM. The quality loss from going FP32 to int8 on these models usually stays below 1% of retrieval recall, practically invisible (Intel + Hugging Face, CPU Optimized Embeddings).

Let us quantify the int8 size. For $P = 568 \times 10^6$ parameters at 1 byte each:

$$\text{size}_{\text{int8}} \approx 568 \times 10^6 \text{ params} \times 1 \text{ byte/param} \approx 568 \text{ MB}$$

In other words, the model fits in the cache and RAM of any server or NUC without blinking, and the bottleneck is integer compute, exactly what AMX/VNNI accelerate. There is nothing in this profile that asks for a GPU.

Runtimes that already do this effortlessly

Nothing needs inventing. The CPU ecosystem for the data plane is mature:

Text Embeddings Inference (TEI) from Hugging Face: a Rust server with CPU backends via ONNX Runtime (recommended) or Intel MKL, and OpenAI-compatible endpoints (/v1/embeddings) plus /rerank (TEI repo). That is, the data plane on CPU exposes exactly the same HTTP contract as a GPU server; the rest of the system never learns which silicon is behind it.
fastembed from Qdrant: a lightweight library that loads embedders on ONNX-CPU and generates dense, sparse and ColBERT vectors (fastembed repo). Designed from the start to run without a GPU.
bge-m3 in ONNX int8 with its three heads (dense / sparse-lexical / ColBERT multivector) exported and quantised, ready for ONNX Runtime CPU.

The Intel and Hugging Face figure that anchors viability: in their benchmark with Optimum Intel + fastRAG on a 4th-generation Xeon (8480+, 56 cores, 1 socket), the int8 variant of BGE-large reaches up to around 10× indexing throughput against FP32 (HF blog, Haystack/deepset). The small print needs reading, so I read it: that ~10× is encoding-only (tokenisation excluded), at sequence length 256, comparing int8 against FP32 on the same CPU. It is not “CPU 10× faster than GPU”, it is “int8 10× faster than FP32 on CPU”. It is still the relevant figure: it tells you that with quantisation the CPU goes from unworkable to perfectly useful for batch ingestion.

Viability table: CPU for each component?

This is the operational table. The column that matters is the caveat, because a bare “yes” or “no” lies.

Component	CPU viable?	Caveat
Chunking (slicing the corpus)	Yes, always	It is regex, parsing and windows; it never had anything to do with the GPU.
Ingest embedding `bge-m3` dense	Yes, its best case	Overnight batch, int8 + AMX/VNNI. This is exactly what the CPU shines at.
Sparse / SPLADE / BM25 head	Yes, CPU-native	Lexical work is pure inverted index; the GPU adds nothing here.
Building the HNSW index (Qdrant, pgvector)	Yes, always CPU	The HNSW graph build is CPU by design in these engines.
Query embedding (online)	Yes	A single short text; tens of ms on CPU, more than enough for chat.
Dense + sparse search + RRF	Yes	Vector search always ran on CPU, even in “GPU” stacks. RRF is sorting lists.
Cross-encoder reranker `bge-reranker-v2-m3` top-20/50	Yes, with care	A cross-encoder evaluates $k$ query-doc pairs: cost $\propto k$. Over 20-50 candidates it works; over hundreds at high QPS, no.
ColBERT late interaction	Marginal on CPU	The token-by-token matrix product of late interaction is heavy; viable at low volumes, suffers with QPS.
LLM generation	No, in practice	A 7B on CPU gives TTFT in seconds. Interactive latency = GPU.

Two rows deserve underlining because they dismantle myths.

“Vector search needs a GPU.” False from the start. The HNSW index, the navigable small-world graph used by Qdrant, pgvector with vector/halfvec, Milvus in CPU mode and almost everything else, was always built and traversed on CPU. Even the stacks advertised as “GPU-accelerated RAG” do the embedding on GPU but the ANN search stays on CPU in the vast majority of deployments; the GPU variants of the index (CAGRA and similar) are the expensive exception, not the norm, and are justified only with billions of vectors and extreme QPS. For a corporate corpus of millions of chunks, HNSW on CPU resolves in single-digit milliseconds.

“The reranker is a model, therefore GPU.” The bge-reranker-v2-m3 reranker is a cross-encoder of around 568M: it runs on CPU. The caveat is the number of pairs. A cross-encoder does not produce a reusable vector; it evaluates the (query, document) pair jointly, so its cost grows linearly with the number of candidates $k$:

$$\text{cost}_{\text{rerank}} \propto k \times \text{forward}(\text{query} + \text{doc})$$

Reranking the top-20 or top-50 coming out of hybrid retrieval is perfectly affordable on CPU. Reranking hundreds of candidates at high QPS is not: there the linear cost explodes and the GPU wins. The rule of thumb: broad cheap recall in the retriever, precision rerank over few candidates. (The detail of hybrid retrieval and reranking is in the fundamentals piece linked below.)

The numbers, with honest methodology

Here comes the part where a lot of people lie by omission. I am going to give throughput ranges, but they are literature ranges and orders of magnitude, not my own measurements on this hardware. Take them as such: the right decision does not depend on nailing the number, it depends on understanding the allocation.

For bge-m3 dense, sequence ≈256 tokens, embedding throughput moves roughly like this:

Platform	Dense throughput (order of magnitude)	Reading
High-end GPU (5090 fp16, TEI)	~12k tok/s+ (indicative)	The ceiling; expensive and scarce.
Large server CPU (Xeon ~56 cores, int8 ONNX)	low band of thousands tok/s	~1/5–1/10 of the GPU, but horizontally scalable and cheap.
Edge CPU / NUC (4-8 cores, int8)	tens to low hundreds tok/s	Enough for overnight ingestion of a local corpus.

The temptation is to read the second row as “CPU is 5-10× slower, discarded”. That is the wrong reading for ingestion. For batch work with no SLA, what rules is not absolute tok/s but throughput per euro and throughput per watt, and there the arithmetic changes sign.

Take a numerical example of allocation. Suppose a corpus of 2 million chunks of around 256 tokens that has to be re-indexed once a day (the corpus changes, the embeddings have to be redone). That is:

$$2 \times 10^6 \text{ chunks} \times 256 \text{ tok/chunk} \approx 5.1 \times 10^8 \text{ tokens}$$

At a conservative CPU throughput of, say, 3,000 tok/s per int8 Xeon server:

$$t_{\text{ingest}} \approx \frac{5.1 \times 10^8 \text{ tok}}{3000 \text{ tok/s}} \approx 1.7 \times 10^5 \text{ s} \approx 47 \text{ hours on a single server}$$

47 hours on one box sounds bad until you remember two things. First, this is embarrassingly parallel: the corpus is sliced and distributed; with 8 CPU servers it drops to around 6 hours, with 16 to around 3 hours, comfortably inside the overnight window. Second, and more importantly: that same work on the GPU blocks the GPU. If the H100 does 12k tok/s, it takes around 12 hours, but they are 12 hours of the H100, the resource the whole organisation fights over in order to generate. Spending the scarce, expensive resource on re-indexing a corpus that changes once a day is bad allocation, even if it is “faster”: you are optimising the wrong tok/s.

The mental rule: for ingestion, optimise throughput/€ and throughput/W; absolute tok/s belongs to the generation plane, where the human stopwatch really is running.

Decision tree: CPU or GPU for this piece?

Reference architecture (a): CPU-only

The first case is a node with no GPU: a NUC, an office Xeon, a sovereign edge server in a branch or in an air-gapped environment. The whole data plane lives there; generation is delegated to a remote GPU endpoint or done in batch with an SLM when latency is not pressing.

The stack:

TEI-CPU serving bge-m3 int8 with dense + sparse (same OpenAI contract /v1/embeddings, plus /rerank for the reranker).
Qdrant with a dense HNSW index plus sparse vectors, native RRF fusion.
Reranker bge-reranker-v2-m3 over the top-k (via TEI’s /rerank).
Gateway that orchestrates and, for generation, calls an external endpoint.

# docker-compose: full RAG data plane on CPU (no GPU)
services:
 tei-embed:
 image: ghcr.io/huggingface/text-embeddings-inference:cpu-latest
 command: ["--model-id", "BAAI/bge-m3", "--pooling", "cls", "--dtype", "int8"]
 ports: ["8081:80"]
 # ONNX/MKL backend: exploits AVX-512+VNNI / AMX if the Xeon supports it

 tei-rerank:
 image: ghcr.io/huggingface/text-embeddings-inference:cpu-latest
 command: ["--model-id", "BAAI/bge-reranker-v2-m3", "--dtype", "int8"]
 ports: ["8082:80"]
 # exposes /rerank — invoked ONLY over top-20/50, never over hundreds

 qdrant:
 image: qdrant/qdrant:latest
 ports: ["6333:6333"]
 volumes: ["./qdrant_storage:/qdrant/storage"]
 # HNSW dense + sparse vectors + RRF, all CPU

Hybrid search with RRF fusion in Qdrant (dense + sparse in a single query):

from qdrant_client import QdrantClient, models

client = QdrantClient(url="http://qdrant:6333")

# query embedding: dense and sparse from TEI-CPU (HTTP wiring omitted)
hits = client.query_points(
 collection_name="corpus",
 prefetch=[
 models.Prefetch(query=dense_vec, using="dense", limit=50),
 models.Prefetch(query=sparse_vec, using="sparse", limit=50),
 ],
 query=models.FusionQuery(fusion=models.Fusion.RRF), # native RRF
 limit=20,
).points
# -> then: POST /rerank (TEI) over these 20, keep the top-5
# -> then: the gateway sends query + top-5 to the GENERATION endpoint (GPU)

Generation, on this CPU-only node, leaves the node: the gateway builds the augmented prompt and sends it to a vLLM endpoint on the GPU cluster (or, if there is no interactive SLA, to an SLM on CPU in batch mode, accepting a TTFT of seconds). The entire data plane, everything above, runs without a single GPU.

Reference architecture (b): the recommended hybrid

This is the one I recommend for the general case with a GPU cluster available: data plane on CPU, generation plane on GPU, connected by OpenAI-compatible HTTP contracts so that each side is replaceable.

Generation server, minimal, on GPU:

# vLLM on the GPU cluster — generation ONLY
services:
 vllm-gen:
 image: vllm/vllm-openai:latest
 command: >
 --model meta-llama/Llama-3.1-8B-Instruct
 --dtype bfloat16 --max-model-len 8192
 --gpu-memory-utilization 0.85
 # exposes /v1/chat/completions — the gateway sends it query + the top-5 already retrieved
 deploy:
 resources:
 reservations:
 devices: [{driver: nvidia, count: 1, capabilities: [gpu]}]

The virtue of the design: because both sides speak the OpenAI contract over HTTP, the CPU data plane and the GPU generation plane scale separately and are replaceable. If tomorrow you want to move the rerank to GPU because QPS went up, you change one URL. If you want to add more CPU ingest nodes, you add them without touching generation. The whole stack is OSS and licence-clean: bge-m3 and bge-reranker-v2-m3 are MIT (bge-m3, reranker), Qdrant is Apache-2.0, TEI and vLLM are OSS.

Applied to the generic 4×H100 cluster

Let us bring this down to the cluster in the series: 4×H100 SXM 80 GB plus a generic CPU fleet (Xeon with AMX, NUCs). The correct allocation:

Build and indexing → CPU fleet. No H100 should spend a cycle re-embedding the corpus. That goes to the Xeon AMX machines (large servers, throughput in the thousands of tok/s in int8) or, for small local corpora, to the NUCs overnight. Overnight re-indexing of a corpus that changes once a day is the textbook case of “unhurried CPU work”.
The H100s → generation. All four cards are reserved for what only they do well: producing tokens at interactive latency. This is what the sibling pieces in the series, sharing a GPU and several models on one GPU, help to squeeze: once ingestion stops competing for the GPU, all the expensive silicon is free to generate and is shared better between models and tenants.
The H100s, at most, → rerank peaks or large embedders. If at some point you need a 7B embedder (gte-Qwen2, NV-Embed) for a domain where bge-m3 does not reach, or a massive rerank at a QPS the CPU cannot absorb, those peaks can indeed visit the GPU. But they are the occasional exception, not the base load.

The auditability angle: ENS / NIS2

There is a compliance argument that rarely gets mentioned and that the CPU/GPU split gives you almost for free.

A CPU-only node with no proprietary driver is easier to audit. There is no closed NVIDIA kernel stack, no CUDA and firmware versions to reconcile with the supply chain, no proprietary driver surface to document for an ENS or a NIS2 assessment. The whole data plane, chunking, embeddings, index, search, runs on OSS software on generic CPU with standard instructions. For a sovereign or classified environment, being able to say “the plane that touches the corpus depends on no proprietary binary” is a real argument, not marketing.

And there is a second auditability angle intrinsic to RAG done well: source traceability. A RAG system that retrieves identifiable chunks and cites them is auditable, since you can reconstruct which document each claim came from, as opposed to context-stuffing or the model’s opaque parametric knowledge, where there is no way to know where a fact came from. That traceability lives in the data plane (what was retrieved, from which source, with what score), precisely the plane we are putting on auditable CPU. The two arguments reinforce each other: the auditable silicon and the auditable chain of evidence are the same plane.

When NOT to take it to CPU

For honesty, and to avoid falling into the mirror image of the hype, the cases where CPU is not the answer:

Generation at interactive latency. The obvious case. A 7B on CPU gives TTFT in seconds: unacceptable for chat. If the user is waiting, generation goes to GPU. With no practical exceptions today.
Massive reranking at high QPS. A cross-encoder or ColBERT over hundreds of candidates, multiplied by many requests per second, saturates the CPU. The cost $\propto k \times \text{QPS}$ crosses the threshold where the GPU pays off. Keep CPU rerank bounded to top-20/50; if you need more breadth at higher QPS, move up to GPU.
Real-time re-indexing with a strict SLA. If the corpus changes continuously and freshness is measured in seconds (not hours), CPU throughput may not fit the window. There, ingest embedding may need a GPU, but note that this is rare: most corporate corpora change at the pace of hours or days, not seconds.
Large embedders (7B). bge-m3 (568M) is comfortable on CPU; a gte-Qwen2 or NV-Embed at 7B is LLM-class again and drags the same cost profile as generation. If your retrieval quality demands a 7B embedder, that embedder lives where 7Bs live: on the GPU.

The sentence that sums it all up: the CPU is the default home of the data plane; the GPU is the justified exception for what is latency-bound and massively online. Start by putting everything on CPU and move up to GPU only what proves it does not fit, not the other way round.

References

Chen, J., et al. BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings. arXiv 2402.03216. https://arxiv.org/abs/2402.03216
BAAI — BGE-M3 model card (568M params, XLM-RoBERTa, 8192 tokens, MIT). https://huggingface.co/BAAI/bge-m3
BAAI — bge-reranker-v2-m3 model card (cross-encoder on top of bge-m3, ~568M). https://huggingface.co/BAAI/bge-reranker-v2-m3
Intel + Hugging Face — CPU Optimized Embeddings with Optimum Intel and fastRAG (~10× indexing for BGE-large int8, 4th-gen Xeon). https://huggingface.co/blog/intel-fast-embedding
deepset / Haystack — CPU-Optimized Embedding Models with fastRAG and Haystack. https://haystack.deepset.ai/blog/cpu-optimized-models-with-fastrag
Hugging Face — Text Embeddings Inference (TEI), CPU ONNX/MKL backends, OpenAI-compatible endpoints. https://github.com/huggingface/text-embeddings-inference
Qdrant — fastembed (ONNX-CPU, dense/sparse/ColBERT) and Hybrid Search with RRF. https://github.com/qdrant/fastembed · https://qdrant.tech/documentation/beginner-tutorials/hybrid-search-fastembed/

Reranker and hybrid retrieval: the committee that decides the 5 chunks the LLM will actually read

Mon, 25 May 2026 14:00:00 +0200

This post takes apart the retrieval layer inside the RAG piece of the six-stage LLMOps pipeline. It sits directly on top of the corpus curation post: if the librarian decided what enters the index, the committee in this post decides what leaves the index for the LLM’s face. It is the stage that moves the real quality of a RAG in production the most, and the one most teams solve with “dense top-k and off we go”.

TL;DR

The most widespread antipattern in real RAG is: curated corpus, dense embeddings with bge-m3, the user’s query straight through, top_k=5 of cosine similarity and off to the LLM. The system delivers answers that look good in the demo, mediocre with real traffic and dreadful with three-word queries, technical jargon, internal abbreviations or anything outside the embedder’s fine-tuning domain. The root cause is almost never the model: it is that a single pass of dense retrieval over the raw query is structurally insufficient for the cases a real user produces. In 2026 the field has consolidated the three-layer retrieval pattern: a broad layer that mixes dense + sparse (BM25 / SPLADE) via Reciprocal Rank Fusion to secure recall, a fine layer with a cross-encoder reranker (BAAI/bge-reranker-v2-m3, Cohere Rerank 3, mxbai-rerank-large) that reorders the 30-50 candidates by true relevance, and optionally a late interaction layer with ColBERT-v2 or an LLM reranker for critical cases. On top of it all, a query rewriting pattern that rewrites malformed queries and HyDE, which generates a hypothetical document so you embed the expected answer instead of the question. This post goes through them one by one: why each layer exists, what latency budget it consumes, the minimum maths of RRF and of the recall/precision trade-off, the dominant OSS stack in 2026, the on-premise hardware for serving it under data sovereignty, and the seven operational traps that kill the stage.

The analogy: the committee for a professorial chair

A university department has a chair to fill and receives 3,000 applications. Nobody is going to read 3,000 CVs in depth. The real process is organised in rounds:

Round 1, mass pre-screening. A system of automatic filters reads each CV in seconds. One route looks at the keywords of the post (publications in specific journals, languages, certifications); another route looks at the overall profile (research area, h-index, career). The two routes run in parallel. Each proposes its shortlist of 30-50 candidates. The two rankings are combined and what remains is a list of some 50-100 candidates for the next round to act on. The metric that matters here is recall: not losing the good candidate through a filter error.

Round 2, careful reading. A member of the panel reads each CV on the shortlist thoroughly and scores it against the real profile of the post. It is slow, 10-20 minutes per CV, but it discriminates far better than the automatic filter. It reorders the 100 candidates into a fine ranking and keeps the 5 finalists. The metric that matters here is precision: that the top-5 is genuinely the real top-5.

Round 3 (optional), personal interview. Long interviews are held for the 5 finalists. Face to face, questions adapted to each candidate’s profile, a check against the reality of the post. It is extremely expensive and very slow, but for critical chairs it justifies the cost.

The retrieval of a modern RAG works exactly like this. Change the vocabulary and the structure holds:

The equivalence is exact. The broad layer guarantees that the right chunk is on the shortlist (recall). The fine layer orders well inside the shortlist (precision). Each layer is justified by an operational asymmetry: the broad one is cheap and parallelises, the fine one is expensive and sequential. Solving everything with the expensive one, a single cross-encoder over the whole corpus, is operationally unsustainable. Solving everything with the cheap one, a single dense top-k, is functionally insufficient.

The problem: why dense on its own is not enough

The argument against dense-only is not theoretical. It is that the queries a real user produces have patterns the dense bi-encoder does not capture well by construction:

Short queries. “IRPF dates 2026” goes through the embedder and produces a vector close to any chunk that mentions dates and IRPF, including chunks from 2023, chunks of generic calendars, chunks of corporate IRPF when you were asking about personal income tax. BM25 disambiguates better because the weight of the infrequent terms (“2026”, “IRPF”) dominates the score.

Infrequent or internal technical terms. Product codes (FBR-X42-PRO), organisation abbreviations (PYM-23-bis), internal variable names. The embedder, trained on a public corpus, breaks them down into sub-tokens and pushes them into the “noisy” zone of the latent space. Two completely different codes can end up 0.02 cosine distance apart. BM25 treats them as exact tokens and scores them correctly.

Polysemy. “Java” as a language vs Java as an island vs Java as coffee. The dense embedding averages the meanings; the ranking becomes mediocre. Here dense can beat BM25 if the query carries enough context, but that “if” is rarely met with real user queries.

Synonyms and rephrasings. “How do I cancel my subscription” vs a chunk titled “Service termination procedure”. Here dense beats BM25: the semantic space brings them together even though they share no tokens. It is exactly the case where dense shines and BM25 fails.

The operational conclusion: no single retriever covers the four cases. The combination covers them all. And for the cases where both retrievers fail (very ambiguous queries, contradictory context between chunks), the third layer comes in: the reranker that reads the query and each chunk together with a cross-encoder, and produces a real relevance score far more discriminating than cosine similarity.

Layer 1 — Hybrid retrieval: dense + sparse

Why BM25 is still alive in 2026

BM25 is from 1994. It is not deep learning. It needs no GPU. And it is still competitive with dense retrievers of hundreds of millions of parameters on many retrieval benchmarks (BEIR, 2021-2024). Its formula is simple:

$$ \mathrm{BM25}(q, d) = \sum_{t \in q} \mathrm{IDF}(t) \cdot \frac{\mathrm{tf}(t,d) \cdot (k_1 + 1)}{\mathrm{tf}(t,d) + k_1 \cdot (1 - b + b \cdot \frac{|d|}{\overline{|d|}})} $$

Where $\mathrm{tf}(t,d)$ is the frequency of term $t$ in document $d$, $|d|$ the document length, $\overline{|d|}$ the mean length of the corpus, and IDF the inverse document frequency. $k_1$ and $b$ are hyperparameters (typically $k_1 = 1.2$, $b = 0.75$). In practice you use pyserini, Tantivy, Elasticsearch or OpenSearch and never write the formula.

SPLADE (Formal et al., 2021) is the neural generation of BM25: it learns sparse weights over the BERT vocabulary, expanding each token to its related terms. It keeps BM25’s interpretability (you can see which query terms match which document terms) and beats plain BM25 on BEIR. The price is that you need GPU inference over the query (and over each document at indexing time, once).

The rule of thumb in 2026:

Corpus < 10M chunks, queries with a broad vocabulary: plain BM25 via Elasticsearch / Tantivy. Trivial indexing, 1-5ms queries.
Corpus 10M-100M, specific domain: SPLADE-v3 indexed in Elasticsearch or Vespa. Slow indexing (you need a GPU once), 5-20ms queries.
Corpus > 100M, latency critical: BM25 with external query expansion (synonyms, query2doc) is the operational option, with SPLADE reserved for the critical segments.

Dense bi-encoder: the second lane

The bi-encoder produces one vector per chunk and another per query, and relevance is the cosine similarity (or dot product) between the two. The dominant multilingual model in 2026 is BAAI/bge-m3: 568M params, 1024 dimensions, support for up to 8,192 context tokens per chunk, multivector (dense + sparse + ColBERT-style multi-vec in the same embedder), trained on 100+ languages. Alternatives:

Embedder	Params	Dim	Multilingual	Comment
`BAAI/bge-m3`	568M	1024	yes	the default all-rounder in 2026
`intfloat/multilingual-e5-large-instruct`	560M	1024	yes	direct competitor, instruct-style queries
`nomic-ai/nomic-embed-text-v1.5`	137M	768	English	fast, variable Matryoshka dimension
`jinaai/jina-embeddings-v3`	570M	1024	yes	with task-specific LoRAs per domain
`mixedbread-ai/mxbai-embed-large-v1`	335M	1024	English	top English on MTEB
`Snowflake/arctic-embed-l-v2.0`	568M	1024	yes	enterprise-oriented, Apache licence

The practical selection criterion: multilingual no matter what if your corpus or queries need it (bge-m3, multilingual-e5, jina-v3, arctic-l-v2), Matryoshka if you want to save vector store memory by reducing dimensions without re-embedding (nomic), and the MTEB leaderboard ranking only as a tie-breaker, never as the sole criterion, since MTEB benchmarks are contaminable and a 1-2 point difference rarely translates into a real improvement in your domain.

The fusion: Reciprocal Rank Fusion (RRF)

You have two rankings, one from BM25 and one from dense. How do you combine them? The naive option, adding the scores, fails because the scores are not comparable (BM25 produces numbers of 0-30, cosine 0-1, L2 distances 0-2…). The option that has consolidated is Reciprocal Rank Fusion (Cormack, Clarke, Buettcher 2009), which ignores absolute scores and uses only the rankings:

$$ \mathrm{RRF}(d) = \sum_{r \in R} \frac{1}{k + \mathrm{rank}_r(d)} $$

Where $R$ is the set of retrievers and $\mathrm{rank}_r(d)$ is the position of document $d$ in the ranking of retriever $r$. The constant $k$ is typically $60$ and smooths the weight of the first positions.

A numerical example. You have BM25 with top-5: [d_a, d_b, d_c, d_d, d_e] and dense with top-5: [d_b, d_f, d_a, d_g, d_h]. You compute the RRF of each candidate:

Doc	BM25 rank	dense rank	$\frac{1}{60+r_{\text{BM25}}}$	$\frac{1}{60+r_{\text{dense}}}$	total RRF
`d_b`	2	1	0.01613	0.01639	0.03252
`d_a`	1	3	0.01639	0.01587	0.03226
`d_c`	3	—	0.01587	0	0.01587
`d_f`	—	2	0	0.01613	0.01613
`d_d`	4	—	0.01562	0	0.01562

d_b wins because it appears well ranked in both. d_a follows it. Those that appear in only one list stay below. With nothing tuned, RRF tends to push the consensus candidates to the top, which is exactly what you want as a coarse filter before the reranker.

Variants: weighted RRF assigns a multiplier per retriever ($w_r \cdot \frac{1}{k + \mathrm{rank}_r(d)}$) when you have reasons to trust one more; learning-to-rank trains a model (LambdaMART or a LightGBM ranker) over features from the two retrievers, a more powerful option but one that demands relevance labels most teams do not have.

In Qdrant, Vespa, Weaviate and Elastic 8.11+ the RRF fusion is built in as a native mode: you ask for hybrid search with a lexical query and a vector and they return the combined top-k without you having to compute the fusion.

Layer 2 — The reranker: bi-encoder vs cross-encoder

The bi-encoder looks at query and document separately and compares the resulting vectors. It is extremely fast (a dot product) and lets you index the document embeddings once and for all. The price is that the model never sees query and document together, and the compressed representation loses nuance.

The cross-encoder looks at query and document concatenated: [CLS] query [SEP] document [SEP] goes through the model and the output is a relevance score. It discriminates far better, since shades of ordering and of local relationship between terms are captured, but the cost explodes: each (query, doc) pair requires a full inference of the model. You cannot precompute anything. If you want to rerank 50 candidates, that is 50 inferences.

Aspect	Bi-encoder	Cross-encoder
Inference per chunk at indexing	yes, once	not precomputable
Inference at query time	1 (query) + dot product per chunk	1 per (query, chunk) pair
Typical latency for top-50	5-20 ms (CPU possible)	80-300 ms (GPU recommended)
Discrimination	medium	high
Typical use	recall (layer 1)	precision (layer 2)

This asymmetry is why the canonical pattern is a two-step funnel: the bi-encoder does cheap broad recall and the cross-encoder does expensive fine precision, over 30-50 candidates instead of millions.

2026 OSS cross-encoder reranker stack

Reranker	Params	Multilingual	top-50 latency (A100)	Comment
`BAAI/bge-reranker-v2-m3`	568M	yes (100+ languages)	~90 ms	the de facto default in 2026
`BAAI/bge-reranker-v2-gemma`	2B	yes	~250 ms	more quality, more cost
`mixedbread-ai/mxbai-rerank-large-v2`	1.5B	English	~180 ms	top English on BEIR rerank
`jinaai/jina-reranker-v2-base-multilingual`	278M	yes	~50 ms	fast and enough for many cases
`Alibaba-NLP/gte-multilingual-reranker-base`	306M	yes	~55 ms	direct competitor to jina
`Cohere Rerank 3`	(closed)	yes (100+ languages)	~120 ms (API)	commercial, not on-prem
`Voyage rerank-2`	(closed)	yes	~100 ms (API)	commercial, not on-prem

For deployments under data sovereignty the obvious choice is bge-reranker-v2-m3 as a starting point, with jina-reranker-v2-base as the lightweight alternative. If the domain is exclusively English and the hardware budget allows it, mxbai-rerank-large-v2 takes a point or two more on BEIR. Cohere and Voyage are the option when you can reach out to an external API, ruled out in ENS / NIS2 scenarios with sensitive data.

Late interaction: ColBERT-v2 as a compromise

There is a third option gaining traction: late interaction, embodied in ColBERT-v2 (Santhanam et al., 2022). The idea is that the document is embedded not as a single vector but as a matrix of embeddings, one per token. And at query time the query is also embedded as a matrix, and the similarity is the sum of the maxima per column (MaxSim):

$$ S(q, d) = \sum_{i \in q} \max_{j \in d} \langle E_q[i], E_d[j] \rangle $$

The result is retrieval with quality close to a cross-encoder but at a much lower cost (there is no need to run the whole model over each pair). The price is space: each chunk is ~150-200 vectors instead of 1, and that multiplies the index size by ~100. For small to medium corpora (<10M chunks) it is manageable; for huge corpora PLAID compression and ColBERT-v2 quantization make it viable up to 100M.

In 2026 the reference implementations are RAGatouille (Python wrapper), JaColBERT (Japanese), Vespa.ai (a search engine with native ColBERT support as a first-class citizen), and the native integration in bge-m3 that produces the ColBERT-style multivectors as part of the same dense model.

My operational recommendation: start with a cross-encoder reranker over the top-30 of the hybrid retrieval. Only if the precision numbers fall short, try late interaction as an alternative to the cross-encoder (not as an additional layer). Late interaction as a third layer rarely pays back the operational cost for most cases.

Query rewriting and HyDE: correcting the query before retrieval

The user’s query is the worst-controlled input in the whole system. The most frequent problems:

Queries that are too short: “GDPR fines” → retrieves any chunk about GDPR or about fines.

Queries with typos or jargon: “i wnat to know how to cancel the premium subscripton” → the embedder is noisy because of the typos.

Multi-intent queries: “compare the premium and enterprise plans and tell me which has the better SLA” → dense will try to embed the three questions into one vector.

Queries with references resolved in context: “and for next month?” → without the previous context, dense does not know what it is about.

Canonical 2026 patterns to mitigate:

Query rewriting with a light LLM. Before retrieval, a small model (Qwen2.5-7B, Llama-3.2-3B, Phi-4-mini) rewrites the user’s query into a “canonical” version: it expands abbreviations, fixes typos, breaks multi-intent into sub-queries, resolves references using the conversation history. Cost: one cheap inference (~50-100 ms). Benefit: the four pathologies above are noticeably reduced.

Multi-query: the LLM generates 3-5 rephrasings of the query and you retrieve with each one; then you fuse with RRF. It increases recall at the cost of time and LLM tokens.

Step-back prompting (Zheng et al., 2023): the LLM generates a more general question than the original query (“what is a premium subscription in this system?”), you retrieve with both and combine. It helps with very specific queries that need context.

HyDE — Hypothetical Document Embeddings (Gao et al., 2022): instead of embedding the question, you ask a light LLM to generate a hypothetical answer and you embed that. The reasoning is that the semantic space of answers is closer to that of the real chunks than the space of questions. It works especially well in domains with very specific vocabulary. Cost: one inference of the light LLM (~100-200 ms). Benefit: in many cases +5-15 points of Recall@10.

The operational trade-off is the added latency. Each technique adds 50-300 ms to the pipeline. In synchronous interactions with a 2s SLO there is still headroom; in agentic interactions with tool-use loops, every ms counts. The practical choice in 2026: rewriting with a light LLM as the default, HyDE for specific cases where it has been measured to help, multi-query and step-back only when the case justifies it with numbers.

Minimum maths: latency budget and metrics

Latency budget of a canonical pipeline

A typical hybrid + rerank pipeline in 2026 production over a corpus of 5M chunks with bge-m3 + bge-reranker-v2-m3 on an H100 with TEI:

Stage	p50 latency	p95 latency	Comment
Query rewrite (LLM 3B)	60 ms	150 ms	only if enabled
HyDE (LLM 3B)	90 ms	220 ms	only if enabled
BM25 / SPLADE top-50	8 ms	20 ms	Elasticsearch or local Tantivy
Dense embedding of the query	12 ms	35 ms	bge-m3 with batch=1
Dense top-50 (filterable HNSW)	6 ms	18 ms	Qdrant with scalar quantization
RRF fusion → top-30	<1 ms	<1 ms	arithmetic
Cross-encoder rerank top-30	60 ms	180 ms	bge-reranker-v2-m3 batched
Top-5 selection + format	<1 ms	<1 ms
Total without rewrite/HyDE	~90 ms	~270 ms	realistic range on an H100
Total with rewrite + HyDE	~240 ms	~640 ms	+ ~150-370 ms

On an RTX 4090 (~3× slower than an H100 on a large cross-encoder, similar on everything else) the numbers shift: ~180 ms p50 without rewrite, ~400 ms p50 with rewrite + HyDE. It is perfectly serviceable for a synchronous assistant with a 2-3s SLO, but tight for one with a 500 ms SLO.

Metrics: nDCG, MRR, Recall@k

There are three classic metrics that any retrieval eval reports:

Recall@k: is the relevant chunk among the top-k? A binary metric, it ignores ordering. The one that matters for the broad layer.

$$ \mathrm{Recall@k} = \frac{|\{\text{relevant}\} \cap \{\text{top-k}\}|}{|\{\text{relevant}\}|} $$

Mean Reciprocal Rank (MRR@k): in which position does the first relevant chunk appear? It penalises placing it far down.

$$ \mathrm{MRR@k} = \frac{1}{|Q|} \sum_{q \in Q} \frac{1}{\mathrm{rank}_q} $$

where $\mathrm{rank}_q$ is the position of the first relevant item for query $q$ (or $\infty$ if it does not appear in the top-k).

nDCG@k: it takes into account multiple relevant chunks with degrees of relevance. The most used metric for evaluating rerankers:

$$ \mathrm{DCG@k} = \sum_{i=1}^{k} \frac{2^{\mathrm{rel}_i} - 1}{\log_2(i+1)} \quad \mathrm{nDCG@k} = \frac{\mathrm{DCG@k}}{\mathrm{IDCG@k}} $$

The operational heuristic: Recall@50 measures the quality of your broad layer (we want > 95%; otherwise layer 2 will never be able to recover what was lost), nDCG@5 measures the quality of your reranker (we want > 0.75 on the golden set to consider the system “good”) and MRR@5 measures the quality of your top-1 (it matters especially in systems with top_k=1 or where the LLM’s answer is based mainly on the first chunk).

Costs and throughput

Assuming a bge-reranker-v2-m3 cross-encoder (568M params) on an H100 SXM with TEI 1.7 and dynamic batching:

Throughput: ~280 queries/sec with top-30 each (= ~8,400 chunk-query pairs/sec).
VRAM used: ~3.5 GB for the model + ~6 GB of KV cache + activations at batch=32.
Energy cost: an H100 SXM draws ~700 W. At 8,400 pairs/sec, the energy cost of pure reranking is ~0.083 mJ per pair. In Spanish industrial electricity bill terms (~0.12 €/kWh, May 2026): ~0.000003 € per rerank of 30 candidates. The bill is negligible compared with the cost of the downstream LLM.

On an RTX 4090 with TEI: ~95 queries/sec at top-30 (~2,850 pairs/sec), VRAM ~3.5 GB + ~5 GB. Serviceable for an internal assistant with modest traffic (~5 QPS sustained, peaks at 20-30).

The canonical pattern — which piece goes where

The 2026 operational reference stack for an on-premise hybrid + rerank RAG under data sovereignty:

 ┌─────────────────┐
User query ────────▶│ Query rewriter │ (optional, LLM 3B)
│ (Qwen2.5-7B-IT) │
└────────┬────────┘
▼
┌─────────────────┐
│ HyDE │ (optional)
│ (same LLM) │
└────────┬────────┘
┌─────────────────┼─────────────────┐
▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ BM25/SPLADE │ │ Dense bi-enc │ │ (metadata │
│ Elasticsearch│ │ bge-m3 + TEI │ │ filters │
│ top-50 │ │ + Qdrant │ │ per tenant) │
│ │ │ top-50 │ │ │
└──────┬───────┘ └──────┬───────┘ └──────┬───────┘
└─────────┬───────┘ │
▼ │
┌──────────────┐ │
│ RRF k=60 │◀─────────────────┘
│ → top-30 │
└──────┬───────┘
▼
┌──────────────────┐
│ Cross-encoder │
│ bge-reranker-v2 │
│ TEI batch=32 │
│ → final top-5 │
└──────┬───────────┘
▼
┌──────────────────┐
│ LLM context │
│ ~2,000 tokens │
└──────────────────┘

Key services:

Sparse index: Elasticsearch 8.16, OpenSearch 2.18 or embedded Tantivy. Native BM25, SPLADE optional via plugin.
Vector index: Qdrant 1.13 (filterable HNSW, scalar quantization, multivectors). Covered in detail in the PostgreSQL + Qdrant in ingestion post.
Embedder serving: HuggingFace’s TEI (Text Embeddings Inference) for bge-m3, multilingual-e5 and compatible models. OpenAI-compatible endpoint.
Reranker serving: TEI too, for the rerankers of the bge-reranker, jina-reranker and mxbai-rerank families. Separate /rerank endpoint.
LLM for rewriter/HyDE: vLLM 0.7+ serving Qwen2.5-7B-Instruct or Phi-4-mini-Instruct. If the main LLM is the same, it shares resources; if not, a dedicated pod with GPTQ-INT4 quantization.
Orchestration: the RAG client (FastAPI, LangChain, LlamaIndex) composes the three lanes. Each parallel call (Sparse + Dense) is made concurrently; rewriter, HyDE and rerank are sequential.

Minimum Kubernetes manifests are already developed in the vLLM on Kubernetes post and the full orchestration piece fits the pattern of the multi-tenant H100 cluster.

On-premise hardware: what fits in an RTX 4090 vs a generic 4×H100 SXM configuration

The operational reality matters. Two typical profiles:

Modest configuration — 1× RTX 4090 (24 GB Ada Lovelace)

It comfortably serves:

bge-m3 (568M) for dense embedding of the query → ~1 GB VRAM
bge-reranker-v2-m3 (568M) cross-encoder → ~3.5 GB VRAM at batch=32
A light LLM for rewriter/HyDE (Qwen2.5-7B GPTQ-INT4) → ~5-6 GB VRAM
The main LLM (Llama-3.1-8B-Instruct FP8) → ~11-12 GB VRAM

Total VRAM used: ~21 GB. That leaves ~3 GB for the main LLM’s KV cache, enough for 4-8 concurrent users with ~4K token contexts. For an internal corporate assistant it is viable. If you need a larger context or more concurrency, you have to split things up: either the main LLM is served outside the 4090 (another GPU, another node), or you fall back on quantising the reranker (not advisable: you lose the nDCG points that justify the cost of adding it).

Generic configuration — 4×H100 SXM (320 GB total, NVLink)

It comfortably serves everything above multiplied by 30-50× in throughput and opens the door to:

A large main LLM (Llama 3.3 70B FP8 or Qwen2.5-72B FP8) on 2 GPUs with tensor parallelism
Large rerankers (bge-reranker-v2-gemma 2B) with quality ~2 points higher
Late interaction (ColBERT-v2) on one of the nodes for critical cases
Several light LLMs in parallel for rewriter, HyDE and the evals judge

The practical rule: the reranker is one of the most efficient things you can put on a GPU. The VRAM used is modest, the throughput is high, and the quality improvement per euro of extra hardware is among the best in the RAG stack.

The seven traps that kill retrieval

Trap 1 — Top-k to the LLM with no reranker. Dense retrieval returns a top-5 and those 5 chunks go straight to the LLM. Without a reranker, the order of the 5 is the cosine similarity order, which is not the real relevance order. The LLM answers based much more on the first chunks than the last ones, and the nuance is lost. Symptom: confident answers with citations that are not the most relevant in the corpus.

Trap 2 — Dense-only with no BM25. Short queries and queries with internal jargon work badly, the demos look pretty, real traffic complains. Symptom: on QA with real support tickets, the system fails specifically on queries of fewer than 5 words or with product codes.

Trap 3 — Reranker badly calibrated on top-k. A layer 1 top-k that is too small (top-10) cuts before the reranker can do its job. A top-k that is too large (top-200) blows up latency without improving nDCG. The canonical sweet spot is top-30 to top-50 from the hybrid → top-5 to top-10 from the reranker. Symptom: the reranker’s nDCG@5 stuck no matter which reranker model you try.

Trap 4 — Embedder and reranker in different languages. The corpus is Spanish + Catalan, the embedder is all-MiniLM (English only), the reranker is bge-reranker-v2-m3 (multilingual). The broad layer retrieves badly because of the embedder, and the reranker has little to rerank. Symptom: Recall@50 < 80%, unrecoverable by changing the reranker.

Trap 5 — RRF with k=1. Someone reads the paper, copies the formula and sets k=1 “because they do not understand what it does”. The result: the first positions of the worst ranking (whichever it is) dominate. The fusion stops averaging and becomes “winner-takes-all”. Symptom: the final ranking is almost identical to that of just one of the retrievers.

Trap 6 — A query rewriter that changes the meaning. The LLM rewriter “improves” "how do I cancel" into "termination procedure for the premium service for corporate customers". It retrieves chunks about corporate customers when the user was a private individual. Over-specification by the rewriter is worse than no rewriter. Symptom: queries from “normal” users come back worse than the old ones. Mitigation: the rewriter must keep the user’s intent and only add context, never assume.

Trap 7 — No retrieval telemetry. The system serves, the user complains, you do not know whether the fault was the corpus, the retrieval, the reranker or the LLM. Without emitting traces with retrieved_chunks_ids, retrieved_chunks_scores, rerank_scores, query_rewritten_to, selected_top_k, debugging is theatre. Symptom: each incident takes days to investigate. The tracing with OTel and MCP piece covers the canonical pattern.

All seven are operational. Just as with the corpus, retrieval does not break because the maths is wrong: it breaks because the discipline relaxes. And as in Eval, the metrics can go up while the real experience gets worse, because the golden set accommodates itself to the system instead of the system accommodating itself to the golden set.

What we have not covered (upcoming posts)

Embedding model selection and fine-tuning: how to choose among the embedders on the MTEB leaderboard without falling into goodharting, when and how to fine-tune an embedder on your own domain with MNR loss or triplet loss, which synthetic dataset is generated with an LLM for training (GPL, InPars, Promptagator), and the gotchas of massive re-embedding when you change model.
Semantic cache for RAG: how to cache semantically similar queries to serve answers without going through retrieval or the LLM, GPTCache, MeanCache, and the cache’s precision/coverage trade-off. It saves 30-70% of the cost on workloads with repeated queries.
Multi-vector and ColBERT-v2 at scale: how PLAID and CITADEL indexes are designed to serve corpora of 100M+ chunks with late interaction without burning the memory budget.
RAG-specific eval — RAGAS deep dive: faithfulness, answer relevance, context precision, context recall, noise sensitivity. How the RAG-specific golden set is built (with chunks labelled as relevant/irrelevant) and which metrics correlate with real user satisfaction.
Function calling and tool-augmented retrieval: when the LLM decides which retriever to invoke (SQL for structured data, vector for unstructured, web search for real time), the ReAct pattern, handling tool errors.
Agentic retrieval loops: when a single retrieval pass is not enough and the agent iterates (planning, sub-queries, summary-and-refine). The latency / quality trade-off and the anti-patterns (infinite loops, diverging sub-queries).

References

BEIR: Thakur et al. (2021). “BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models.” NeurIPS 2021 Datasets and Benchmarks Track. https://github.com/beir-cellar/beir
SPLADE: Formal et al. (2021). “SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking.” SIGIR 2021. https://arxiv.org/abs/2107.05720
ColBERT-v2: Santhanam et al. (2022). “ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction.” NAACL 2022. https://arxiv.org/abs/2112.01488
Reciprocal Rank Fusion: Cormack, Clarke, Buettcher (2009). “Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods.” SIGIR 2009. https://dl.acm.org/doi/10.1145/1571941.1572114
HyDE: Gao et al. (2022). “Precise Zero-Shot Dense Retrieval without Relevance Labels.” https://arxiv.org/abs/2212.10496
Step-back prompting: Zheng et al. (2023). “Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models.” https://arxiv.org/abs/2310.06117
BGE-M3: Chen et al. (2024). “BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation.” https://arxiv.org/abs/2402.03216
MTEB leaderboard: https://huggingface.co/spaces/mteb/leaderboard — útil como punto de partida, peligroso como criterio único (overfitting al benchmark, dataset contamination).
TEI (Text Embeddings Inference): https://github.com/huggingface/text-embeddings-inference — el motor de serving de HuggingFace para embedders y rerankers de tamaño pequeño-mediano. Endpoint OpenAI-compatible.
Qdrant hybrid search: https://qdrant.tech/documentation/concepts/hybrid-queries/ — implementación nativa de RRF y multi-vector queries.
RAGatouille: https://github.com/AnswerDotAI/RAGatouille — wrapper Python para ColBERT-v2 que reduce drásticamente la curva de entrada.
Cohere Rerank 3: https://docs.cohere.com/docs/rerank-2 — referencia técnica del reranker comercial multilingüe líder.

Reranker on lo0 — Tech Blog

Serving embeddings and rerankers with TEI in production

TL;DR

The analogy: the stamping press that only stamps vectors

What TEI is, exactly

The endpoints

The backends

Why a dedicated server and not calling the model inline

Dynamic batching: the mechanics of the press

The number: throughput with batching vs without batching

Deployment: compose for embedder and reranker

Embedder (bge-m3) on GPU

Reranker (bge-reranker-v2-m3) on GPU

CPU for batch ingestion

Integration: how ingestion and the gateway consume it

Sizing: memory, replicas and latency budget

Footprint of a 568M encoder

Latency budget: short query vs top-50 rerank

Applied to the generic 4×H100 cluster

Conclusion

See also

References

Taking RAG to the CPU: separating the data plane from the generation plane

TL;DR

The analogy: the library and the librarian

The three phases and their compute profiles

Why ingestion fits on CPU: the embedder is not an LLM

Runtimes that already do this effortlessly

Viability table: CPU for each component?

The numbers, with honest methodology

Decision tree: CPU or GPU for this piece?

Reference architecture (a): CPU-only

Reference architecture (b): the recommended hybrid

Applied to the generic 4×H100 cluster

The auditability angle: ENS / NIS2

When NOT to take it to CPU

See also

References

Reranker and hybrid retrieval: the committee that decides the 5 chunks the LLM will actually read

TL;DR

The analogy: the committee for a professorial chair

The problem: why dense on its own is not enough

Layer 1 — Hybrid retrieval: dense + sparse

Why BM25 is still alive in 2026

Dense bi-encoder: the second lane

The fusion: Reciprocal Rank Fusion (RRF)

Layer 2 — The reranker: bi-encoder vs cross-encoder

2026 OSS cross-encoder reranker stack

Late interaction: ColBERT-v2 as a compromise

Query rewriting and HyDE: correcting the query before retrieval

Minimum maths: latency budget and metrics

Latency budget of a canonical pipeline

Metrics: nDCG, MRR, Recall@k

Costs and throughput

The canonical pattern — which piece goes where

On-premise hardware: what fits in an RTX 4090 vs a generic 4×H100 SXM configuration

The seven traps that kill retrieval

What we have not covered (upcoming posts)

See also

References

Embedder (`bge-m3`) on GPU

Reranker (`bge-reranker-v2-m3`) on GPU