Reranker and hybrid retrieval: the committee that decides the 5 chunks the LLM will actually read

· By José Manuel Román
Contents

This post takes apart the retrieval layer inside the RAG piece of the six-stage LLMOps pipeline. It sits directly on top of the corpus curation post: if the librarian decided what enters the index, the committee in this post decides what leaves the index for the LLM’s face. It is the stage that moves the real quality of a RAG in production the most, and the one most teams solve with “dense top-k and off we go”.

TL;DR

The most widespread antipattern in real RAG is: curated corpus, dense embeddings with bge-m3, the user’s query straight through, top_k=5 of cosine similarity and off to the LLM. The system delivers answers that look good in the demo, mediocre with real traffic and dreadful with three-word queries, technical jargon, internal abbreviations or anything outside the embedder’s fine-tuning domain. The root cause is almost never the model: it is that a single pass of dense retrieval over the raw query is structurally insufficient for the cases a real user produces. In 2026 the field has consolidated the three-layer retrieval pattern: a broad layer that mixes dense + sparse (BM25 / SPLADE) via Reciprocal Rank Fusion to secure recall, a fine layer with a cross-encoder reranker (BAAI/bge-reranker-v2-m3, Cohere Rerank 3, mxbai-rerank-large) that reorders the 30-50 candidates by true relevance, and optionally a late interaction layer with ColBERT-v2 or an LLM reranker for critical cases. On top of it all, a query rewriting pattern that rewrites malformed queries and HyDE, which generates a hypothetical document so you embed the expected answer instead of the question. This post goes through them one by one: why each layer exists, what latency budget it consumes, the minimum maths of RRF and of the recall/precision trade-off, the dominant OSS stack in 2026, the on-premise hardware for serving it under data sovereignty, and the seven operational traps that kill the stage.

The analogy: the committee for a professorial chair

A university department has a chair to fill and receives 3,000 applications. Nobody is going to read 3,000 CVs in depth. The real process is organised in rounds:

Round 1, mass pre-screening. A system of automatic filters reads each CV in seconds. One route looks at the keywords of the post (publications in specific journals, languages, certifications); another route looks at the overall profile (research area, h-index, career). The two routes run in parallel. Each proposes its shortlist of 30-50 candidates. The two rankings are combined and what remains is a list of some 50-100 candidates for the next round to act on. The metric that matters here is recall: not losing the good candidate through a filter error.

Round 2, careful reading. A member of the panel reads each CV on the shortlist thoroughly and scores it against the real profile of the post. It is slow, 10-20 minutes per CV, but it discriminates far better than the automatic filter. It reorders the 100 candidates into a fine ranking and keeps the 5 finalists. The metric that matters here is precision: that the top-5 is genuinely the real top-5.

Round 3 (optional), personal interview. Long interviews are held for the 5 finalists. Face to face, questions adapted to each candidate’s profile, a check against the reality of the post. It is extremely expensive and very slow, but for critical chairs it justifies the cost.

The retrieval of a modern RAG works exactly like this. Change the vocabulary and the structure holds:

User query"how do I cancel subscription"+ rewriter / HyDELayer 1a · SparseBM25 / SPLADEtop-50 lexicalLayer 1b · Densebi-encoder (bge-m3)top-50 semanticLayer 2 · FusionReciprocal Rank Fusion→ top-30 combinedLayer 3 · Rerankercross-encoder bge-rerank-v2→ top-5 reorderedLayer 4 (optional) · ColBERT / LLM rerankerlate interaction or LLM judgefor critical casesLLM context5 chunks ready to answer~2,000 tokensbroad recall~50-100ms totalk=60 typicalfine precision~80-300ms

The equivalence is exact. The broad layer guarantees that the right chunk is on the shortlist (recall). The fine layer orders well inside the shortlist (precision). Each layer is justified by an operational asymmetry: the broad one is cheap and parallelises, the fine one is expensive and sequential. Solving everything with the expensive one, a single cross-encoder over the whole corpus, is operationally unsustainable. Solving everything with the cheap one, a single dense top-k, is functionally insufficient.

The problem: why dense on its own is not enough

The argument against dense-only is not theoretical. It is that the queries a real user produces have patterns the dense bi-encoder does not capture well by construction:

Short queries. “IRPF dates 2026” goes through the embedder and produces a vector close to any chunk that mentions dates and IRPF, including chunks from 2023, chunks of generic calendars, chunks of corporate IRPF when you were asking about personal income tax. BM25 disambiguates better because the weight of the infrequent terms (“2026”, “IRPF”) dominates the score.

Infrequent or internal technical terms. Product codes (FBR-X42-PRO), organisation abbreviations (PYM-23-bis), internal variable names. The embedder, trained on a public corpus, breaks them down into sub-tokens and pushes them into the “noisy” zone of the latent space. Two completely different codes can end up 0.02 cosine distance apart. BM25 treats them as exact tokens and scores them correctly.

Polysemy. “Java” as a language vs Java as an island vs Java as coffee. The dense embedding averages the meanings; the ranking becomes mediocre. Here dense can beat BM25 if the query carries enough context, but that “if” is rarely met with real user queries.

Synonyms and rephrasings. “How do I cancel my subscription” vs a chunk titled “Service termination procedure”. Here dense beats BM25: the semantic space brings them together even though they share no tokens. It is exactly the case where dense shines and BM25 fails.

The operational conclusion: no single retriever covers the four cases. The combination covers them all. And for the cases where both retrievers fail (very ambiguous queries, contradictory context between chunks), the third layer comes in: the reranker that reads the query and each chunk together with a cross-encoder, and produces a real relevance score far more discriminating than cosine similarity.

Layer 1 — Hybrid retrieval: dense + sparse

Why BM25 is still alive in 2026

BM25 is from 1994. It is not deep learning. It needs no GPU. And it is still competitive with dense retrievers of hundreds of millions of parameters on many retrieval benchmarks (BEIR, 2021-2024). Its formula is simple:

$$ \mathrm{BM25}(q, d) = \sum_{t \in q} \mathrm{IDF}(t) \cdot \frac{\mathrm{tf}(t,d) \cdot (k_1 + 1)}{\mathrm{tf}(t,d) + k_1 \cdot (1 - b + b \cdot \frac{|d|}{\overline{|d|}})} $$

Where $\mathrm{tf}(t,d)$ is the frequency of term $t$ in document $d$, $|d|$ the document length, $\overline{|d|}$ the mean length of the corpus, and IDF the inverse document frequency. $k_1$ and $b$ are hyperparameters (typically $k_1 = 1.2$, $b = 0.75$). In practice you use pyserini, Tantivy, Elasticsearch or OpenSearch and never write the formula.

SPLADE (Formal et al., 2021) is the neural generation of BM25: it learns sparse weights over the BERT vocabulary, expanding each token to its related terms. It keeps BM25’s interpretability (you can see which query terms match which document terms) and beats plain BM25 on BEIR. The price is that you need GPU inference over the query (and over each document at indexing time, once).

The rule of thumb in 2026:

  • Corpus < 10M chunks, queries with a broad vocabulary: plain BM25 via Elasticsearch / Tantivy. Trivial indexing, 1-5ms queries.
  • Corpus 10M-100M, specific domain: SPLADE-v3 indexed in Elasticsearch or Vespa. Slow indexing (you need a GPU once), 5-20ms queries.
  • Corpus > 100M, latency critical: BM25 with external query expansion (synonyms, query2doc) is the operational option, with SPLADE reserved for the critical segments.

Dense bi-encoder: the second lane

The bi-encoder produces one vector per chunk and another per query, and relevance is the cosine similarity (or dot product) between the two. The dominant multilingual model in 2026 is BAAI/bge-m3: 568M params, 1024 dimensions, support for up to 8,192 context tokens per chunk, multivector (dense + sparse + ColBERT-style multi-vec in the same embedder), trained on 100+ languages. Alternatives:

EmbedderParamsDimMultilingualComment
BAAI/bge-m3568M1024yesthe default all-rounder in 2026
intfloat/multilingual-e5-large-instruct560M1024yesdirect competitor, instruct-style queries
nomic-ai/nomic-embed-text-v1.5137M768Englishfast, variable Matryoshka dimension
jinaai/jina-embeddings-v3570M1024yeswith task-specific LoRAs per domain
mixedbread-ai/mxbai-embed-large-v1335M1024Englishtop English on MTEB
Snowflake/arctic-embed-l-v2.0568M1024yesenterprise-oriented, Apache licence

The practical selection criterion: multilingual no matter what if your corpus or queries need it (bge-m3, multilingual-e5, jina-v3, arctic-l-v2), Matryoshka if you want to save vector store memory by reducing dimensions without re-embedding (nomic), and the MTEB leaderboard ranking only as a tie-breaker, never as the sole criterion, since MTEB benchmarks are contaminable and a 1-2 point difference rarely translates into a real improvement in your domain.

The fusion: Reciprocal Rank Fusion (RRF)

You have two rankings, one from BM25 and one from dense. How do you combine them? The naive option, adding the scores, fails because the scores are not comparable (BM25 produces numbers of 0-30, cosine 0-1, L2 distances 0-2…). The option that has consolidated is Reciprocal Rank Fusion (Cormack, Clarke, Buettcher 2009), which ignores absolute scores and uses only the rankings:

$$ \mathrm{RRF}(d) = \sum_{r \in R} \frac{1}{k + \mathrm{rank}_r(d)} $$

Where $R$ is the set of retrievers and $\mathrm{rank}_r(d)$ is the position of document $d$ in the ranking of retriever $r$. The constant $k$ is typically $60$ and smooths the weight of the first positions.

A numerical example. You have BM25 with top-5: [d_a, d_b, d_c, d_d, d_e] and dense with top-5: [d_b, d_f, d_a, d_g, d_h]. You compute the RRF of each candidate:

DocBM25 rankdense rank$\frac{1}{60+r_{\text{BM25}}}$$\frac{1}{60+r_{\text{dense}}}$total RRF
d_b210.016130.016390.03252
d_a130.016390.015870.03226
d_c3—0.0158700.01587
d_f—200.016130.01613
d_d4—0.0156200.01562

d_b wins because it appears well ranked in both. d_a follows it. Those that appear in only one list stay below. With nothing tuned, RRF tends to push the consensus candidates to the top, which is exactly what you want as a coarse filter before the reranker.

Variants: weighted RRF assigns a multiplier per retriever ($w_r \cdot \frac{1}{k + \mathrm{rank}_r(d)}$) when you have reasons to trust one more; learning-to-rank trains a model (LambdaMART or a LightGBM ranker) over features from the two retrievers, a more powerful option but one that demands relevance labels most teams do not have.

In Qdrant, Vespa, Weaviate and Elastic 8.11+ the RRF fusion is built in as a native mode: you ask for hybrid search with a lexical query and a vector and they return the combined top-k without you having to compute the fusion.

Layer 2 — The reranker: bi-encoder vs cross-encoder

The bi-encoder looks at query and document separately and compares the resulting vectors. It is extremely fast (a dot product) and lets you index the document embeddings once and for all. The price is that the model never sees query and document together, and the compressed representation loses nuance.

The cross-encoder looks at query and document concatenated: [CLS] query [SEP] document [SEP] goes through the model and the output is a relevance score. It discriminates far better, since shades of ordering and of local relationship between terms are captured, but the cost explodes: each (query, doc) pair requires a full inference of the model. You cannot precompute anything. If you want to rerank 50 candidates, that is 50 inferences.

AspectBi-encoderCross-encoder
Inference per chunk at indexingyes, oncenot precomputable
Inference at query time1 (query) + dot product per chunk1 per (query, chunk) pair
Typical latency for top-505-20 ms (CPU possible)80-300 ms (GPU recommended)
Discriminationmediumhigh
Typical userecall (layer 1)precision (layer 2)

This asymmetry is why the canonical pattern is a two-step funnel: the bi-encoder does cheap broad recall and the cross-encoder does expensive fine precision, over 30-50 candidates instead of millions.

2026 OSS cross-encoder reranker stack

RerankerParamsMultilingualtop-50 latency (A100)Comment
BAAI/bge-reranker-v2-m3568Myes (100+ languages)~90 msthe de facto default in 2026
BAAI/bge-reranker-v2-gemma2Byes~250 msmore quality, more cost
mixedbread-ai/mxbai-rerank-large-v21.5BEnglish~180 mstop English on BEIR rerank
jinaai/jina-reranker-v2-base-multilingual278Myes~50 msfast and enough for many cases
Alibaba-NLP/gte-multilingual-reranker-base306Myes~55 msdirect competitor to jina
Cohere Rerank 3(closed)yes (100+ languages)~120 ms (API)commercial, not on-prem
Voyage rerank-2(closed)yes~100 ms (API)commercial, not on-prem

For deployments under data sovereignty the obvious choice is bge-reranker-v2-m3 as a starting point, with jina-reranker-v2-base as the lightweight alternative. If the domain is exclusively English and the hardware budget allows it, mxbai-rerank-large-v2 takes a point or two more on BEIR. Cohere and Voyage are the option when you can reach out to an external API, ruled out in ENS / NIS2 scenarios with sensitive data.

Late interaction: ColBERT-v2 as a compromise

There is a third option gaining traction: late interaction, embodied in ColBERT-v2 (Santhanam et al., 2022). The idea is that the document is embedded not as a single vector but as a matrix of embeddings, one per token. And at query time the query is also embedded as a matrix, and the similarity is the sum of the maxima per column (MaxSim):

$$ S(q, d) = \sum_{i \in q} \max_{j \in d} \langle E_q[i], E_d[j] \rangle $$

The result is retrieval with quality close to a cross-encoder but at a much lower cost (there is no need to run the whole model over each pair). The price is space: each chunk is ~150-200 vectors instead of 1, and that multiplies the index size by ~100. For small to medium corpora (<10M chunks) it is manageable; for huge corpora PLAID compression and ColBERT-v2 quantization make it viable up to 100M.

In 2026 the reference implementations are RAGatouille (Python wrapper), JaColBERT (Japanese), Vespa.ai (a search engine with native ColBERT support as a first-class citizen), and the native integration in bge-m3 that produces the ColBERT-style multivectors as part of the same dense model.

My operational recommendation: start with a cross-encoder reranker over the top-30 of the hybrid retrieval. Only if the precision numbers fall short, try late interaction as an alternative to the cross-encoder (not as an additional layer). Late interaction as a third layer rarely pays back the operational cost for most cases.

Query rewriting and HyDE: correcting the query before retrieval

The user’s query is the worst-controlled input in the whole system. The most frequent problems:

Queries that are too short: “GDPR fines” → retrieves any chunk about GDPR or about fines.

Queries with typos or jargon: “i wnat to know how to cancel the premium subscripton” → the embedder is noisy because of the typos.

Multi-intent queries: “compare the premium and enterprise plans and tell me which has the better SLA” → dense will try to embed the three questions into one vector.

Queries with references resolved in context: “and for next month?” → without the previous context, dense does not know what it is about.

Canonical 2026 patterns to mitigate:

Query rewriting with a light LLM. Before retrieval, a small model (Qwen2.5-7B, Llama-3.2-3B, Phi-4-mini) rewrites the user’s query into a “canonical” version: it expands abbreviations, fixes typos, breaks multi-intent into sub-queries, resolves references using the conversation history. Cost: one cheap inference (~50-100 ms). Benefit: the four pathologies above are noticeably reduced.

Multi-query: the LLM generates 3-5 rephrasings of the query and you retrieve with each one; then you fuse with RRF. It increases recall at the cost of time and LLM tokens.

Step-back prompting (Zheng et al., 2023): the LLM generates a more general question than the original query (“what is a premium subscription in this system?”), you retrieve with both and combine. It helps with very specific queries that need context.

HyDE — Hypothetical Document Embeddings (Gao et al., 2022): instead of embedding the question, you ask a light LLM to generate a hypothetical answer and you embed that. The reasoning is that the semantic space of answers is closer to that of the real chunks than the space of questions. It works especially well in domains with very specific vocabulary. Cost: one inference of the light LLM (~100-200 ms). Benefit: in many cases +5-15 points of Recall@10.

The operational trade-off is the added latency. Each technique adds 50-300 ms to the pipeline. In synchronous interactions with a 2s SLO there is still headroom; in agentic interactions with tool-use loops, every ms counts. The practical choice in 2026: rewriting with a light LLM as the default, HyDE for specific cases where it has been measured to help, multi-query and step-back only when the case justifies it with numbers.

Minimum maths: latency budget and metrics

Latency budget of a canonical pipeline

A typical hybrid + rerank pipeline in 2026 production over a corpus of 5M chunks with bge-m3 + bge-reranker-v2-m3 on an H100 with TEI:

Stagep50 latencyp95 latencyComment
Query rewrite (LLM 3B)60 ms150 msonly if enabled
HyDE (LLM 3B)90 ms220 msonly if enabled
BM25 / SPLADE top-508 ms20 msElasticsearch or local Tantivy
Dense embedding of the query12 ms35 msbge-m3 with batch=1
Dense top-50 (filterable HNSW)6 ms18 msQdrant with scalar quantization
RRF fusion → top-30<1 ms<1 msarithmetic
Cross-encoder rerank top-3060 ms180 msbge-reranker-v2-m3 batched
Top-5 selection + format<1 ms<1 ms
Total without rewrite/HyDE~90 ms~270 msrealistic range on an H100
Total with rewrite + HyDE~240 ms~640 ms+ ~150-370 ms

On an RTX 4090 (~3× slower than an H100 on a large cross-encoder, similar on everything else) the numbers shift: ~180 ms p50 without rewrite, ~400 ms p50 with rewrite + HyDE. It is perfectly serviceable for a synchronous assistant with a 2-3s SLO, but tight for one with a 500 ms SLO.

Metrics: nDCG, MRR, Recall@k

There are three classic metrics that any retrieval eval reports:

Recall@k: is the relevant chunk among the top-k? A binary metric, it ignores ordering. The one that matters for the broad layer.

$$ \mathrm{Recall@k} = \frac{|\{\text{relevant}\} \cap \{\text{top-k}\}|}{|\{\text{relevant}\}|} $$

Mean Reciprocal Rank (MRR@k): in which position does the first relevant chunk appear? It penalises placing it far down.

$$ \mathrm{MRR@k} = \frac{1}{|Q|} \sum_{q \in Q} \frac{1}{\mathrm{rank}_q} $$

where $\mathrm{rank}_q$ is the position of the first relevant item for query $q$ (or $\infty$ if it does not appear in the top-k).

nDCG@k: it takes into account multiple relevant chunks with degrees of relevance. The most used metric for evaluating rerankers:

$$ \mathrm{DCG@k} = \sum_{i=1}^{k} \frac{2^{\mathrm{rel}_i} - 1}{\log_2(i+1)} \quad \mathrm{nDCG@k} = \frac{\mathrm{DCG@k}}{\mathrm{IDCG@k}} $$

The operational heuristic: Recall@50 measures the quality of your broad layer (we want > 95%; otherwise layer 2 will never be able to recover what was lost), nDCG@5 measures the quality of your reranker (we want > 0.75 on the golden set to consider the system “good”) and MRR@5 measures the quality of your top-1 (it matters especially in systems with top_k=1 or where the LLM’s answer is based mainly on the first chunk).

Costs and throughput

Assuming a bge-reranker-v2-m3 cross-encoder (568M params) on an H100 SXM with TEI 1.7 and dynamic batching:

  • Throughput: ~280 queries/sec with top-30 each (= ~8,400 chunk-query pairs/sec).
  • VRAM used: ~3.5 GB for the model + ~6 GB of KV cache + activations at batch=32.
  • Energy cost: an H100 SXM draws ~700 W. At 8,400 pairs/sec, the energy cost of pure reranking is ~0.083 mJ per pair. In Spanish industrial electricity bill terms (~0.12 €/kWh, May 2026): ~0.000003 € per rerank of 30 candidates. The bill is negligible compared with the cost of the downstream LLM.

On an RTX 4090 with TEI: ~95 queries/sec at top-30 (~2,850 pairs/sec), VRAM ~3.5 GB + ~5 GB. Serviceable for an internal assistant with modest traffic (~5 QPS sustained, peaks at 20-30).

The canonical pattern — which piece goes where

The 2026 operational reference stack for an on-premise hybrid + rerank RAG under data sovereignty:

                       ┌─────────────────┐
  User query ────────▶│ Query rewriter  │  (optional, LLM 3B)
                       │ (Qwen2.5-7B-IT) │
                       └────────┬────────┘
                                ▼
                       ┌─────────────────┐
                       │     HyDE        │  (optional)
                       │ (same LLM)      │
                       └────────┬────────┘
              ┌─────────────────┼─────────────────┐
              ▼                 ▼                 ▼
       ┌──────────────┐  ┌──────────────┐  ┌──────────────┐
       │ BM25/SPLADE  │  │ Dense bi-enc │  │ (metadata    │
       │ Elasticsearch│  │ bge-m3 + TEI │  │  filters     │
       │ top-50       │  │ + Qdrant     │  │  per tenant) │
       │              │  │ top-50       │  │              │
       └──────┬───────┘  └──────┬───────┘  └──────┬───────┘
              └─────────┬───────┘                 │
                        ▼                         │
                ┌──────────────┐                  │
                │   RRF k=60   │◀─────────────────┘
                │   → top-30   │
                └──────┬───────┘
                       ▼
              ┌──────────────────┐
              │ Cross-encoder    │
              │ bge-reranker-v2  │
              │ TEI batch=32     │
              │ → final top-5    │
              └──────┬───────────┘
                     ▼
              ┌──────────────────┐
              │   LLM context    │
              │   ~2,000 tokens  │
              └──────────────────┘

Key services:

  • Sparse index: Elasticsearch 8.16, OpenSearch 2.18 or embedded Tantivy. Native BM25, SPLADE optional via plugin.
  • Vector index: Qdrant 1.13 (filterable HNSW, scalar quantization, multivectors). Covered in detail in the PostgreSQL + Qdrant in ingestion post.
  • Embedder serving: HuggingFace’s TEI (Text Embeddings Inference) for bge-m3, multilingual-e5 and compatible models. OpenAI-compatible endpoint.
  • Reranker serving: TEI too, for the rerankers of the bge-reranker, jina-reranker and mxbai-rerank families. Separate /rerank endpoint.
  • LLM for rewriter/HyDE: vLLM 0.7+ serving Qwen2.5-7B-Instruct or Phi-4-mini-Instruct. If the main LLM is the same, it shares resources; if not, a dedicated pod with GPTQ-INT4 quantization.
  • Orchestration: the RAG client (FastAPI, LangChain, LlamaIndex) composes the three lanes. Each parallel call (Sparse + Dense) is made concurrently; rewriter, HyDE and rerank are sequential.

Minimum Kubernetes manifests are already developed in the vLLM on Kubernetes post and the full orchestration piece fits the pattern of the multi-tenant H100 cluster.

On-premise hardware: what fits in an RTX 4090 vs a generic 4×H100 SXM configuration

The operational reality matters. Two typical profiles:

Modest configuration — 1× RTX 4090 (24 GB Ada Lovelace)

It comfortably serves:

  • bge-m3 (568M) for dense embedding of the query → ~1 GB VRAM
  • bge-reranker-v2-m3 (568M) cross-encoder → ~3.5 GB VRAM at batch=32
  • A light LLM for rewriter/HyDE (Qwen2.5-7B GPTQ-INT4) → ~5-6 GB VRAM
  • The main LLM (Llama-3.1-8B-Instruct FP8) → ~11-12 GB VRAM

Total VRAM used: ~21 GB. That leaves ~3 GB for the main LLM’s KV cache, enough for 4-8 concurrent users with ~4K token contexts. For an internal corporate assistant it is viable. If you need a larger context or more concurrency, you have to split things up: either the main LLM is served outside the 4090 (another GPU, another node), or you fall back on quantising the reranker (not advisable: you lose the nDCG points that justify the cost of adding it).

Generic configuration — 4×H100 SXM (320 GB total, NVLink)

It comfortably serves everything above multiplied by 30-50× in throughput and opens the door to:

  • A large main LLM (Llama 3.3 70B FP8 or Qwen2.5-72B FP8) on 2 GPUs with tensor parallelism
  • Large rerankers (bge-reranker-v2-gemma 2B) with quality ~2 points higher
  • Late interaction (ColBERT-v2) on one of the nodes for critical cases
  • Several light LLMs in parallel for rewriter, HyDE and the evals judge

The practical rule: the reranker is one of the most efficient things you can put on a GPU. The VRAM used is modest, the throughput is high, and the quality improvement per euro of extra hardware is among the best in the RAG stack.

The seven traps that kill retrieval

Trap 1 — Top-k to the LLM with no reranker. Dense retrieval returns a top-5 and those 5 chunks go straight to the LLM. Without a reranker, the order of the 5 is the cosine similarity order, which is not the real relevance order. The LLM answers based much more on the first chunks than the last ones, and the nuance is lost. Symptom: confident answers with citations that are not the most relevant in the corpus.

Trap 2 — Dense-only with no BM25. Short queries and queries with internal jargon work badly, the demos look pretty, real traffic complains. Symptom: on QA with real support tickets, the system fails specifically on queries of fewer than 5 words or with product codes.

Trap 3 — Reranker badly calibrated on top-k. A layer 1 top-k that is too small (top-10) cuts before the reranker can do its job. A top-k that is too large (top-200) blows up latency without improving nDCG. The canonical sweet spot is top-30 to top-50 from the hybrid → top-5 to top-10 from the reranker. Symptom: the reranker’s nDCG@5 stuck no matter which reranker model you try.

Trap 4 — Embedder and reranker in different languages. The corpus is Spanish + Catalan, the embedder is all-MiniLM (English only), the reranker is bge-reranker-v2-m3 (multilingual). The broad layer retrieves badly because of the embedder, and the reranker has little to rerank. Symptom: Recall@50 < 80%, unrecoverable by changing the reranker.

Trap 5 — RRF with k=1. Someone reads the paper, copies the formula and sets k=1 “because they do not understand what it does”. The result: the first positions of the worst ranking (whichever it is) dominate. The fusion stops averaging and becomes “winner-takes-all”. Symptom: the final ranking is almost identical to that of just one of the retrievers.

Trap 6 — A query rewriter that changes the meaning. The LLM rewriter “improves” "how do I cancel" into "termination procedure for the premium service for corporate customers". It retrieves chunks about corporate customers when the user was a private individual. Over-specification by the rewriter is worse than no rewriter. Symptom: queries from “normal” users come back worse than the old ones. Mitigation: the rewriter must keep the user’s intent and only add context, never assume.

Trap 7 — No retrieval telemetry. The system serves, the user complains, you do not know whether the fault was the corpus, the retrieval, the reranker or the LLM. Without emitting traces with retrieved_chunks_ids, retrieved_chunks_scores, rerank_scores, query_rewritten_to, selected_top_k, debugging is theatre. Symptom: each incident takes days to investigate. The tracing with OTel and MCP piece covers the canonical pattern.

All seven are operational. Just as with the corpus, retrieval does not break because the maths is wrong: it breaks because the discipline relaxes. And as in Eval, the metrics can go up while the real experience gets worse, because the golden set accommodates itself to the system instead of the system accommodating itself to the golden set.

What we have not covered (upcoming posts)

  • Embedding model selection and fine-tuning: how to choose among the embedders on the MTEB leaderboard without falling into goodharting, when and how to fine-tune an embedder on your own domain with MNR loss or triplet loss, which synthetic dataset is generated with an LLM for training (GPL, InPars, Promptagator), and the gotchas of massive re-embedding when you change model.
  • Semantic cache for RAG: how to cache semantically similar queries to serve answers without going through retrieval or the LLM, GPTCache, MeanCache, and the cache’s precision/coverage trade-off. It saves 30-70% of the cost on workloads with repeated queries.
  • Multi-vector and ColBERT-v2 at scale: how PLAID and CITADEL indexes are designed to serve corpora of 100M+ chunks with late interaction without burning the memory budget.
  • RAG-specific eval — RAGAS deep dive: faithfulness, answer relevance, context precision, context recall, noise sensitivity. How the RAG-specific golden set is built (with chunks labelled as relevant/irrelevant) and which metrics correlate with real user satisfaction.
  • Function calling and tool-augmented retrieval: when the LLM decides which retriever to invoke (SQL for structured data, vector for unstructured, web search for real time), the ReAct pattern, handling tool errors.
  • Agentic retrieval loops: when a single retrieval pass is not enough and the agent iterates (planning, sub-queries, summary-and-refine). The latency / quality trade-off and the anti-patterns (infinite loops, diverging sub-queries).

See also

References