Embeddings in 2026: the three families (dense, sparse, multi-vector), the model zoo and the decision that matters in production

This post opens the data sub-saga inside the six-stage LLMOps pipeline by going into the piece that holds up three-layer retrieval: the embedder. If the librarian of curation decided what enters the index and the faculty committee decided what reaches the model’s face, this post looks at the one in the middle: the cartographer who draws the map you search over.

TL;DR

The conversation about embeddings has been simplified in production to the point where “which embedder do you use” always gets the same answer: OpenAI text-embedding-3-large in the demo, bge-m3 in the “sovereign-ready” version. That simplification hides the fact that an embedder is three different models at once, dense single-vector, learned sparse (SPLADE) and multi-vector late-interaction (ColBERT), and that in 2026 the leading models do not compete within the same family: gte-Qwen2-7B-instruct and NV-Embed-v2 break MTEB in dense single-vector, SPLADE-v3 and the sparse head of bge-m3 dominate the learned lexical descriptor, Jina-ColBERT-v2 and ColNomic-7B are the strongest in multilingual multi-vector, and Snowflake Arctic Embed L 2.0 has slipped in as the small multilingual favourite with decent Matryoshka. This post takes apart the three families with their maths (InfoNCE with τ, MaxSim, SPLADE’s FLOPS regularization, Matryoshka Representation Learning), goes through the open source model zoo with the dimension, licence and niche of each one, poses the specific problem of multilingual Spanish, which cuts the list of viable models to fewer than six, gives the real storage cost per million chunks with int8 / binary / TurboQuant, describes how they are served on-premise with TEI, Infinity and vLLM --task embed, sets the minimum hardware at an RTX 4090 and the good hardware at a 4×H100 cluster, lists the seven operational traps that sink quality without warning (corpus drift, forgotten normalisation, badly chosen Matryoshka dimension, missing hard negatives, chat template slipped into the embedder, tokenizer drift, MTEB overfit) and closes with a license-clean stack for sovereign production.

The analogy: three librarians cataloguing the same book

A technical library receives a new book and, before putting it on the shelves, has to generate a searchable identifier for it. Three librarians with three different trades live in the library, and all three catalogue the same book at once:

Librarian A, the thematic one. He reads the whole book and gives it a single rich RFID tag. That tag is a vector of 1,024 numbers where each coordinate encodes a latent semantic axis (one the librarian never verbalises: he has learned it by reading a hundred million previous books). Two books about Kubernetes in production will end up with RFID tags very close together in the space even if one talks about Linkerd and the other about Cilium, because they share thematic axes. To search, you compare the question’s tag with the book’s and return the nearest ones by cosine. It is fast, it scales to millions, and it loses fine nuance. This is the dense single-vector embedder: bge-m3 in dense mode, gte-Qwen2-7B-instruct, Snowflake Arctic Embed L 2.0, multilingual-e5-large-instruct.

Librarian B, the lexical one. He summarises nothing. What he does is write on a card the weighted list of relevant terms from the book, expanded with synonyms from the field. The Kubernetes book carries on its card “kubernetes 4.2, linkerd 3.8, cilium 3.7, service-mesh 4.1, sidecar 3.2, mtls 2.9, ebpf 2.6, k8s 4.0…”, each term with a weight. The trick is that the model does the expansion: if your question says “service mesh” and the original book only said “Linkerd”, librarian B’s card did record “service-mesh 4.1” because he understood the relationship. To search, you intersect the question’s card with the book’s the old-fashioned way: inverted index, posting lists. It is decisive when the reader writes few, very specific words (product names, errors, jargon). This is the learned sparse embedder: SPLADE-v3 or the sparse head of bge-m3. It is the modern successor to BM25, not its rival; we will see why.

Librarian C, the copyist. He gives up on summarising. He takes every word on every page of the book and gives it a 128-dimension mini-RFID. He ends up with a 30,000-token book turned into 30,000 mini-RFIDs. When you search for something, the librarian compares every word of your question with every word of the book and keeps the maximum for each word of the question, adding them up. It captures nuance the other two lose by construction (proper nouns, numbers, specific phrasings), but its filing system is an order of magnitude larger in disk space. This is ColBERT-v2 / Jina-ColBERT-v2 / ColNomic-7B: late interaction, MaxSim.

And then there is the all-rounder librarian who does all three jobs in a single pass: bge-m3. A single 568 M parameter model that simultaneously returns the thematic RFID tag (1,024-d), the weighted list of terms (sparse) and the set of per-token mini-RFIDs (128-d). That is why it has settled in as the standard embedder for multilingual on-premise RAG: a single model.encode(chunk) produces the three outputs that feed hybrid retrieval without orchestrating three different models.

What an embedding really is

A text embedding is a function f : text → ℝᵈ trained so that two “semantically similar” texts produce nearby vectors in that space. The important part is not “vector”, any fixed-length hash is one, but what closeness means. Closeness is defined implicitly by the loss the model is trained with.

Almost every modern embedder is trained with InfoNCE (also called Multiple-Negatives Ranking Loss in sentence-transformers):

\mathcal{L}_{\text{InfoNCE}} = -\log \frac{\exp(\text{sim}(q, d^+)/\tau)}{\sum_{d \in \mathcal{B}} \exp(\text{sim}(q, d)/\tau)}

For each (query, doc⁺) pair the model has to assign the positive more similarity than all the other documents in the batch B, where the other documents act as free in-batch negatives. The temperature τ (typically 0.02–0.07, almost always 0.05) controls how sharp the distribution gets: lower τ means the model becomes more demanding about the positive but less stable. A large batch size |B| means many more negatives per gradient, so a better informed model. That is why embedders are trained with batch ≥ 1,024 on H100 clusters with AllGather across nodes to stack every negative in the cluster into a single effective batch.

On top of the in-batch negatives comes hard negatives mining: documents deliberately picked because they are “almost in the answer” (typically the next 10-100 neighbours from a previous BM25 / dense retrieval). Without hard negatives the model learns to discriminate the trivial and real quality on BEIR / MTEB drops 5-10 points.

This matters because the embedder family depends on what you feed into sim(·,·):

The loss is the same in all three cases, InfoNCE, but the geometry of the space changes, and with it quality, storage cost and search latency.

The three families in detail

Dense single-vector, the cartographer

The embedder reads the full chunk, passes it through an encoder transformer (XLM-RoBERTa, BERT, a Mistral decoder with a prompt) and aggregates the token representations into a single vector by:

• CLS pooling: uses the embedding of the [CLS] token. Standard in BERT-base.
• Mean pooling: plain average of the embeddings of all tokens. Standard in multilingual-e5-large-instruct, bge-m3 dense, Snowflake Arctic Embed L 2.0.
• Last-token pooling: for decoder-LLM based embedders (e5-mistral-7b-instruct, gte-Qwen2-7B-instruct, NV-Embed-v2) that take the final token as the aggregate. It works because the model is causal and the last token “has seen” the whole context.
• Latent-attention pooling: new in NV-Embed-v2. A learnable attention layer that weights tokens instead of averaging them. +2-3 MTEB points over mean pooling.

After pooling the vector is normalised to norm 1: v ← v / ‖v‖₂. With normalised vectors, cosine and dot product coincide:

\cos(q, d) = \frac{q \cdot d}{\|q\|\|d\|} = q \cdot d, \qquad \|q - d\|^2 = 2 - 2\,(q \cdot d)

That is why almost every vector DB indexes by inner product and leaves it to the user to normalise beforehand (Qdrant, Faiss IP, Milvus). If you forget to normalise the query vector but the corpus vectors are normalised, retrieval degrades silently: the query’s magnitude distorts the ranking. It is the number one bug in RAG in production.

A minimal numerical example, two normalised 4-d vectors:

q = [0.5, 0.5, 0.5, 0.5]          ‖q‖ = 1
d₁ = [0.6, 0.4, 0.5, 0.5]         ‖d₁‖ = 1.005, normalised [0.597, 0.398, 0.498, 0.498]
q · d₁ = 0.5·0.597 + 0.5·0.398 + 0.5·0.498 + 0.5·0.498 = 0.995

Closeness of nearly 1, as expected. If d₁ had not been normalised, q · d₁ = 1.0 and it would appear closer than any d that is perfectly aligned but with a norm < 1.005. Normalisation is not a detail: it is the contract of the space.

Learned sparse, the lexical descriptor

SPLADE-v3 (Naver, March 2024) has consolidated the modern version of the lexical librarian. Internally it is a small BERT (~110 M parameters, DistilBERT/BERT base) that produces, for each input token, a distribution over the whole vocabulary (30,522 dimensions in BERT WordPiece), and then does max-pool over the tokens:

w_j = \max_{i \in \text{seq}} \log\bigl(1 + \text{ReLU}(W_{ij})\bigr)

where Wᵢⱼ is the logit of input token i for vocabulary term j. The log(1+ReLU) saturates high logits (it stops a single word from dominating the vector) and the ReLU cuts the negatives. The result is a 30,522-dimension vector of which typically 50-200 entries stay non-zero.

The key part is the FLOPS regularization added to the loss during training:

\mathcal{L}_{\text{FLOPS}} = \lambda \cdot \sum_{j=1}^{V} \bar{w}_j^2, \qquad \bar{w}_j = \frac{1}{|B|}\sum_{i \in B} w_{ij}

It penalises the expected cost of the posting lists: if a vocabulary word appears on average in many documents, adding it to a new document costs double. The model learns to generate sparse vectors by construction.

What does this do to the text “Linkerd 3.8 brings mTLS by default”? The model does not only write the literal terms, it also writes, with a smaller but non-zero weight, “service-mesh”, “kubernetes”, “tls”, “sidecar”, “envoy”, “istio” (its competitor, also semantically related), “encryption”, “k8s”. That semantic expansion of the document is what sets SPLADE apart from BM25. BM25 only knows what was literally in the text; SPLADE knows what an expert would add as a descriptor.

In practice SPLADE-v3 beats BM25 by 3-6 MRR@10 points on MS MARCO and dominates BEIR zero-shot. The cost is ~2-4× the query latency of BM25 over the same inverted index, which static pruning can mitigate.

For the multilingual case, bge-m3 with its sparse head is the only maintainable option: SPLADE-v3 is trained in English and the multilingual ports are in an experimental state.

Multi-vector, the copyist

ColBERT-v2 (Stanford, NAACL 2022) introduced the late interaction paradigm. Instead of compressing the document to a single vector, it leaves it as a matrix (|d|, k) with a k-dimension vector per token. The similarity between query and document is computed token by token and aggregated with MaxSim:

s(q, d) = \sum_{i \in q} \max_{j \in d} \langle q_i, d_j \rangle

What is being computed: for each word of the query, find the document token that fits it best and add that similarity. The sum is over the query, not over the document. This lets a 30,000-token document compete fairly with a 200-token one, because the query always adds |q| terms.

Why does it give more quality than dense single-vector? Because compressing to one vector loses information about where each idea was. If the query is “which version brought mTLS by default in Linkerd”, the dense summary of the document only knows that the chunk is about “Linkerd and mTLS”; ColBERT’s copyist can match “which version” with “3.8” because it stores the embedding of the 3.8 token separately. On BEIR / out-of-domain, late interaction beats single-vector by between +2 and +6 nDCG@10 with the same backbone.

The price is storage. Per document, a 1024-dimension dense single-vector in fp16 takes 1024 × 2 = 2 KB. ColBERT-v2 with 128-dimension tokens for a 256-token chunk takes 256 × 128 × 2 = 65,536 B ≈ 64 KB: 32× more space. With ColBERT-v2’s nbits=2 residual compression it drops to ~16 KB (8×). Jina-ColBERT-v2 adds Matryoshka over the token dimensions (truncatable to 128 / 96 / 64), cutting another 50%.

For 1 million chunks:

ColBERT in on-premise production is reserved for corpora of up to a few million chunks, or applied only as a reranker over the top-100 from the first committee (dense + sparse), as described in the reranker post.

Matryoshka, the truncatable dimension

An operational lever that changed the conversation about embeddings between 2024 and 2026 is Matryoshka Representation Learning (Kusupati et al., NeurIPS 2022). The trick: during training, on top of the loss over the full D-dimension vector, the same loss is computed over prefixes of the vector:

\mathcal{L}_{\text{MRL}} = \sum_{k \in \{64, 128, 256, 512, 1024\}} \alpha_k \cdot \mathcal{L}_{\text{InfoNCE}}\bigl(\text{emb}[:k]\bigr)

The first k dimensions of the embedding are trained to be, on their own, a valid embedding. At inference time, if you want a cheaper embedding, you truncate the vector: the first quarter is already a usable embedding. Without Matryoshka, truncating wrecks the geometry: the first 256 dimensions of an embedding trained only at 1024-d encode nothing coherent.

Typical degradation in MTEB nDCG@10 when truncating an MRL embedder:

Native MRL models in 2026 (the ones that let you pick the dimension at runtime without retraining):

• jina-embeddings-v3 (1024 → 32, fine steps, CC-BY-NC-4.0)
• jina-embeddings-v4 (2048 → 128, multimodal text+image, CC-BY-NC-4.0)
• nomic-embed-text-v2-moe (768 → 256, Apache 2.0)
• Snowflake-arctic-embed-l-v2.0 (1024 → 256, Apache 2.0)
• mxbai-embed-large-v1 and mxbai-embed-2d-large-v1 (the latter also truncatable in layer depth)
• Stella_en_1.5B_v5 (multiple steps 512 / 768 / 1024 / 2048 / 4096 / 6144 / 8192, MIT, English only)
• text-embedding-3-large (OpenAI, 3072 → 256, API only)
• voyage-3 family (1024 → 256 / 512 / 1024 / 2048, API only)

Production recommendation: always use an MRL model even if you do not truncate at the start, because it simplifies the future quantisation decision. And above all, evaluate truncation on your real corpus: the average MTEB degradation of “-3 points” becomes “-12 points” in a niche domain.

The 2026 open source model zoo

What follows is a spec sheet per model. Verified as of 2026-06: HuggingFace cards + reference papers + MTEB / MMTEB leaderboard.

Dense single-vector

Learned sparse

Multi-vector (late interaction)

The leaderboard, with caution

MTEB / MMTEB (Massive Multilingual Text Embedding Benchmark, Enevoldsen et al., arxiv 2502.13595) is the standard thermometer. Top retrieval on MMTEB in mid-2026 is dominated by Qwen3-Embedding-8B (~70.6 multilingual avg) and Llama-Embed-Nemotron-8B. Below them, the 7B models (gte-Qwen2-7B, NV-Embed-v2) and the 568M ones (bge-m3, Snowflake-Arctic-L-2.0) compete task by task.

Trap: MTEB has started to saturate through dataset contamination. The higher the ranking, the more likely it is that the model saw subsets of the evaluation datasets during training. The rule in production: the leaderboard is for discarding bad models, not for picking the best one. The final decision is made on your own domain eval set, generated with the LLM-as-judge or evals recipe.

The specific problem of multilingual Spanish

For a Spanish customer serving corporate, legal or support documentation in Spanish (and often with Catalan / Portuguese / English mixed in), the embedder zoo narrows to fewer than six viable models. The operational exclusions:

1. English-only models: e5-mistral-7b-instruct, stella-en-1.5B-v5, Linq-Embed-Mistral, mxbai-embed-large-v1, NV-Embed-v2, SPLADE-v3. They cut Spanish performance below an acceptable level: translating the query into English before searching is one route, but it introduces latency, tokenisation drift and another model dependency.
2. Models with a non-commercial licence: jina-embeddings-v3, jina-embeddings-v4, NV-Embed-v2, Linq-Embed-Mistral. They work for a PoC, but commercial production needs an explicit agreement with the vendor. Unless you have the licence signed, they have to be excluded.
3. Models with a Gemma licence: bge-multilingual-gemma2. Allowed for internal use, awkward to redistribute weights to a customer.

The ones left, ordered by practical choice in sovereign production:

1. BAAI/bge-m3 — MIT, 568 M, 100+ languages (including Spanish and Catalan, explicitly trained), 8,192 tokens, tri-mode dense+sparse+colbert. A reasonable default. Fits in an RTX 4090. TEI and Infinity serve it natively.
2. Snowflake/snowflake-arctic-embed-l-v2.0 — Apache 2.0, same size, explicit Matryoshka, better MIRACL/CLEF than bge-m3 on some multilingual tasks, no sparse head. If the priority is pure MMTEB in Spanish.
3. intfloat/multilingual-e5-large-instruct — MIT, 560 M, veteran baseline. Its 512-token window is the big limitation: long documents have to be split first. If what you already have in production works, do not migrate out of fashion.
4. Alibaba-NLP/gte-Qwen2-7B-instruct — Apache 2.0, 32k context, high quality in Spanish (Qwen2 is well trained on Spanish). If the chunks are long (more than 4k tokens) and you have the GPU to serve it (it does not fit in a 4090; it does in an H100). It fits alongside an LLM on an H100 80GB with care.
5. nomic-ai/nomic-embed-text-v2-moe — Apache 2.0, 305 M active, MRL, ~100 languages. If latency and cost per token rule: the MoE gives it disproportionate throughput for its quality.
6. jinaai/jina-colbert-v2 — Apache 2.0, multilingual multi-vector, as a reranker or as the main retrieval on a small corpus (< 1 M chunks). The only license-clean multi-vector in Spanish.

The rule of thumb: bge-m3 as dense+sparse on the front line, jina-colbert-v2 as a third reranking layer when the use case warrants it, and Snowflake Arctic L 2.0 as the alternative if the corpus-specific eval prefers its geometry.

Serving embeddings on-premise

Three engines split the on-prem embedding serving landscape in 2026, with different profiles.

Text Embeddings Inference (TEI), the standard

huggingface/text-embeddings-inference is a server written in Rust with a Candle / ONNX backend, integrated FlashAttention and dynamic batching by tokens. It exposes an OpenAI-compatible /v1/embeddings API and supports the three bge-m3 modes simultaneously from version 1.5 onwards. For multilingual production it is the obvious default.

# values.yaml — TEI serving multilingual bge-m3 on an RTX 4090
image: ghcr.io/huggingface/text-embeddings-inference:1.5
args:
  - --model-id=BAAI/bge-m3
  - --pooling=cls
  - --max-batch-tokens=16384
  - --max-concurrent-requests=512
  - --dtype=float16
resources:
  limits:
    nvidia.com/gpu: 1

Indicative throughput with bge-m3, fp16, 512-token sequence, batch 32:

• RTX 4090 (24 GB): ~8–15 k tokens/s
• A100 80 GB: ~60 k tokens/s sustained
• H100 80 GB: ~40–80 k tokens/s, with fp8 about 50% more

(The ranges are approximate and depend on the real batch, the average sequence length and compilation with FA2/FA3.)

Infinity, the flexible one

michaelfeil/infinity (MIT) is a multi-model FastAPI server able to load bge-m3, Snowflake Arctic, Jina-v3, Nomic, ColPali, CLAP and rerankers simultaneously behind the same OpenAI-style API. Backend PyTorch + Optimum (ONNX/TensorRT) or CTranslate2. Useful when you need to serve several different embedders (one for text, another for code, another for images) behind a single endpoint, or when the model does not yet have TEI support.

vLLM --task embed, for the 7B embedders

When the embedder is really an LLM decoder turned into an embedder (e5-mistral-7b-instruct, gte-Qwen2-7B-instruct, NV-Embed-v2, Stella-1.5B), the natural place to serve it is vLLM, which already has the PagedAttention and continuous batching stack in production:

vllm serve Alibaba-NLP/gte-Qwen2-7B-instruct \
  --task embed \
  --dtype bfloat16 \
  --max-model-len 32768 \
  --trust-remote-code

vLLM detects the correct pooling (last-token on the Qwen / Mistral based ones) and exposes an OpenAI-compatible /v1/embeddings. For inference clusters already running vLLM with an LLM on another port, it is the natural way to serve the embedder without standing up another stack.

fastembed, the lightweight one

qdrant/fastembed loads bge-small, MiniLM, ColBERT and BM25/SPLADE sparse in ONNX-CPU. It is not competitive on throughput against TEI/Infinity with a GPU, but it is the right option when you have to serve embeddings on a NUC node without a GPU (see mixed NVIDIA + Intel environments) or when the embedder is part of the client (a preview in a UI, prior scoring at the edge).

Storage, quantisation and the corpus arithmetic

The embedding does not stay in the embedder’s memory: it lives in the vector DB index and is materialised every time you ingest a new chunk. The storage cost calculation is what decides the final dimension, not MTEB quality. For 1 million chunks with a dense embedder at 1024-d:

fp32        : 1,024 dims × 4 B × 1 M = 4,096 MB ≈ 4.0 GB
fp16/halfvec: 1,024 dims × 2 B × 1 M = 2,048 MB ≈ 2.0 GB
int8 (SQ)   : 1,024 dims × 1 B × 1 M = 1,024 MB ≈ 1.0 GB
binary      :   128  B × 1 M         ≈    128 MB

The quantisation options in order of real use (mid-2026):

1. halfvec (fp16): the default in pgvector 0.7+ and in any serious vector DB. Zero MTEB loss, 2× compression. Always turn it on.
2. Scalar Quantization int8 (SQ): each vector component is mapped to int8 with a global min/max. Typical recall@10 loss: 0–1%. 4× compression. The Qdrant default, supported in Milvus and Weaviate.
3. Binary quantisation: bit = sign(v_i - μ_i). 32× raw compression. Cold loss 5–15%. Mitigated with a Hadamard / TurboQuant pre-rotation (Qdrant 1.18, Dec 2025): it pre-multiplies by a random orthogonal matrix that spreads the energy across dimensions before binarising. After TurboQuant the loss drops to 1–3%. It also combines with rescoring over the original fp16 for the top-100 candidates.
4. Product Quantization (PQ): the FAISS classic. Up to 64× compression, 2–5% loss. More complex to operate (it needs a trained codebook); in 2026 it has given ground to binary + rescoring.

A corpus of 100 million chunks (a real figure for a large corporate RAG) with dense bge-m3:

The difference between 200 GB and 12.5 GB is the difference between needing a dedicated vector DB node with 8 NVMe in RAID and fitting in the RAM of a single node. For large corpora, quantisation is no longer an optimisation: it is the only way to operate.

The vector DB integration

The vector DBs of 2026 have become hybrid DBs that index all three types at once. The quick map:

For sovereign on-prem production in Spanish, the easiest combination to operate in 2026 is Qdrant + bge-m3: a single collection indexes the three modes of the same model, the hybrid query with RRF is done in one call, TurboQuant quantisation brings the corpus down to manageable levels, and the operator is a Go binary with simple backups to S3/MinIO. pgvector + bge-m3 is the other reasonable option when you already have Postgres with HA and do not want to add a second DB to the operational inventory; you lose native multi-vector, but you gain cross-cutting SQL over the chunks.

The HNSW parameters you absolutely have to touch:

• M: connections per node in the graph. 16–32 typical. Higher means more recall, more RAM. For small corpora (<1 M) M=16; for medium corpora (10 M) M=24; for large corpora M=32+IVF-PQ or M=32+binary.
• ef_construction: search width during construction. 100–400. Higher means a better graph, slower construction. Build with ef_construction=400 even if it is slow; you pay it once.
• ef_search: width during the query. 50–200. The main knob for the recall/latency trade-off at runtime. Start at 64 and measure.

Implications for on-premise inference

The embedder does not share hardware with the LLM as comfortably as it might seem. The numbers:

• bge-m3 (568 M) takes about 568 × 2 = 1,136 MB in fp16 for the weights, plus the batch KV cache, plus temporary activations. In practice it is served comfortably in 6–8 GB of VRAM even at a high batch. It fits alongside a 7B-Q4 LLM on an RTX 4090.
• gte-Qwen2-7B-instruct needs ~14 GB fp16 for weights alone. It does not fit next to a 7B LLM on a 4090; on an H100 80 GB it does, with care over simultaneous batching.
• jina-colbert-v2 (560 M) takes ~1.1 GB of weights, but the storage of the multi-vector index is the real cost: 8 GB per million chunks even with Matryoshka and compression.

On the RTX 4090 (24 GB)

A realistic minimum stack for a Spanish RAG with a corpus under 1 M chunks:

GPU 24 GB ┐
          ├─ TEI bge-m3 (dense + sparse + colbert) │ ~6 GB VRAM, ~12 k tok/s
          └─ vLLM Qwen2.5-7B-Instruct AWQ Q4       │ ~8 GB VRAM, ~80 tok/s
CPU/RAM  ┐
          ├─ Qdrant with bge-m3 dense + sparse + colbert │ ~3 GB RAM per M chunks
          └─ FastAPI gateway (LiteLLM)

It serves a few tens of RAG QPS with decent multilingual quality. It is the PoC configuration and the deployment for a small site.

On the 4×H100 80 GB cluster

For the production case with several million chunks and an SLO of p99 < 500 ms:

H100 #1 (80 GB) ── vLLM Qwen3-72B-Instruct AWQ + Qwen2.5-7B speculative ┐
H100 #2 (80 GB) ── vLLM gte-Qwen2-7B-instruct (embedding 32k ctx)        │ LLM + large embed
H100 #3 (80 GB) ── TEI bge-m3 multi-tenant + jina-colbert-v2 reranker    │ medium embed
H100 #4 (80 GB) ── Hold-out for canary / shadow                           │ see canary post
                                                                          ┘
Qdrant cluster (3 CPU nodes + NVMe) ── 100 M chunks indexed (binary + TurboQuant + rescoring)

This configuration separates the large LLM from the large embedder (they share the Qwen2 architecture but compete for VRAM if put on the same GPU) and leaves a whole H100 for variants in canary. bge-m3 fits with room to spare alongside the reranker on a single H100, serving tens of thousands of requests/min.

The seven operational traps of the embedder

1. Not normalising the query vector. Cosine and dot product coincide only when both vectors are unit vectors. If you forget v ← v / ‖v‖₂ in the client, the results are “almost right”, the top-1 is still correct on trivial queries, the top-10 no longer is, and nobody notices until RAG quality drops 8 points. Fix: bake normalisation into the embedder adapter, not into the client.

2. Chat template slipped into the embedder. Some LLM-based embedders (e5-mistral-7b-instruct, gte-Qwen2-7B) expect a specific instruction prompt before the text to embed ("Instruct: Retrieve relevant passages\nQuery: ..."). Forgetting it leaves performance ~5 MTEB points lower. Fix: read the usage_template in the model card and put it into the embedder wrapper.

3. Badly chosen Matryoshka dimension. The default in many Qdrant / pgvector clients is dim=768. If your embedder is natively MRL at 1024 → 768, fine. If it is 1024 without MRL, truncating to 768 wrecks the space (typical loss -8 MTEB points). Fix: use the model’s native dim and truncate only when storage demands it, and only on MRL models.

4. Missing hard negatives in fine-tuning. When the embedder is fine-tuned with your own data (which should be standard practice for corporate RAG), if the mini-batch only carries positives and in-batch negatives from the same domain, the model learns that anything outside the domain is negative, but inside the domain it does not discriminate. Fix: mine hard negatives with BM25 / dense from your own corpus before fine-tuning.

5. Corpus drift without reindexing. When you retrain or replace the embedder but only apply the new model to new chunks, you end up with the index mixing two incompatible geometries. Chunks from the old and the new model are not comparable by cosine. Fix: every change of embedder is a full reindexing of the corpus, planned as an operational retrain.

6. Tokenizer drift between client and model. The Python client that prepares the queries uses its own tokenizer (sometimes tiktoken by default) and truncates at 8,192 tokens. The embedder uses XLM-R with sentencepiece and truncates at 8,192 of its own tokenizer. Long queries are truncated differently; the corpus embeddings are consistent but the query ones are not. Fix: use the model’s tokenizer in the client or in the wrapper.

7. MTEB overfit as a selection guide. The MTEB leaderboard has become a contaminated metric: there is evidence that leading models saw subsets of the evaluation datasets during training. The model that is +0.5 points above the second is not necessarily better for your domain. Fix: your own domain eval set (100-300 labelled query-doc pairs) run with the evals recipe decides.

License-clean stack for sovereign production

Let us write down the recommended ending. For a Spanish organisation serving corporate RAG on-prem under ENS / ISO 42001 / EU AI Act, with a corpus of 1-50 M chunks in Spanish + English + Catalan:

The alternative stack, if explicit MMTEB in Spanish weighs more than the tri-modality of bge-m3: replace bge-m3 with Snowflake/snowflake-arctic-embed-l-v2.0 (Apache 2.0, MRL → 256) and explicitly add SPLADE-v3 or plain BM25 for the sparse layer. It loses the elegance of the single forward and gains 1-2 points on Spanish MIRACL.

Conclusion

The embedder is the easiest piece of a RAG to oversimplify and the one that decides real quality the most once everything else is in place. The three families (dense, sparse, multi-vector) are not three options to choose between but three trades that bge-m3 performs in a single pass and that hybrid retrieval consumes in parallel. The maths that matters is modest, InfoNCE with τ, MaxSim, FLOPS regularization, MRL, but the operational traps are many and quiet: forgotten normalisation, missing chat template, badly chosen Matryoshka dimension, tokenizer drift. For sovereign production in Spanish the list of viable models fits in fewer than ten, and the real decision comes down to “bge-m3 or Snowflake Arctic L 2.0”, with jina-colbert-v2 added as layer three when the fine-grained quality justifies the cost. The license-clean stack fits on an RTX 4090 for a PoC and scales to a 4×H100 cluster for real production.

See also

• Serving embeddings and rerankers with TEI in production — how to serve these models in production with batching and an OpenAI-compatible API.

• Taking RAG to the CPU: separating the data plane from the generation plane — why these encoders (~500M) are the ideal case for running on CPU and freeing up the GPU.

• The six-stage LLMOps pipeline — where the data / retrieval piece fits.

• RAG corpus curation — what enters the index before the embedder sees it.

• Reranker and hybrid retrieval — what the committee that consumes the embeddings does.

• Ontologies and knowledge graphs in LLMOps — the type layer that enriches the embedding with queryable metadata; chunks are not just vectors but typed instances against a TBox.

• Evals for LLMs — how you decide whether an embedder is really better than the current one.

• Capacity planning for LLM inference — sizing the GPU to serve embedder + LLM on the same cluster.

References

• Chen et al. M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. arXiv:2402.03216. https://arxiv.org/abs/2402.03216
• Sturua et al. Jina Embeddings v3: Multilingual Embeddings With Task LoRA. arXiv:2409.10173. https://arxiv.org/abs/2409.10173
• Günther et al. Jina Embeddings v4: Universal Embeddings for Multimodal Multilingual Retrieval. arXiv:2506.18902. https://arxiv.org/abs/2506.18902
• Nussbaum et al. Nomic Embed v2: Multilingual Mixture of Experts. arXiv:2502.07972. https://arxiv.org/abs/2502.07972
• Wang et al. Improving Text Embeddings with Large Language Models (E5-Mistral). arXiv:2401.00368.
• Yu et al. Arctic-Embed 2.0: Multilingual Retrieval Without Compromise. Snowflake, 2024-12. https://www.snowflake.com/blog/arctic-embed-2-multilingual/
• Lee et al. NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models. arXiv:2405.17428.
• Khattab y Zaharia. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. SIGIR 2020.
• Santhanam et al. ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction. arXiv:2112.01488.
• Jha et al. Jina-ColBERT-v2: A General-Purpose Multilingual Late Interaction Retriever. arXiv:2408.16672.
• Lassance et al. SPLADE-v3. arXiv:2403.06789.
• Kusupati et al. Matryoshka Representation Learning. NeurIPS 2022, arXiv:2205.13147.
• Enevoldsen et al. MMTEB: Massive Multilingual Text Embedding Benchmark. arXiv:2502.13595.
• Hugging Face Text Embeddings Inference. https://github.com/huggingface/text-embeddings-inference
• Michael Feil. Infinity. https://github.com/michaelfeil/infinity
• Qdrant. TurboQuant 1.18 release notes. https://qdrant.tech/articles/turboquant-quantization/
• pgvector. Release notes 0.7 / 0.8. https://github.com/pgvector/pgvector
• Hugging Face. Embedding Quantization. https://huggingface.co/blog/embedding-quantization