Mlops on lo0 — Tech Blog

Anatomy of an LLM request in production, May 2026: a tour of the six stages following a single request

Fri, 22 May 2026 16:00:00 +0200

TL;DR

Across several series the blog has laid out the pieces that hold up an LLM system in production: the Data stage (dataset versioning, ingestion and vector stores, RAG over Kafka), the Tune stage (continuous fine-tuning), the Eval stage (evals as the layer after tracing, guardrails and safety), the Deploy stage (KV cache, PagedAttention, disaggregated serving, multi-tenant GPU cluster, vLLM on Kubernetes, LLM operators on K8s), the Observe stage (tracing with AgentSight, MCP observability, eBPF + drift), the Retrain stage (closing the feedback → dataset → adapter loop), and the cross-cutting components (prompt versioning and data versioning). What is missing is joining them up: seeing a single request cross every piece in order, in one coherent story. That is what this post does. We take a specific request from a multi-tenant support chatbot, rewind it back to the data that trained the adapter serving it today, follow it forward through serving, watch it reach the feedback store when the user marks a thumbs-down, and leave it as the seed of the next quarterly retrain cycle. The route works as a mental map and as an integrator’s guide: the system does not hold up if a single one of the seven pieces (six stages + two cross-cutting) is broken or absent. The practical lesson of the tour is not a new one. It is that everything is connected, that local measurements lie when taken in isolation, and that the real cost of not operating one stage well is paid by another stage further down the line.

You are here: every stage at once

Unlike previous posts, where the mini-map marked a single active box, this one covers the whole pipeline. It is the only post on the blog that activates the six stages and the two cross-cutting components simultaneously, because we follow a real request that crosses all of them.

The analogy: forensic analysis of a request

When an air accident happens, the forensic analysis does not limit itself to the last seconds of the flight. The investigation team rewinds to the maintenance of the previous six months, the manufacturer’s protocols, the pilot’s record, the weather briefing, the controller’s decisions, the incident history on the same model. The conclusion is rarely “the wing broke”; it is “the wing broke because an inspection protocol written a certain way did not detect microcracks that the 2014 calculation model did not consider critical and that were critical beyond a certain fatigue cycle”.

When an LLM request in production fails or succeeds, there is also a long causal chain behind it. The answer the user sees is the last frame; what determined it starts months earlier and branches across six operational stages. If you only look at the last frame, you attribute the result to the model. If you look at the whole chain, you see that the model is one of twelve factors and rarely the most important.

This post does that forensic analysis, but the other way round: instead of starting from a failure and rewinding, we start from a specific request that works and break down what had to happen for it to work, and what will happen to it afterwards. It is a guided tour, not an incident diagnosis. But the mental discipline is the same: no stage is autonomous, and understanding the system means understanding the bridges between stages, not just the boxes.

The scenario: a multi-tenant support chatbot for regulated customers

For the tour we use a concrete realistic scenario, representative enough that the observations carry over to most serious deployments in May 2026. It is an LLM customer support SaaS product, sold to several corporate customers (multi-tenant) in regulated sectors (banking, insurance, health). The product:

Accepts natural-language questions through a chat embedded in the customer’s website.
Retrieves relevant fragments from the customer’s internal knowledge base (product documents, contractual terms, FAQs).
Answers citing the fragment used, in the user’s language, with a tone adapted to the brand.
If the question requires an action (cancel, change plan, open a case), it invokes a tool from the catalogue exposed via MCP, with user confirmation.
It is contractually obliged not to invent customer data, to respect the tenant’s style guide, and to meet ENS / NIS2 compliance on data in the EU.

The infrastructure is on-premise over Kubernetes (RKE2 with the Cilium CNI, in the pattern covered in Cilium BGP and CNI networking). The GPU plane is a generic 4×H100 SXM cluster (80 GB each, NVLink) for serving and production training, plus an RTX 4090 (24 GB) for development, offline evals and ad-hoc re-embeddings.

The request we are going to follow:

User (end customer, Spanish, premium plan with an insurer): “How do I cancel my premium subscription? I have been paying for three months and I have not used half of the cover.”

At first sight, a simple question. Let us see what is behind it.

Step 0 — The frame: the answer the user sees

At 11:47:32 on a Tuesday, the user sends the message. 4.2 seconds later, this appears in the UI:

To cancel your premium subscription, follow these steps: (1) Sign in to the customer portal with your ID number and password. (2) Go to My policies → Premium Life 2024. (3) Press Request cancellation. (4) Select the reason (in your case, “not using the product”) and confirm. You will receive a confirmation email within 24-48 h.

Before you cancel, let me remind you that your plan includes a free annual medical check-up and 24/7 travel assistance, cover that is lost on cancellation. If you like, I can help you review your use of the cover over the past three months to see whether switching to a more tailored plan suits you better than cancelling.

Source: [Cancellation policy, section 4.2 of the 2024 General Terms]

After reading it, the user marks a thumbs-down and writes in the form: “the answer is correct but the tone is too salesy; I only wanted to know how to cancel”.

That sequence, question, answer, thumbs-down with structured feedback, is the last visible frame. Let us go backwards to understand what had to happen for the answer to come out that way.

Rewinding backwards: what was already in place before the request

Before the user typed, the system already had a model loaded in serving, an active prompt labelled production, an up-to-date vector index, a versioned dataset from the last fine-tuning, and a golden eval set that validated the promotion. Each of those artefacts got there through a process. We go through four jumps backwards.

t = −90 days — The previous Retrain stage closes the earlier cycle

Three months ago, during a quarterly Retrain cycle, two things happened. The first: the support team reviewed the feedback accumulated over the previous six months and saw a pattern, the model answering premium users in an excessively formal tone, with users reporting that it “feels robotic”. The second: a one-off incident (a customer cancelling over an answer perceived as curt) triggered an incident-driven mini-cycle.

The process, covered in detail in the Retrain post, followed five sub-processes:

Feedback capture — explicit thumbs-down plus implicit feedback (abandonments, retries) accumulated in a feedback_signals table in Postgres, all of them with a trace_id that allows rewinding to the exact context.
Triage by root cause — the “curt tone” incident cluster was categorised as a prompt issue (it was not the model answering badly, it was the system prompt asking for an overly formal register). One sub-cluster was a model issue (in some cases the model dug its heels in even with a warmer prompt).
Dataset enrichment — the team manually annotated 280 cases where the model was too curt, labelled with the reference answer (“how it should have answered”). Double annotation on the critical 20%; cases with a quality score below 4 were left out.
Cadence decision — the incident was treated as incident-driven; the rest of the quarterly Retrain followed the calendar.
Promotion — the new customer_support_v7 adapter went through eval gates against customer_support_v6, a 5% canary for a week, and was promoted when the golden set metrics showed a stable improvement in the “tone / clarity” segment with no regressions elsewhere.

The result: the adapter active in production when the user sent the Step 0 request is customer_support_v7, trained on the enriched dataset enriched_retrain_2026_q1 version 3, with double lineage back to the original incident.

t = −60 days — The Data stage: the enriched dataset is versioned and enters circulation

Immediately after Retrain, the Data stage of the LLMOps pipeline does its work. Three critical operations, covered in detail in the data versioning post:

Immutable versioning of the enriched dataset with DVC, sha256 hash propagated to the registry. The identifier (enriched_retrain_2026_q1, v3, sha256:9af...) becomes the luggage tag that will travel through the coming stages.
Schema contract validated by CI: every row meets the JSON Schema of the entry the trainer expects (example_id, input.user_query, input.retrieved_context, expected_output, rubric, segment, difficulty). A validation fails in CI if any row breaks the contract.
Holdout segregation check: the normalised sha256 hash of each input is compared against every hash in the active golden eval set (customer_support_golden_v12). Zero overlaps = the dataset does not contaminate the eval. Had there been a single one, CI would have blocked the merge.

In parallel, the RAG corpus (product manuals, FAQs, general terms of the insurer tenant) is kept alive. The ingestion pipeline keeps capturing changes from the customer’s CMS: a new section of the cancellation policy was modified in February and reindexed in Qdrant. As the post about RAG over Kafka explains, the corpus is not retrained with every change: only the delta is re-embedded, and lakeFS keeps a branch of the embeddings bucket with the new version. The branch is merged into main when recall@10 over a representative set of queries stays above the threshold (0.78 in this system).

t = −45 days — The Tune stage: the customer_support_v7 adapter is trained

Three weeks after closing the dataset, training of the new LoRA adapter starts. As the continuous fine-tuning post details, the production pattern in 2026 avoids retraining the base model, which is expensive, slow and irreversible, and favours a LoRA adapter over a stable base model (in this system, Llama 3 70B-instruct quantised to INT8 for serving). The training:

Runs over 4 of the H100s (NVLink, tensor parallel) for ~6 hours.
Uses transformers + PEFT + bitsandbytes, monitored by MLflow.
Every step records the dataset_id, dataset_version and dataset_hash as an input artifact in MLflow.
The output, a customer_support_v7.safetensors file of ~280 MB with the LoRA weights, is uploaded to MinIO with its own hash, and MLflow records model_id, model_version, parent_dataset.

At this point the lineage chain is closed for this leg:

enriched_retrain_2026_q1, v3, sha256:9af...
│
▼
mlflow run train, run_id: 0xa721...
│
▼
customer_support_v7, sha256:5c1...

t = −38 days — The Eval stage: the v7 adapter goes through eval gates

The freshly trained adapter is not promoted. It goes through an eval suite covered in detail in the post about evals. The golden eval set, customer_support_golden_v12, 850 human-curated examples with an inter-annotator kappa of 0.81, is run against two models: the candidate v7 adapter and the v6 currently in production. The metrics:

Metric	v6 (prod)	v7 (cand.)	Threshold
Faithfulness to the RAG fragment	0.87	0.89	≥ 0.82
Toxicity (low is good)	0.012	0.011	≤ 0.02
“Warm but professional” tone (judge LLM)	0.71	0.84	≥ 0.78
Format compliance (structured markdown)	0.94	0.93	≥ 0.90
Helpful-but-not-pushy (judge LLM)	0.66	0.79	≥ 0.75
Latency p95 (ms)	2,840	2,910	≤ 3,500

To this is added the guardrails and safety suite covered in the guardrails post: jailbreak resistance, PII leakage detection, prompt injection over MCP tools. The v7 improves on safety in two metrics and ties on the rest.

The v7 enters a 5% traffic canary for 7 days, under close monitoring. At the end of the canary, the online metrics confirm what the offline ones anticipated: an improvement in tone and helpfulness, equivalent latency, no new failure modes. Promotion approved. The v7 moves to the production label.

t = −31 days — The Deploy stage: the v7 adapter enters serving

The customer_support_v7 adapter is promoted to the serving cluster. Three pieces covered in separate posts come into play.

vLLM as the inference engine. The engine lives on Kubernetes, deployed via a dedicated Operator, as the post about LLM operators and the post about vLLM on K8s explain. The operator is responsible for detecting the new adapter in the registry, hot-loading it without restarting the engine (a native vLLM capability with --enable-lora), and directing traffic based on the label.

Disaggregated serving. As the post about disaggregated serving details, the system separates prefill (compute-intensive, throughput-bound) and decode (memory-intensive, latency-bound) into different GPU pools. The user’s request, when it arrives, prefills in a specialised pod and decodes in another, communicating over NVLink plus a shared KV cache fabric.

Multi-tenant GPU cluster. The H100 cluster serves several tenants, not just the insurer from Step 0. As the post about the multi-tenant cluster explains, isolation is realised on four planes: the Kubernetes namespace, ACLs over adapters (only the tenant’s namespace loads its LoRAs), partitioning of the KV cache per tenant (one tenant cannot read another’s cached prefixes), and a tokens-per-minute quota enforced at the gateway.

Synchronised prompt registry. The product’s system_prompt lives in Langfuse with the production label. The active version is customer_support_system_prompt, version 12. The gateway reads the prompt from Langfuse on the request path (with a few seconds of cache so it does not hammer the registry). Detailed in the prompt versioning post.

The result at t = −31 days: the combination (adapter v7, prompt v12, golden v12) is active and served. The system is ready for the request that will arrive 31 days later.

Moving forward: the user’s request crosses the system

Back to Step 0: 11:47:32 on a Tuesday. The user presses Enter. We go in real time, in milliseconds.

t = 0 ms — Entry through the gateway

The user’s browser POSTs to chat.aseguradora-ejemplo.com/api/chat. The traffic crosses the edge load balancer and enters the SaaS product’s API gateway. The gateway:

Authenticates the user’s JWT (an end customer of the insurer tenant).
Extracts the tenant_id and checks that its tokens-per-minute quota is not exhausted.
Resolves which model_id, adapter_id and prompt_id correspond to this tenant and product. In this case: llama-3-70b-int8 + customer_support_v7 + prompt label production.
Builds a unique trace_id (W3C TraceContext, propagable to OTel) and starts a root span.

At 8 ms, the gateway passes the request to the prefill pool.

t = 8 ms — Pulling the versioned prompt

Before serving, the OpenAI-compatible client the engine uses internally pulls the active system prompt. As the post about prompt versioning details, the pattern is:

prompt = prompt_registry.pull(
 name="customer_support_system_prompt",
 label="production", # now pointing at v12
)
# A local 30 s cache reduces the round-trip to 0.1 % of requests

The OTel span of the prompt pull carries the attributes gen_ai.prompt.id = customer_support_system_prompt, gen_ai.prompt.version = 12, gen_ai.prompt.label = production. They are propagated to all children.

t = 12 ms — RAG retrieval

The system needs context from the tenant’s knowledge base. It runs:

query_embedding = encoder.encode(user_query)
chunks = qdrant.search(
 collection=f"tenant_{tenant_id}_kb_v3",
 vector=query_embedding,
 limit=4,
 score_threshold=0.72,
)
reranked = reranker.rerank(user_query, chunks, top_k=2)

At 38 ms, the reranker returns two fragments: one from the Cancellation policy, section 4.2 and another from Premium plan benefits, section 2.1. As the post about PostgreSQL + Qdrant details, the tenant’s corpus is kept isolated by collection and ACL: no tenant can read another’s chunks.

t = 40 ms — Building the final payload

The engine composes:

[system_prompt v12]
+ [retrieved context: 2 chunks]
+ [short session history: 1 previous turn]
+ [user query]

Total: ~1,850 tokens of context. The OTel span records gen_ai.request.input_tokens = 1850, gen_ai.request.model = llama-3-70b-int8, gen_ai.request.adapter = customer_support_v7.

t = 45 ms — Prefill

The payload enters the prefill pool. The GPU processes the 1,850 tokens in a single parallel pass, computing for each token its K and V vectors (attention key and value). Those vectors are materialised as the KV cache, covered in detail in the KV cache fundamentals post. The resulting cache takes up ~120 MB of VRAM in INT8.

Here a key optimisation appears: the v12 system prompt is cached in the prefill pool (prefix caching, covered in the post about PagedAttention). Since the system prompt is the same for this tenant, the first ~500 tokens of the context are not recomputed: they are read from the prefix cache. That reduces the effective prefill from 1,850 tokens to ~1,350 tokens, saving ~270 ms of compute.

At 580 ms (effective prefill), the TTFT (time to first token) is ready. The first token leaves for the decode pool.

t = 580 ms — Decode (streaming)

The decode pool receives the prefilled KV cache and starts generation token by token. As the post about disaggregated serving details, the prefill/decode separation is what lets a multi-tenant system keep TPS stable: the decode pool is sized to sustain thousands of sessions decoding in parallel at a low cost per token, while the prefill pool is sized for bursts of short TTFT.

Generation at ~80 tokens per second. The answer will be ~290 tokens. Total decode time: ~3.6 s. Streaming: the user starts seeing words from t = 580 ms.

While decode advances, the engine emits child spans on every iteration with gen_ai.response.tokens_generated, gen_ai.response.cache_hit_ratio, gen_ai.response.cumulative_latency. The post about AgentSight and the post about MCP observability with OTel cover the detailed instrumentation of this layer.

t = 4,200 ms — Complete answer, root span closed

Generation finishes. The engine closes the root span with gen_ai.response.completion_tokens = 290, gen_ai.response.finish_reason = stop, gen_ai.response.total_latency_ms = 4200. The user sees the final answer. The session is ready for a next turn or for the user to click thumbs-up/thumbs-down.

By this point, every active stage has taken part:

Data (pre-existing): the indexed RAG corpus, the dataset that trained the adapter, the golden set that validated it.
Tune (pre-existing): the v7 adapter trained 45 days ago.
Eval (pre-existing): the gates that approved the promotion.
Deploy (at this very moment): vLLM + disaggregated + KV cache + multi-tenant.
Observe (at this very moment): the OTel spans emitted to Langfuse + Tempo, the metrics to Prometheus.
Retrain (about to activate): the feedback the user will leave in 15 seconds.

In parallel: Observe is watching

While the request happens, several pieces of Observe run in parallel and leave a structured trail.

OTel tracing. Every span (gateway, prompt pull, retrieval, prefill, decode) travels to Langfuse and to an OTel collector that forwards them to a backend (Tempo / Jaeger). The unique trace_id links all the spans. As the post about tracing with AgentSight details, end-to-end propagation is the main enabler of post-incident debugging: without it, you cannot reconstruct what happened three weeks later.

Runtime metrics. The engine emits Prometheus metrics per interval: gpu_utilization, kv_cache_usage, tokens_per_second, queue_depth, prefill_latency_p95, decode_latency_p95. The metrics are not tied to a trace; they are aggregated by tenant and service.

Online LLM-as-judge. A configurable percentage of answers (2% in this system) is also run through a judge LLM in the background, which scores the answer against a simple rubric (correct / partial / incorrect + a tone score). The judge does not block the answer to the user; it feeds the dashboard.

Statistical drift. In parallel, a slower pipeline computes drift over the distribution of inputs and outputs. As the post about eBPF + drift explains, low-level monitoring (latency, error rate per endpoint) is complemented with statistical drift detection (KS test, embedding distance) that spots when “something is wrong” before a thumbs-down confirms it.

Safety and guardrails monitor. The guardrails post describes the layer that watches for jailbreak attempts, PII leakage and prompt injection via MCP tools. In this case, none of them fires.

All these pieces operate continuously, not per request. But this particular request left its trail in every one of them.

The feedback: the loop closes

Fifteen seconds after reading the answer, the user marks a thumbs-down and writes in the form: “the answer is correct but the tone is too salesy; I only wanted to know how to cancel”. That apparently trivial gesture triggers an important sequence.

Insertion into feedback_signals

As the post about Retrain details, the thumbs-down is persisted as a structured row in a Postgres table:

INSERT INTO feedback_signals (
 signal_id, trace_id, request_id, signal_type, signal_value,
 prompt_id, prompt_version, model, user_segment, occurred_at
) VALUES (
 gen_random_uuid(),
 '4f5...', -- the trace_id from Step 0
 'r-22a...', -- request_id
 'thumbs',
 '{"vote":"down","reason":"too pushy","text":"I only wanted to know how to cancel"}',
 'customer_support_system_prompt',
 12,
 'llama-3-70b-int8+customer_support_v7',
 'premium-es',
 '2026-05-19T11:47:51+02:00'
);

With this, the row is linked by trace_id to everything that happened: prompt v12, retrieved context, complete output, latency metrics, judge score (0.82 in this case, considered good by the judge, though the human disagrees).

Triage by root cause

The MLE team runs triage the next morning. Combining heuristic rules, LLM-as-classifier and human review:

The signal is not a model issue: the model answered correctly to the prompt it received.
It is not a retrieval issue: the retrieved chunks were the right ones.
It is not an infra issue: latency was normal.
It is a prompt issue: the v12 system prompt instructs the model to “offer alternatives before processing destructive actions”. That instruction generates the “salesy tone” in some contexts.

The incident accumulates with others from the month in the “salesy tone” cluster. When the cluster crosses a threshold (typically 30-50 incidents of the same type or a percentage of the total), it will enter an incident-driven mini-cycle or wait for the quarterly Retrain, depending on the size.

The next cycle picks it up

Three months later, in the next quarterly Retrain, this feedback is one of many that will motivate two changes:

A new prompt version v13 with an adjusted instruction: “offer alternatives only if the user does not express a clear intention to cancel”.
A possible reinforcement of the adapter with cases of a more direct tone for premium-es. If the cluster justifies it.

The v13 will go into its own eval gate. The golden set will grow with cases where the correct tone is “direct, not salesy”. The v8 of the adapter (if it arrives) will retrain over the enriched dataset enriched_retrain_2026_q2 that already contains this annotated case.

The loop closes. The Step 0 request has contributed to the version of the system that will serve another user three months later.

What goes in each trace: identity and traceability

If the reader looks at the seven identifiers that are everywhere in this route, they see the network of identities that makes all of the above possible. It is the identity infrastructure of an LLM system in production:

trace_id 4f5... (unique per request)
request_id r-22a... (idem)
prompt_id customer_support_system_prompt
prompt_version 12
prompt_label production
dataset_id enriched_retrain_2026_q1
dataset_version v3 (sha256:9af...)
model_id llama-3-70b-int8
adapter_id customer_support_v7 (sha256:5c1...)
deployment_id d-prod-7b
schema_version 3.2
tenant_id aseguradora-ejemplo
user_segment premium-es
golden_set_id customer_support_golden_v12

If a single piece of that set is missing or does not propagate, the chain breaks. The next incident investigated will land in “we cannot rewind to the origin because the system did not record it”. That is why the cross-cutting components, prompt versioning and data versioning, are not luxuries: they are the connection without which the other six stages operate blind.

Synthesis diagram: how the pieces fit

 ┌─────────────────────────────────────────┐
│ User (end customer, B2C) │
└─────────────────┬───────────────────────┘
│ chat msg + JWT
▼
┌─────────────────────────────────────────┐
│ Edge LB + WAF + Cilium CNI │
└─────────────────┬───────────────────────┘
│ HTTPS, internal mTLS
▼
┌─────────────────────────────────────────────────┐
│ API Gateway (auth, quota, model routing) │
│ - Resolves tenant → model + adapter + prompt │
│ - Starts trace_id (W3C) │
└──────┬─────────────────────┬────────────────────┘
│ │
(pull prompt) │ │ (pull config)
▼ ▼
┌────────────────────┐ ┌──────────────────────┐
│ Langfuse Prompt │ │ Model registry │
│ Registry (v12) │ │ (adapter v7) │
└─────────┬──────────┘ └──────────┬───────────┘
│ │
└──────────┬───────────────┘
│ payload ready
▼
┌──────────────────────────────────────────┐
│ vLLM engine (K8s Operator) │
│ ┌──────────────┐ ┌──────────────┐ │
│ │ Pool prefill │ → │ Pool decode │ │
│ │ (H100×N) │ │ (H100×M) │ │
│ └──────┬───────┘ └──────┬───────┘ │
│ │ KV cache fabric │ │
│ └──────────────────┘ │
│ - prefix caching of the system prompt │
│ - PagedAttention │
└──────┬───────────────────────────────────┘
│ tokens stream
▼
┌─────────────────────────────────────────┐
│ User sees the answer + thumbs/UX UI │
└─────────────────┬───────────────────────┘
│ feedback (15 s later)
▼
┌─────────────────────────────────────────┐
│ feedback_signals (Postgres) │
│ + Langfuse scores │
└─────────────────┬───────────────────────┘
│
┌────────────────────────┼────────────────────────┐
│ │ │
▼ ▼ ▼
triage quarterly Retrain cycle dataset_id
root cause or incident-driven enriched (DVC)
│
▼
Tune of v8
(next cycle)
In parallel throughout the request, OTel instrumentation:
spans → Tempo / Jaeger ; events → Langfuse ; metrics → Prometheus

The on-premise stack applied

Taking the above to a generic on-premise consultant-profile infrastructure (RTX 4090 + 4×H100 SXM cluster):

Layer	Typical resources
Network plane	Edge LB (HAProxy / nginx ingress) + Cilium CNI with BGP, covered in Cilium BGP
K8s compute plane	RKE2 with two manager nodes + a GPU node pool
Production GPU plane	4× H100 SXM (NVLink, 80 GB each), partitioned via MIG into prefill/decode pools
Development GPU plane	1× RTX 4090 (24 GB) for offline evals, drift-check embeddings, smoke tests
Storage plane	MinIO or Ceph object store; DVC remote + lakeFS backend
OLTP data plane	Postgres 18 with replication; pgvector 0.8 for small cases
Vector plane	Qdrant or Milvus for large RAG corpora
Stream plane	Kafka (Redpanda / pure Apache) + Schema Registry; CDC with Debezium or Flink CDC
Observability plane	OTel Collector + Tempo (traces) + Prometheus (metrics) + Loki (logs); Langfuse for the LLM-specific part
Runtime security plane	Tetragon, covered in the runtime security post

The real density is not the sum of the boxes: it is the operations that tie the boxes together. A cluster with every piece but no versioning discipline, no end-to-end trace_id propagation, no schema contracts and no cadenced retraining is a cluster that serves LLM once and then ages. The difference between a project and a platform is exactly that.

Ten bridges between stages where the system breaks

The route reveals something important: failures are rarely inside a stage; they are on the bridges between stages. Ten common bridges:

Data → Tune: the dataset does not propagate its (dataset_id, dataset_version) to the trainer. The same dataset trained twice produces two model_id values that cannot be told apart.
Tune → Eval: the trained model does not propagate its lineage to the eval run. The eval passes, but there is no record of which dataset it was trained on. Three months later, irreproducible.
Eval → Deploy: promotion happens without the serving system recording which version of the adapter it is serving at each moment. The day the model gives a dangerous answer, nobody knows which adapter answered.
Deploy → Observe: the engine does not emit gen_ai.request.adapter, gen_ai.prompt.version, gen_ai.dataset.version as span attributes. The traces exist but cannot be crossed with the lineage.
Observe → Retrain: the feedback is captured in a tool (Langfuse, Phoenix) but nobody reads it. The Retrain stage “exists”, but feedback piles up untriaged.
Retrain → Data: the enriched dataset goes into the next Tune without passing through versioning discipline, schema contract and holdout check. Silent contamination of the golden set.
Prompt versioning ↔ everything: the prompt_id, prompt_version does not propagate to the spans. The day the team discovers that a prompt change regressed the system, it cannot isolate which one or when.
Data versioning ↔ everything: the dataset_id, dataset_version does not show up in experiment tracking. “v8 is retrained” but nobody can prove it was over the enriched dataset and not the old one.
MCP ↔ tools: the system invokes tools (cancellation, policy changes) but does not record a gen_ai.tool.invocation_id linked to the trace. The actions end up dissociated from the answer that generated them.
Schema Registry ↔ data: datasets version content but not schema. A breaking change in expected_output breaks the eval silently; nobody notices anything until a human reviews the results.

The bridges are covered throughout the blog. Operations enforce them. The team’s culture keeps them.

How to walk through the blog

If you arrive at this post from outside and want a reading route:

The map: The six-stage LLMOps pipeline — the master map of everything else.
The context: MLOps specific to LLMs in 2026 — the landscape and why LLMOps is not classic MLOps.
Inference from the inside out: KV cache → PagedAttention deep dive → Disaggregated serving → Multi-tenant GPU cluster → vLLM on K8s → LLM operators on K8s.
Data: Data versioning with DVC and lakeFS → PostgreSQL + Qdrant ingestion → RAG over Kafka.
Tune: Continuous fine-tuning in production.
Eval: Evals: the layer after tracing → Guardrails and safety.
Observe: AgentSight LLM tracing → MCP observability with OTel → On-device eBPF + drift.
Retrain: Closing the feedback → dataset → adapter loop.
Cross-cutting: Prompt versioning with Langfuse and MLflow.
Supporting infrastructure (the base everything is built on): RKE2 with Cilium BGP, Hubble + eBPF observability, Tetragon runtime security.

What we have not covered (yet)

At the first level the main things are there. The following posts on the blog, when the topics justify it, could go deeper into:

Schema Registry for LLM data and prompts: the other half of the data contract.
A dedicated AI Gateway: LiteLLM, Portkey, Kong AI Gateway as a control plane.
OTel gen_ai semantic conventions: the emerging standard that ties the seven identifiers of the “identity” block into well-formed spans.
Federated learning over regulated customer data: how to train without centralising the corpus.
Capacity planning for shared multi-tenant clusters.
Disaster recovery of an LLM service: how to reproduce the state of the system 30 days back.
Cost accounting per tenant: tokens × weights × adapter × infrastructure → invoice.

References

W3C Trace Context — end-to-end propagation of traceparent and tracestate.
OpenTelemetry GenAI Semantic Conventions — gen_ai.* attributes for LLM spans.
Langfuse documentation — observability and prompt registry.
vLLM documentation — production inference engine with PagedAttention and LoRA hot-swap.
Kubernetes Operators — the declarative management pattern.
MLflow Tracking and Model Registry — lineage of runs and input artifacts.
DVC and lakeFS — dataset versioning, unified in Nov 2025.
OpenLineage — an open standard for lineage events between systems.
ENS / NIS2: compliance frameworks that apply to operators in the EU; recommended reading for the context in which the scenario operates.

Kubeflow in depth: which pieces are worth it in an on-premise LLM platform and which ones you already have

Mon, 31 Aug 2026 09:30:00 +0200

Second post in the platform and self-service run, after Backstage and before Knative. The question here is not what Kubeflow is, but which parts of Kubeflow still make sense once you have built half a platform on your own.

TL;DR

Kubeflow graduated in the CNCF on 17 August 2026, and that headline hides the important one: it is no longer a monolithic platform, it is an umbrella of independent subprojects, each with its own repository, its own version and its own life. That changes how you have to evaluate it entirely. The 2020 question, should I install Kubeflow?, has become the 2026 one, which piece of Kubeflow do I install, and at what cost?

For a platform that already has RKE2, vLLM, KServe, Flux and Volcano or Kueue, the honest answer is that Kubeflow as a whole duplicates almost everything you already have. KServe no longer lives under Kubeflow: it left, and since November 2025 it has been a CNCF project in its own right, so anyone serving models with KServe does not need Kubeflow to serve. The Trainer leans on Volcano and Kueue, it does not replace them. And the full manifest drags in Istio, Dex and cert-manager as hard dependencies that clash with what you already operate. The only thing that may be worth installing on its own is Pipelines, as an orchestrator for data and training flows, and perhaps the Trainer for its fine-tuning templates. The rest you either already have, or do not need.

The analogy: the hypermarket and the specialist shop

Kubeflow was born as a hypermarket: a single building with everything an ML team might need, from notebooks to the inference server, taking in pipelines and hyperparameter tuning along the way. It made sense in 2019, when assembling each piece separately was a project in itself and there were no standards.

By 2026 the hypermarket has been converted into a street of specialist shops. The butcher’s counter, KServe, went independent, opened its own premises and is doing better alone. The fishmonger’s, the Trainer, now depends on the central market next door, Volcano and Kueue, instead of having its own cold store. And the shared building still demands that you put in its wiring, its plumbing and its security system (Istio, cert-manager, Dex) even though you already have your own.

Anyone arriving today with a half-full fridge, which is the case for anyone with a platform already running, does not need the whole hypermarket. They need to know which shop to walk into for what they are missing, and to avoid buying twice what they already have at home.

What Kubeflow is in 2026

Kubeflow was created by Google in 2017 and graduated in the CNCF on 17 August 2026, reaching the top level alongside Kubernetes or Prometheus. The graduation figures are serious: more than 6,600 contributors from over a thousand organisations, and adopters such as Bloomberg, NVIDIA, Red Hat or Spotify.

The structural change is the one that matters for the decision. The full distribution, the Kubeflow Community Distribution, is numbered by year and month (the current one is 26.03, from March 2026, on a six-monthly cadence), but each component is by now a subproject with its own repository and its own cycle, usable independently. The distribution only packages them together. That means “installing Kubeflow” is no longer an atomic decision: it is a shopping list.

Component by component

Kubeflow Pipelines: the piece you may actually want

Pipelines orchestrates containerised ML workflows. They are written in Python with its SDK, which compiles to an intermediate YAML representation so they are portable, and underneath they run on Argo Workflows. It is the piece most likely to add value to an existing platform: a flow of “ingest, embeddings, fine-tuning, evaluation, deployment” is exactly what you need to operate RAG and adapters repeatably.

The small print has two parts. The first is that if you already use plain Argo Workflows, Pipelines adds a layer of SDK, interface and metadata on top of the same engine: it does not bring a new scheduler, it brings the typed component model and the runs interface. The second is that its metadata and lineage system, MLMD, is being removed from Pipelines itself, so it is not wise to build the platform’s traceability on that particular piece.

Kubeflow Trainer: it leans on your scheduler, it does not replace it

The old Training Operator, the one with PyTorchJob and TFJob, has been rewritten as Kubeflow Trainer, with a single TrainJob API that unifies all frameworks. Version 2.2 positions itself explicitly for distributed training and LLM fine-tuning, with support for PyTorch, DeepSpeed, HuggingFace and company.

A common expectation needs correcting here. The Trainer does not replace Volcano or Kueue: it leans on them. The TrainJob carries a podGroupPolicy field that automatically creates the Volcano PodGroup objects for gang scheduling, and it integrates with Kueue’s queues. If you already built that scheduler, as we saw in its post, the Trainer adds the TrainJob abstraction and some fine-tuning templates on top of what you already have, not a new scheduling layer. It may be worth it for the templates; not for the scheduler.

Katib: only half relevant

Katib does Kubernetes-native AutoML: hyperparameter optimisation with algorithms such as Bayesian optimisation, TPE or Hyperband, plus architecture search. In the LLM era it has repositioned itself, with official documentation for tuning fine-tuning hyperparameters and even for tuning RAG pipelines.

The practical reading is lukewarm. Architecture search is irrelevant for anyone doing LoRA and RAG. Hyperparameter optimisation can be useful for learning-rate or LoRA-rank sweeps, but it competes with doing it from the SDK of the QLoRA runbook itself or with Ray Tune, and it rarely justifies dragging in Katib for that alone.

KServe: the piece that is no longer in Kubeflow

This is the fact that settles half the evaluation. KServe left Kubeflow. It was born inside the project in 2019, was donated to the Linux Foundation in 2022, was renamed from KFServing to KServe and graduated from Kubeflow that same year, and since November 2025 it has been a CNCF incubating project in its own right. In the 2026 taxonomy it appears as an ecosystem project, external, not as a core component.

The consequence is direct: anyone already serving with KServe does not need Kubeflow to serve. And KServe is exactly where the LLM action is: its 0.15 release strengthened the vLLM backend and introduced an LLMInferenceService resource with disaggregated serving, prefix caching, per-variant autoscaling and OpenAI-compatible APIs. If your serving layer is vLLM on KServe, you already have that layer complete and Kubeflow adds nothing to it.

Model Registry, Notebooks and the rest

The Model Registry, now reorganised under the name Kubeflow Hub, is an index of models, versions and metadata. It stores pointers and states, not the bytes. Against MLflow, it lacks the experiment and metric tracking that is MLflow’s strength, to the point that the guides recommend combining them rather than substituting one for the other; and against an OCI registry, it does not package the binary, which still lives wherever you put it. It is still a young component, below version 1.0.

The Notebooks live in two versions, the stable 1 and a 2 redesigned on CRDs that is still in alpha. The Spark Operator remains in the core; Feast, by contrast, has left the core and is now an ecosystem project, a nuance worth being clear about before counting on it.

Multi-tenancy: powerful and tied to Istio

Isolation between teams is done with Profiles, a CRD that wraps a namespace and gives it RoleBindings, ServiceAccounts and Istio authorisation policies that validate an identity header derived from OIDC. It is a complete model, but it has an Achilles heel: it depends entirely on the Istio sidecar and on that header. If traffic bypasses the mesh or someone forges the header, isolation breaks, and all multi-tenant security ends up coupled to Istio, which clashes head-on if you already operate another mesh or no service mesh at all.

The cost of installing the whole thing

Here is the main reason not to install the full distribution on top of an existing platform. The 26.03 manifest drags in as hard dependencies: Istio, cert-manager, Dex for OIDC, OAuth2-Proxy, and Knative Serving and Eventing for KServe. It recommends 16 GB of RAM and 8 vCPU as a minimum, and the aggregate of all components comes to around 4.4 CPU cores and 12 GB of memory for the control plane alone.

The project itself acknowledges the problem. There is an open thread in its repository, bluntly titled “community feedback on Kubeflow’s complexity”, where it is admitted that Kubeflow practically forces a dedicated cluster because of the assumptions it makes about what is installed, that Istio and Dex should be swappable and not fixed dependencies, and that a lighter Helm that installs only what is needed is missing. For anyone already operating cert-manager, an ingress and GitOps with Flux, the full manifest does not coexist: it duplicates and tramples the base infrastructure.

Kubeflow against the alternatives

The field is wide and almost all the comparative literature is written by vendors with a product of their own, so it has to be read with tweezers. With that caution:

Plain Argo Workflows is the engine Pipelines uses underneath; if all you need is container graphs, Pipelines is overhead. MLflow is strong in experiment tracking and lightweight registry, installs in minutes, and does not orchestrate distributed training: it is complementary, and in fact we use it in the post on prompt versioning. Flyte is a typed orchestrator that many perceive as lighter to operate than full Kubeflow. And Ray on Kubernetes is the most serious competitor for LLM workloads, because with a single runtime it covers training, serving and tuning, overlapping at once with the Trainer, Katib and KServe.

Where Kubeflow wins is in breadth and in governance: a CNCF-graduated umbrella with adoption by large companies. Where it loses is in operational weight, in hard dependencies, and in the fact that its best pieces are already independent projects that do not require the rest.

The real fit with LLM workloads

The project has moved towards GenAI, and that has to be acknowledged: there is an SDK with LLM fine-tuning templates, the Trainer adds support for reinforcement post-training methods, and new agent-oriented pieces have appeared. It is not empty marketing, there is real work there.

But the question for a specific platform is not whether Kubeflow does LLM: it is which part I have not already duplicated. On a stack of RKE2, vLLM, KServe, Flux and Volcano or Kueue, the balance looks like this:

Component	Does it add anything new to your stack?
KServe	No: you already have it, and it is no longer Kubeflow’s
Trainer	Only the fine-tuning templates; you already have the scheduler
Katib	Marginal: LoRA HP sweeps, little more
Pipelines	Possibly yes, as a flow orchestrator, but it runs on Argo
Model Registry	Optional: competes with MLflow and with your OCI registry
Notebooks	Depends on what you use for notebooks today
Multi-tenancy	No, if you already isolate by namespace without tying everything to Istio

Operational traps and honest scepticism

Do not install the full manifest on a platform that already works. You are going to duplicate Istio, cert-manager and the rest, and fight a dependency war with no prize at the end. If something in Kubeflow interests you, install it as a standalone subproject.

KServe is not Kubeflow. It is the most common category error of 2026. If someone justifies building all of Kubeflow “to serve models”, you already know they have not checked that KServe left long ago and lives on its own.

The Trainer does not save you the scheduler. It needs Volcano or Kueue underneath. If you expected it to solve GPU gang scheduling by itself, it does not.

Multi-tenancy ties you to Istio. Before adopting Profiles, ask yourself whether you want security between teams to depend on one specific mesh. If you already isolate well by namespace and RBAC, you may not need that layer.

MLMD is on its way out. Do not build your pipelines’ traceability on the metadata system the project itself is retiring.

For an inference factory

The conclusion is comfortable to apply. Kubeflow in 2026 is not a yes-or-no decision, it is a menu you order from à la carte. On a platform that already serves with KServe, schedules with Volcano and deploys with Flux, the menu shrinks a great deal.

If you need to orchestrate data and training flows repeatably, and you do not want to build plain Argo Workflows with its ergonomics, Pipelines is the piece worth evaluating, installed standalone. If you do distributed fine-tuning frequently and want a uniform abstraction on top of your scheduler, the Trainer may save you some templates, knowing that it leans on the Volcano you already have. Everything else you either have, or a specialist piece covers better.

The expensive mistake would be installing the whole hypermarket to buy bread. Kubeflow stopped being that; treat it as what it is, a street of shops, and walk only into the one you are missing. The next piece in this run, Knative, is precisely one of those specialist shops, the one that decides whether switching off the GPU when nobody is using it pays off or costs you dear.

Sources

CNCF, CNCF announces Kubeflow’s graduation — https://www.cncf.io/announcements/2026/08/17/cncf-announces-kubeflows-graduation-solidifying-the-standard-for-cloud-native-ai-operations/
CNCF, Kubeflow unveils new cloud native innovations to supercharge AI — https://www.cncf.io/blog/2026/07/28/kubeflow-unveils-new-cloud-native-innovations-to-supercharge-ai/
CNCF, KServe becomes a CNCF incubating project — https://www.cncf.io/blog/2025/11/11/kserve-becomes-a-cncf-incubating-project/
Kubeflow Docs, Introduction / components — https://www.kubeflow.org/docs/started/introduction/
Kubeflow Docs, Pipelines overview — https://www.kubeflow.org/docs/components/pipelines/overview/
Kubeflow Trainer Docs, Volcano gang scheduling and Kueue — https://trainer.kubeflow.org/en/latest/operator-guides/job-scheduling/volcano.html
Kubeflow Docs, Migrating to Kubeflow Trainer v2 — https://www.kubeflow.org/docs/components/trainer/operator-guides/migration/
Kubeflow Docs, Katib overview — https://www.kubeflow.org/docs/components/katib/overview/
Kubeflow Docs, Hyperparameter optimization for LLM fine-tuning — https://www.kubeflow.org/docs/components/katib/user-guides/llm-hp-optimization/
Kubeflow Docs, Model Registry overview — https://www.kubeflow.org/docs/components/model-registry/overview/
Kubeflow Docs, Multi-tenancy design — https://www.kubeflow.org/docs/concepts/multi-tenancy/design/
GitHub, kubeflow/manifests (dependencias y recursos) — https://github.com/kubeflow/manifests
GitHub, kubeflow/manifests #2451: community feedback on complexity — https://github.com/kubeflow/manifests/issues/2451
InfoQ, Kubeflow expands AI capabilities as CNCF graduation nears — https://www.infoq.com/news/2026/08/kubeflow/

Reranker and hybrid retrieval: the committee that decides the 5 chunks the LLM will actually read

Mon, 25 May 2026 14:00:00 +0200

This post takes apart the retrieval layer inside the RAG piece of the six-stage LLMOps pipeline. It sits directly on top of the corpus curation post: if the librarian decided what enters the index, the committee in this post decides what leaves the index for the LLM’s face. It is the stage that moves the real quality of a RAG in production the most, and the one most teams solve with “dense top-k and off we go”.

TL;DR

The most widespread antipattern in real RAG is: curated corpus, dense embeddings with bge-m3, the user’s query straight through, top_k=5 of cosine similarity and off to the LLM. The system delivers answers that look good in the demo, mediocre with real traffic and dreadful with three-word queries, technical jargon, internal abbreviations or anything outside the embedder’s fine-tuning domain. The root cause is almost never the model: it is that a single pass of dense retrieval over the raw query is structurally insufficient for the cases a real user produces. In 2026 the field has consolidated the three-layer retrieval pattern: a broad layer that mixes dense + sparse (BM25 / SPLADE) via Reciprocal Rank Fusion to secure recall, a fine layer with a cross-encoder reranker (BAAI/bge-reranker-v2-m3, Cohere Rerank 3, mxbai-rerank-large) that reorders the 30-50 candidates by true relevance, and optionally a late interaction layer with ColBERT-v2 or an LLM reranker for critical cases. On top of it all, a query rewriting pattern that rewrites malformed queries and HyDE, which generates a hypothetical document so you embed the expected answer instead of the question. This post goes through them one by one: why each layer exists, what latency budget it consumes, the minimum maths of RRF and of the recall/precision trade-off, the dominant OSS stack in 2026, the on-premise hardware for serving it under data sovereignty, and the seven operational traps that kill the stage.

The analogy: the committee for a professorial chair

A university department has a chair to fill and receives 3,000 applications. Nobody is going to read 3,000 CVs in depth. The real process is organised in rounds:

Round 1, mass pre-screening. A system of automatic filters reads each CV in seconds. One route looks at the keywords of the post (publications in specific journals, languages, certifications); another route looks at the overall profile (research area, h-index, career). The two routes run in parallel. Each proposes its shortlist of 30-50 candidates. The two rankings are combined and what remains is a list of some 50-100 candidates for the next round to act on. The metric that matters here is recall: not losing the good candidate through a filter error.

Round 2, careful reading. A member of the panel reads each CV on the shortlist thoroughly and scores it against the real profile of the post. It is slow, 10-20 minutes per CV, but it discriminates far better than the automatic filter. It reorders the 100 candidates into a fine ranking and keeps the 5 finalists. The metric that matters here is precision: that the top-5 is genuinely the real top-5.

Round 3 (optional), personal interview. Long interviews are held for the 5 finalists. Face to face, questions adapted to each candidate’s profile, a check against the reality of the post. It is extremely expensive and very slow, but for critical chairs it justifies the cost.

The retrieval of a modern RAG works exactly like this. Change the vocabulary and the structure holds:

The equivalence is exact. The broad layer guarantees that the right chunk is on the shortlist (recall). The fine layer orders well inside the shortlist (precision). Each layer is justified by an operational asymmetry: the broad one is cheap and parallelises, the fine one is expensive and sequential. Solving everything with the expensive one, a single cross-encoder over the whole corpus, is operationally unsustainable. Solving everything with the cheap one, a single dense top-k, is functionally insufficient.

The problem: why dense on its own is not enough

The argument against dense-only is not theoretical. It is that the queries a real user produces have patterns the dense bi-encoder does not capture well by construction:

Short queries. “IRPF dates 2026” goes through the embedder and produces a vector close to any chunk that mentions dates and IRPF, including chunks from 2023, chunks of generic calendars, chunks of corporate IRPF when you were asking about personal income tax. BM25 disambiguates better because the weight of the infrequent terms (“2026”, “IRPF”) dominates the score.

Infrequent or internal technical terms. Product codes (FBR-X42-PRO), organisation abbreviations (PYM-23-bis), internal variable names. The embedder, trained on a public corpus, breaks them down into sub-tokens and pushes them into the “noisy” zone of the latent space. Two completely different codes can end up 0.02 cosine distance apart. BM25 treats them as exact tokens and scores them correctly.

Polysemy. “Java” as a language vs Java as an island vs Java as coffee. The dense embedding averages the meanings; the ranking becomes mediocre. Here dense can beat BM25 if the query carries enough context, but that “if” is rarely met with real user queries.

Synonyms and rephrasings. “How do I cancel my subscription” vs a chunk titled “Service termination procedure”. Here dense beats BM25: the semantic space brings them together even though they share no tokens. It is exactly the case where dense shines and BM25 fails.

The operational conclusion: no single retriever covers the four cases. The combination covers them all. And for the cases where both retrievers fail (very ambiguous queries, contradictory context between chunks), the third layer comes in: the reranker that reads the query and each chunk together with a cross-encoder, and produces a real relevance score far more discriminating than cosine similarity.

Layer 1 — Hybrid retrieval: dense + sparse

Why BM25 is still alive in 2026

BM25 is from 1994. It is not deep learning. It needs no GPU. And it is still competitive with dense retrievers of hundreds of millions of parameters on many retrieval benchmarks (BEIR, 2021-2024). Its formula is simple:

$$ \mathrm{BM25}(q, d) = \sum_{t \in q} \mathrm{IDF}(t) \cdot \frac{\mathrm{tf}(t,d) \cdot (k_1 + 1)}{\mathrm{tf}(t,d) + k_1 \cdot (1 - b + b \cdot \frac{|d|}{\overline{|d|}})} $$

Where $\mathrm{tf}(t,d)$ is the frequency of term $t$ in document $d$, $|d|$ the document length, $\overline{|d|}$ the mean length of the corpus, and IDF the inverse document frequency. $k_1$ and $b$ are hyperparameters (typically $k_1 = 1.2$, $b = 0.75$). In practice you use pyserini, Tantivy, Elasticsearch or OpenSearch and never write the formula.

SPLADE (Formal et al., 2021) is the neural generation of BM25: it learns sparse weights over the BERT vocabulary, expanding each token to its related terms. It keeps BM25’s interpretability (you can see which query terms match which document terms) and beats plain BM25 on BEIR. The price is that you need GPU inference over the query (and over each document at indexing time, once).

The rule of thumb in 2026:

Corpus < 10M chunks, queries with a broad vocabulary: plain BM25 via Elasticsearch / Tantivy. Trivial indexing, 1-5ms queries.
Corpus 10M-100M, specific domain: SPLADE-v3 indexed in Elasticsearch or Vespa. Slow indexing (you need a GPU once), 5-20ms queries.
Corpus > 100M, latency critical: BM25 with external query expansion (synonyms, query2doc) is the operational option, with SPLADE reserved for the critical segments.

Dense bi-encoder: the second lane

The bi-encoder produces one vector per chunk and another per query, and relevance is the cosine similarity (or dot product) between the two. The dominant multilingual model in 2026 is BAAI/bge-m3: 568M params, 1024 dimensions, support for up to 8,192 context tokens per chunk, multivector (dense + sparse + ColBERT-style multi-vec in the same embedder), trained on 100+ languages. Alternatives:

Embedder	Params	Dim	Multilingual	Comment
`BAAI/bge-m3`	568M	1024	yes	the default all-rounder in 2026
`intfloat/multilingual-e5-large-instruct`	560M	1024	yes	direct competitor, instruct-style queries
`nomic-ai/nomic-embed-text-v1.5`	137M	768	English	fast, variable Matryoshka dimension
`jinaai/jina-embeddings-v3`	570M	1024	yes	with task-specific LoRAs per domain
`mixedbread-ai/mxbai-embed-large-v1`	335M	1024	English	top English on MTEB
`Snowflake/arctic-embed-l-v2.0`	568M	1024	yes	enterprise-oriented, Apache licence

The practical selection criterion: multilingual no matter what if your corpus or queries need it (bge-m3, multilingual-e5, jina-v3, arctic-l-v2), Matryoshka if you want to save vector store memory by reducing dimensions without re-embedding (nomic), and the MTEB leaderboard ranking only as a tie-breaker, never as the sole criterion, since MTEB benchmarks are contaminable and a 1-2 point difference rarely translates into a real improvement in your domain.

The fusion: Reciprocal Rank Fusion (RRF)

You have two rankings, one from BM25 and one from dense. How do you combine them? The naive option, adding the scores, fails because the scores are not comparable (BM25 produces numbers of 0-30, cosine 0-1, L2 distances 0-2…). The option that has consolidated is Reciprocal Rank Fusion (Cormack, Clarke, Buettcher 2009), which ignores absolute scores and uses only the rankings:

$$ \mathrm{RRF}(d) = \sum_{r \in R} \frac{1}{k + \mathrm{rank}_r(d)} $$

Where $R$ is the set of retrievers and $\mathrm{rank}_r(d)$ is the position of document $d$ in the ranking of retriever $r$. The constant $k$ is typically $60$ and smooths the weight of the first positions.

A numerical example. You have BM25 with top-5: [d_a, d_b, d_c, d_d, d_e] and dense with top-5: [d_b, d_f, d_a, d_g, d_h]. You compute the RRF of each candidate:

Doc	BM25 rank	dense rank	$\frac{1}{60+r_{\text{BM25}}}$	$\frac{1}{60+r_{\text{dense}}}$	total RRF
`d_b`	2	1	0.01613	0.01639	0.03252
`d_a`	1	3	0.01639	0.01587	0.03226
`d_c`	3	—	0.01587	0	0.01587
`d_f`	—	2	0	0.01613	0.01613
`d_d`	4	—	0.01562	0	0.01562

d_b wins because it appears well ranked in both. d_a follows it. Those that appear in only one list stay below. With nothing tuned, RRF tends to push the consensus candidates to the top, which is exactly what you want as a coarse filter before the reranker.

Variants: weighted RRF assigns a multiplier per retriever ($w_r \cdot \frac{1}{k + \mathrm{rank}_r(d)}$) when you have reasons to trust one more; learning-to-rank trains a model (LambdaMART or a LightGBM ranker) over features from the two retrievers, a more powerful option but one that demands relevance labels most teams do not have.

In Qdrant, Vespa, Weaviate and Elastic 8.11+ the RRF fusion is built in as a native mode: you ask for hybrid search with a lexical query and a vector and they return the combined top-k without you having to compute the fusion.

Layer 2 — The reranker: bi-encoder vs cross-encoder

The bi-encoder looks at query and document separately and compares the resulting vectors. It is extremely fast (a dot product) and lets you index the document embeddings once and for all. The price is that the model never sees query and document together, and the compressed representation loses nuance.

The cross-encoder looks at query and document concatenated: [CLS] query [SEP] document [SEP] goes through the model and the output is a relevance score. It discriminates far better, since shades of ordering and of local relationship between terms are captured, but the cost explodes: each (query, doc) pair requires a full inference of the model. You cannot precompute anything. If you want to rerank 50 candidates, that is 50 inferences.

Aspect	Bi-encoder	Cross-encoder
Inference per chunk at indexing	yes, once	not precomputable
Inference at query time	1 (query) + dot product per chunk	1 per (query, chunk) pair
Typical latency for top-50	5-20 ms (CPU possible)	80-300 ms (GPU recommended)
Discrimination	medium	high
Typical use	recall (layer 1)	precision (layer 2)

This asymmetry is why the canonical pattern is a two-step funnel: the bi-encoder does cheap broad recall and the cross-encoder does expensive fine precision, over 30-50 candidates instead of millions.

2026 OSS cross-encoder reranker stack

Reranker	Params	Multilingual	top-50 latency (A100)	Comment
`BAAI/bge-reranker-v2-m3`	568M	yes (100+ languages)	~90 ms	the de facto default in 2026
`BAAI/bge-reranker-v2-gemma`	2B	yes	~250 ms	more quality, more cost
`mixedbread-ai/mxbai-rerank-large-v2`	1.5B	English	~180 ms	top English on BEIR rerank
`jinaai/jina-reranker-v2-base-multilingual`	278M	yes	~50 ms	fast and enough for many cases
`Alibaba-NLP/gte-multilingual-reranker-base`	306M	yes	~55 ms	direct competitor to jina
`Cohere Rerank 3`	(closed)	yes (100+ languages)	~120 ms (API)	commercial, not on-prem
`Voyage rerank-2`	(closed)	yes	~100 ms (API)	commercial, not on-prem

For deployments under data sovereignty the obvious choice is bge-reranker-v2-m3 as a starting point, with jina-reranker-v2-base as the lightweight alternative. If the domain is exclusively English and the hardware budget allows it, mxbai-rerank-large-v2 takes a point or two more on BEIR. Cohere and Voyage are the option when you can reach out to an external API, ruled out in ENS / NIS2 scenarios with sensitive data.

Late interaction: ColBERT-v2 as a compromise

There is a third option gaining traction: late interaction, embodied in ColBERT-v2 (Santhanam et al., 2022). The idea is that the document is embedded not as a single vector but as a matrix of embeddings, one per token. And at query time the query is also embedded as a matrix, and the similarity is the sum of the maxima per column (MaxSim):

$$ S(q, d) = \sum_{i \in q} \max_{j \in d} \langle E_q[i], E_d[j] \rangle $$

The result is retrieval with quality close to a cross-encoder but at a much lower cost (there is no need to run the whole model over each pair). The price is space: each chunk is ~150-200 vectors instead of 1, and that multiplies the index size by ~100. For small to medium corpora (<10M chunks) it is manageable; for huge corpora PLAID compression and ColBERT-v2 quantization make it viable up to 100M.

In 2026 the reference implementations are RAGatouille (Python wrapper), JaColBERT (Japanese), Vespa.ai (a search engine with native ColBERT support as a first-class citizen), and the native integration in bge-m3 that produces the ColBERT-style multivectors as part of the same dense model.

My operational recommendation: start with a cross-encoder reranker over the top-30 of the hybrid retrieval. Only if the precision numbers fall short, try late interaction as an alternative to the cross-encoder (not as an additional layer). Late interaction as a third layer rarely pays back the operational cost for most cases.

Query rewriting and HyDE: correcting the query before retrieval

The user’s query is the worst-controlled input in the whole system. The most frequent problems:

Queries that are too short: “GDPR fines” → retrieves any chunk about GDPR or about fines.

Queries with typos or jargon: “i wnat to know how to cancel the premium subscripton” → the embedder is noisy because of the typos.

Multi-intent queries: “compare the premium and enterprise plans and tell me which has the better SLA” → dense will try to embed the three questions into one vector.

Queries with references resolved in context: “and for next month?” → without the previous context, dense does not know what it is about.

Canonical 2026 patterns to mitigate:

Query rewriting with a light LLM. Before retrieval, a small model (Qwen2.5-7B, Llama-3.2-3B, Phi-4-mini) rewrites the user’s query into a “canonical” version: it expands abbreviations, fixes typos, breaks multi-intent into sub-queries, resolves references using the conversation history. Cost: one cheap inference (~50-100 ms). Benefit: the four pathologies above are noticeably reduced.

Multi-query: the LLM generates 3-5 rephrasings of the query and you retrieve with each one; then you fuse with RRF. It increases recall at the cost of time and LLM tokens.

Step-back prompting (Zheng et al., 2023): the LLM generates a more general question than the original query (“what is a premium subscription in this system?”), you retrieve with both and combine. It helps with very specific queries that need context.

HyDE — Hypothetical Document Embeddings (Gao et al., 2022): instead of embedding the question, you ask a light LLM to generate a hypothetical answer and you embed that. The reasoning is that the semantic space of answers is closer to that of the real chunks than the space of questions. It works especially well in domains with very specific vocabulary. Cost: one inference of the light LLM (~100-200 ms). Benefit: in many cases +5-15 points of Recall@10.

The operational trade-off is the added latency. Each technique adds 50-300 ms to the pipeline. In synchronous interactions with a 2s SLO there is still headroom; in agentic interactions with tool-use loops, every ms counts. The practical choice in 2026: rewriting with a light LLM as the default, HyDE for specific cases where it has been measured to help, multi-query and step-back only when the case justifies it with numbers.

Minimum maths: latency budget and metrics

Latency budget of a canonical pipeline

A typical hybrid + rerank pipeline in 2026 production over a corpus of 5M chunks with bge-m3 + bge-reranker-v2-m3 on an H100 with TEI:

Stage	p50 latency	p95 latency	Comment
Query rewrite (LLM 3B)	60 ms	150 ms	only if enabled
HyDE (LLM 3B)	90 ms	220 ms	only if enabled
BM25 / SPLADE top-50	8 ms	20 ms	Elasticsearch or local Tantivy
Dense embedding of the query	12 ms	35 ms	bge-m3 with batch=1
Dense top-50 (filterable HNSW)	6 ms	18 ms	Qdrant with scalar quantization
RRF fusion → top-30	<1 ms	<1 ms	arithmetic
Cross-encoder rerank top-30	60 ms	180 ms	bge-reranker-v2-m3 batched
Top-5 selection + format	<1 ms	<1 ms
Total without rewrite/HyDE	~90 ms	~270 ms	realistic range on an H100
Total with rewrite + HyDE	~240 ms	~640 ms	+ ~150-370 ms

On an RTX 4090 (~3× slower than an H100 on a large cross-encoder, similar on everything else) the numbers shift: ~180 ms p50 without rewrite, ~400 ms p50 with rewrite + HyDE. It is perfectly serviceable for a synchronous assistant with a 2-3s SLO, but tight for one with a 500 ms SLO.

Metrics: nDCG, MRR, Recall@k

There are three classic metrics that any retrieval eval reports:

Recall@k: is the relevant chunk among the top-k? A binary metric, it ignores ordering. The one that matters for the broad layer.

$$ \mathrm{Recall@k} = \frac{|\{\text{relevant}\} \cap \{\text{top-k}\}|}{|\{\text{relevant}\}|} $$

Mean Reciprocal Rank (MRR@k): in which position does the first relevant chunk appear? It penalises placing it far down.

$$ \mathrm{MRR@k} = \frac{1}{|Q|} \sum_{q \in Q} \frac{1}{\mathrm{rank}_q} $$

where $\mathrm{rank}_q$ is the position of the first relevant item for query $q$ (or $\infty$ if it does not appear in the top-k).

nDCG@k: it takes into account multiple relevant chunks with degrees of relevance. The most used metric for evaluating rerankers:

$$ \mathrm{DCG@k} = \sum_{i=1}^{k} \frac{2^{\mathrm{rel}_i} - 1}{\log_2(i+1)} \quad \mathrm{nDCG@k} = \frac{\mathrm{DCG@k}}{\mathrm{IDCG@k}} $$

The operational heuristic: Recall@50 measures the quality of your broad layer (we want > 95%; otherwise layer 2 will never be able to recover what was lost), nDCG@5 measures the quality of your reranker (we want > 0.75 on the golden set to consider the system “good”) and MRR@5 measures the quality of your top-1 (it matters especially in systems with top_k=1 or where the LLM’s answer is based mainly on the first chunk).

Costs and throughput

Assuming a bge-reranker-v2-m3 cross-encoder (568M params) on an H100 SXM with TEI 1.7 and dynamic batching:

Throughput: ~280 queries/sec with top-30 each (= ~8,400 chunk-query pairs/sec).
VRAM used: ~3.5 GB for the model + ~6 GB of KV cache + activations at batch=32.
Energy cost: an H100 SXM draws ~700 W. At 8,400 pairs/sec, the energy cost of pure reranking is ~0.083 mJ per pair. In Spanish industrial electricity bill terms (~0.12 €/kWh, May 2026): ~0.000003 € per rerank of 30 candidates. The bill is negligible compared with the cost of the downstream LLM.

On an RTX 4090 with TEI: ~95 queries/sec at top-30 (~2,850 pairs/sec), VRAM ~3.5 GB + ~5 GB. Serviceable for an internal assistant with modest traffic (~5 QPS sustained, peaks at 20-30).

The canonical pattern — which piece goes where

The 2026 operational reference stack for an on-premise hybrid + rerank RAG under data sovereignty:

 ┌─────────────────┐
User query ────────▶│ Query rewriter │ (optional, LLM 3B)
│ (Qwen2.5-7B-IT) │
└────────┬────────┘
▼
┌─────────────────┐
│ HyDE │ (optional)
│ (same LLM) │
└────────┬────────┘
┌─────────────────┼─────────────────┐
▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ BM25/SPLADE │ │ Dense bi-enc │ │ (metadata │
│ Elasticsearch│ │ bge-m3 + TEI │ │ filters │
│ top-50 │ │ + Qdrant │ │ per tenant) │
│ │ │ top-50 │ │ │
└──────┬───────┘ └──────┬───────┘ └──────┬───────┘
└─────────┬───────┘ │
▼ │
┌──────────────┐ │
│ RRF k=60 │◀─────────────────┘
│ → top-30 │
└──────┬───────┘
▼
┌──────────────────┐
│ Cross-encoder │
│ bge-reranker-v2 │
│ TEI batch=32 │
│ → final top-5 │
└──────┬───────────┘
▼
┌──────────────────┐
│ LLM context │
│ ~2,000 tokens │
└──────────────────┘

Key services:

Sparse index: Elasticsearch 8.16, OpenSearch 2.18 or embedded Tantivy. Native BM25, SPLADE optional via plugin.
Vector index: Qdrant 1.13 (filterable HNSW, scalar quantization, multivectors). Covered in detail in the PostgreSQL + Qdrant in ingestion post.
Embedder serving: HuggingFace’s TEI (Text Embeddings Inference) for bge-m3, multilingual-e5 and compatible models. OpenAI-compatible endpoint.
Reranker serving: TEI too, for the rerankers of the bge-reranker, jina-reranker and mxbai-rerank families. Separate /rerank endpoint.
LLM for rewriter/HyDE: vLLM 0.7+ serving Qwen2.5-7B-Instruct or Phi-4-mini-Instruct. If the main LLM is the same, it shares resources; if not, a dedicated pod with GPTQ-INT4 quantization.
Orchestration: the RAG client (FastAPI, LangChain, LlamaIndex) composes the three lanes. Each parallel call (Sparse + Dense) is made concurrently; rewriter, HyDE and rerank are sequential.

Minimum Kubernetes manifests are already developed in the vLLM on Kubernetes post and the full orchestration piece fits the pattern of the multi-tenant H100 cluster.

On-premise hardware: what fits in an RTX 4090 vs a generic 4×H100 SXM configuration

The operational reality matters. Two typical profiles:

Modest configuration — 1× RTX 4090 (24 GB Ada Lovelace)

It comfortably serves:

bge-m3 (568M) for dense embedding of the query → ~1 GB VRAM
bge-reranker-v2-m3 (568M) cross-encoder → ~3.5 GB VRAM at batch=32
A light LLM for rewriter/HyDE (Qwen2.5-7B GPTQ-INT4) → ~5-6 GB VRAM
The main LLM (Llama-3.1-8B-Instruct FP8) → ~11-12 GB VRAM

Total VRAM used: ~21 GB. That leaves ~3 GB for the main LLM’s KV cache, enough for 4-8 concurrent users with ~4K token contexts. For an internal corporate assistant it is viable. If you need a larger context or more concurrency, you have to split things up: either the main LLM is served outside the 4090 (another GPU, another node), or you fall back on quantising the reranker (not advisable: you lose the nDCG points that justify the cost of adding it).

Generic configuration — 4×H100 SXM (320 GB total, NVLink)

It comfortably serves everything above multiplied by 30-50× in throughput and opens the door to:

A large main LLM (Llama 3.3 70B FP8 or Qwen2.5-72B FP8) on 2 GPUs with tensor parallelism
Large rerankers (bge-reranker-v2-gemma 2B) with quality ~2 points higher
Late interaction (ColBERT-v2) on one of the nodes for critical cases
Several light LLMs in parallel for rewriter, HyDE and the evals judge

The practical rule: the reranker is one of the most efficient things you can put on a GPU. The VRAM used is modest, the throughput is high, and the quality improvement per euro of extra hardware is among the best in the RAG stack.

The seven traps that kill retrieval

Trap 1 — Top-k to the LLM with no reranker. Dense retrieval returns a top-5 and those 5 chunks go straight to the LLM. Without a reranker, the order of the 5 is the cosine similarity order, which is not the real relevance order. The LLM answers based much more on the first chunks than the last ones, and the nuance is lost. Symptom: confident answers with citations that are not the most relevant in the corpus.

Trap 2 — Dense-only with no BM25. Short queries and queries with internal jargon work badly, the demos look pretty, real traffic complains. Symptom: on QA with real support tickets, the system fails specifically on queries of fewer than 5 words or with product codes.

Trap 3 — Reranker badly calibrated on top-k. A layer 1 top-k that is too small (top-10) cuts before the reranker can do its job. A top-k that is too large (top-200) blows up latency without improving nDCG. The canonical sweet spot is top-30 to top-50 from the hybrid → top-5 to top-10 from the reranker. Symptom: the reranker’s nDCG@5 stuck no matter which reranker model you try.

Trap 4 — Embedder and reranker in different languages. The corpus is Spanish + Catalan, the embedder is all-MiniLM (English only), the reranker is bge-reranker-v2-m3 (multilingual). The broad layer retrieves badly because of the embedder, and the reranker has little to rerank. Symptom: Recall@50 < 80%, unrecoverable by changing the reranker.

Trap 5 — RRF with k=1. Someone reads the paper, copies the formula and sets k=1 “because they do not understand what it does”. The result: the first positions of the worst ranking (whichever it is) dominate. The fusion stops averaging and becomes “winner-takes-all”. Symptom: the final ranking is almost identical to that of just one of the retrievers.

Trap 6 — A query rewriter that changes the meaning. The LLM rewriter “improves” "how do I cancel" into "termination procedure for the premium service for corporate customers". It retrieves chunks about corporate customers when the user was a private individual. Over-specification by the rewriter is worse than no rewriter. Symptom: queries from “normal” users come back worse than the old ones. Mitigation: the rewriter must keep the user’s intent and only add context, never assume.

Trap 7 — No retrieval telemetry. The system serves, the user complains, you do not know whether the fault was the corpus, the retrieval, the reranker or the LLM. Without emitting traces with retrieved_chunks_ids, retrieved_chunks_scores, rerank_scores, query_rewritten_to, selected_top_k, debugging is theatre. Symptom: each incident takes days to investigate. The tracing with OTel and MCP piece covers the canonical pattern.

All seven are operational. Just as with the corpus, retrieval does not break because the maths is wrong: it breaks because the discipline relaxes. And as in Eval, the metrics can go up while the real experience gets worse, because the golden set accommodates itself to the system instead of the system accommodating itself to the golden set.

What we have not covered (upcoming posts)

Embedding model selection and fine-tuning: how to choose among the embedders on the MTEB leaderboard without falling into goodharting, when and how to fine-tune an embedder on your own domain with MNR loss or triplet loss, which synthetic dataset is generated with an LLM for training (GPL, InPars, Promptagator), and the gotchas of massive re-embedding when you change model.
Semantic cache for RAG: how to cache semantically similar queries to serve answers without going through retrieval or the LLM, GPTCache, MeanCache, and the cache’s precision/coverage trade-off. It saves 30-70% of the cost on workloads with repeated queries.
Multi-vector and ColBERT-v2 at scale: how PLAID and CITADEL indexes are designed to serve corpora of 100M+ chunks with late interaction without burning the memory budget.
RAG-specific eval — RAGAS deep dive: faithfulness, answer relevance, context precision, context recall, noise sensitivity. How the RAG-specific golden set is built (with chunks labelled as relevant/irrelevant) and which metrics correlate with real user satisfaction.
Function calling and tool-augmented retrieval: when the LLM decides which retriever to invoke (SQL for structured data, vector for unstructured, web search for real time), the ReAct pattern, handling tool errors.
Agentic retrieval loops: when a single retrieval pass is not enough and the agent iterates (planning, sub-queries, summary-and-refine). The latency / quality trade-off and the anti-patterns (infinite loops, diverging sub-queries).

References

BEIR: Thakur et al. (2021). “BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models.” NeurIPS 2021 Datasets and Benchmarks Track. https://github.com/beir-cellar/beir
SPLADE: Formal et al. (2021). “SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking.” SIGIR 2021. https://arxiv.org/abs/2107.05720
ColBERT-v2: Santhanam et al. (2022). “ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction.” NAACL 2022. https://arxiv.org/abs/2112.01488
Reciprocal Rank Fusion: Cormack, Clarke, Buettcher (2009). “Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods.” SIGIR 2009. https://dl.acm.org/doi/10.1145/1571941.1572114
HyDE: Gao et al. (2022). “Precise Zero-Shot Dense Retrieval without Relevance Labels.” https://arxiv.org/abs/2212.10496
Step-back prompting: Zheng et al. (2023). “Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models.” https://arxiv.org/abs/2310.06117
BGE-M3: Chen et al. (2024). “BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation.” https://arxiv.org/abs/2402.03216
MTEB leaderboard: https://huggingface.co/spaces/mteb/leaderboard — útil como punto de partida, peligroso como criterio único (overfitting al benchmark, dataset contamination).
TEI (Text Embeddings Inference): https://github.com/huggingface/text-embeddings-inference — el motor de serving de HuggingFace para embedders y rerankers de tamaño pequeño-mediano. Endpoint OpenAI-compatible.
Qdrant hybrid search: https://qdrant.tech/documentation/concepts/hybrid-queries/ — implementación nativa de RRF y multi-vector queries.
RAGatouille: https://github.com/AnswerDotAI/RAGatouille — wrapper Python para ColBERT-v2 que reduce drásticamente la curva de entrada.
Cohere Rerank 3: https://docs.cohere.com/docs/rerank-2 — referencia técnica del reranker comercial multilingüe líder.

RAG corpus curation: the active librarian who decides what goes in, what goes out and what gets signed

Mon, 25 May 2026 11:00:00 +0200

This is the curation layer inside stage 1 (Data) of the six-stage LLMOps pipeline. It complements the other Data posts: dataset versioning for the four versionable artefacts, PostgreSQL + Qdrant ingestion in microservices for the outbox + CDC pattern, and RAG over Kafka and a datalake for streaming transport. This post is not about moving data: it is about what to do with data before you let a model read it.

TL;DR

A RAG system serving mediocre answers is rarely the retriever’s fault, nor the model’s. The root cause is usually the corpus: three nearly identical versions of the same PDF that make top-k return the same thing three times, an old manual nobody removed that contradicts the current one, a free-text field with customer numbers that the model quotes verbatim, a scanned PDF with dirty OCR that the chunker cut in the middle of a sentence. None of that is fixed by changing the model, the embedder, the reranker or the prompt. It is fixed by curating the corpus. This post takes apart the five operational layers of curation (schema-validated ingest, three-level deduplication, PII anonymisation measured with precision/recall, anti-contamination against the golden eval set, chunk→trace lineage), the minimum maths needed to avoid fooling yourself, the 2026 stack (Presidio, Unstructured, Argilla, LangChain text splitters, OpenLineage, Marquez, Great Expectations), the seven traps that reduce the stage to theatre, and the on-premise hardware to sustain all of it without sending anything sensitive to external APIs.

The analogy: the active librarian

A serious librarian does not accept books by the kilo. When someone offers a new volume:

They look at the spine, the ISBN and the stamp: is it legible? Is it catalogued correctly? Does it belong to a recognised collection? Without valid metadata, it does not get in. This is the schema check.
They check whether they already hold a copy: is it exactly the same book? Is it a later edition of the same one? Is it a translated version of something already on the shelves? If so, they decide explicitly what to do (replace, archive the old one, withdraw both from lending). This is dedup at its three levels.
They flag what is restricted: if the book contains identifiable personal data, some pages cannot be lent as they are, so they must be redacted, anonymised or moved to the reserved section. This is PII anonymisation.
They verify it is not this year’s final exam book: if it is, it stays out of the public collection until the syllabus changes, because if students can consult it the exam stops measuring what it is meant to measure. This is anti-contamination against the golden eval set.
They write it into the register: this book, this edition, this provenance, this date, this person who approved the entry. This is lineage.

If the book passes all five, it joins the collection. If it fails any of them, it goes to an auditable quarantine shelf with the reason for rejection. The difference between a good collection and a mediocre one is not size: it is how much discipline you apply across the five layers, every day, to every new book that arrives.

A RAG corpus is exactly that. The only differences are scale (thousands or millions of documents per month) and the fact that the “readers” are LLMs that cannot tell a duplicate from a reinforced truth, nor a PII value from a synthetic example, nor a contaminated fragment from an authentic one.

The four data artefacts and where the RAG corpus fits

Before going down to the five layers it is worth being clear about which corpus we are curating. The Data stage of the pipeline manages four distinct artefacts, each with its own discipline. The data versioning post lists them; here I reorder them from a curation perspective:

Artefact	Who consumes it	Dominant curation
Training dataset	Tune (fine-tuning the model or an adapter)	aggressive dedup + quality filters + label balancing
RAG corpus	Deploy (retrieval at request time)	the 5 layers in this post
Golden eval set	Eval (promotion gates)	strict hold-out + stratification + maintenance driven by incidents
Retrain enriched dataset	Retrain (closing the loop)	production feedback + human triage

The RAG corpus is the most volatile of the four and the one most exposed to the end user: every answer the system serves literally contains fragments of it. A duplicate in the training dataset degrades learning but ends up buried in the weights; a duplicate in the RAG corpus shows up in today’s answer and tomorrow’s. That justifies the extra discipline that follows.

Layer 1 — Schema-validated ingest

Every piece entering the corpus has to arrive with structured metadata validated against a schema. This is not bureaucracy: it is the only way to make the following layers (dedup, PII, lineage) work without friction.

The canonical pattern is to define a schema in JSON Schema or Pydantic that every document must satisfy:

class CorpusDocument(BaseModel):
 source_system: str # e.g. "confluence", "salesforce", "manual_pdf"
 source_id: str # unique ID in the source system
 version: str # document version (semver or date)
 language: str # ISO 639-1: "es", "en"
 title: str
 body: str
 captured_at: datetime
 captured_by: str # pipeline or human
 sensitivity: Literal["public", "internal", "restricted"]
 schema_version: str # version of the schema itself, not of the document

Any piece that fails this contract is rejected at ingest and never reaches the following layers. Validation is done with Great Expectations (declarative suites), Pandera (more pythonic, integrates with pandas) or Soda (oriented towards continuous data quality). The choice is a matter of style; what matters is:

Validation suites live in code and are versioned with the pipeline, not in a notebook off to one side.
Rejection generates an auditable event (quarantine) with the specific schema failure as its reason, not a log line lost in stdout.
The schema itself is versioned. When it changes, previous documents are reprocessed or explicit backward compatibility is maintained.

The RAG over Kafka post covers the Schema Registry pattern (Confluent Schema Registry or Apicurio) that materialises this in streaming: every message on the topic is validated against the registered schema before propagating downstream. For batch or pull, Great Expectations is the equivalent.

Common trap: leaving the body field free with no further validation. You have to tighten it up with minimum/maximum length (a PDF that yields 12 characters after extraction is almost certainly broken), valid encoding (UTF-8 with no control characters), proportion of alphanumeric characters (dirty OCR returns a soup of symbols). These are simple rules that filter 80% of the noise with no need for AI.

Layer 2 — Three-level deduplication

The most expensive and most silent mistake in a RAG corpus is the duplicate. A document that appears three times in the corpus makes retrieval’s top-k return it three times, wasting two slots and reinforcing a single source. The LLM reads it as if three independent sources agreed, when in fact it is the same thing repeated.

Deduplication is done at three levels, in this order by cost:

Level A — Exact dedup (SHA-256 hash)

Compute the hash of the normalised content (trim, lower-case where applicable, remove redundant whitespace) and compare it against an index of already ingested hashes. On a match, discard or replace. Cost: $O(1)$ per document. It catches literal duplicates (the same PDF uploaded twice, two byte-for-byte copies of the same HTML).

Level B — Near-duplicate (MinHash + LSH)

Nearly identical documents with minor differences (a different header, an updated date, one version in Castilian and another in Galician with minimal changes). The canonical algorithm is MinHash with Locality-Sensitive Hashing (LSH), which approximates Jaccard similarity over shingles of k tokens. For n documents, comparing all against all is $O(n^2)$, which is unworkable for large corpora. LSH cuts the cost to $O(n)$ most-likely buckets.

A typical threshold is Jaccard ≥ 0.80 over shingles of 5 tokens. The standard libraries are datasketch (Python, MIT) or dedup (Python, MIT). A numerical example: for 1 M short documents (300 tokens each), datasketch.MinHashLSH with 128 permutations and threshold 0.8 takes about 2 GB of RAM and processes the full corpus in around 30 minutes on a modern CPU. The fraction of duplicates found in a real enterprise corpus is usually between 5% and 25%; removing them cuts storage and improves retrieval quality at the same time.

Level C — Semantic dedup (cosine over embeddings)

Documents saying the same thing in different words, such as paraphrases, translations or rewritten versions, are not caught by MinHash. This is where semantic similarity comes in: compute the embedding of each document and compare the cosine between pairs.

The problem is quadratic cost: for n documents, computing every pair is $O(n^2)$. For n = 1 M and 768-dimensional embeddings (a typical model such as BAAI/bge-base-en-v1.5), that is 5×10^11 dot products, which is unworkable. The solution is the same idea as LSH but over dense vectors: HNSW (Hierarchical Navigable Small World) or IVF (Inverted File) to build an approximate search index. For each new document you run a k-NN query against the index and examine only the k nearest neighbours.

A sensible threshold for treating something as a semantic duplicate: cosine ≥ 0.95. Below 0.95 the documents are related but distinct; above it, they are almost always the same information rewritten. The exact threshold is calibrated by observing precision/recall over a human-annotated sample; 100 pairs confirmed by a reviewer is reasonable for setting it.

A numerical example: with qdrant or pgvector as the HNSW index and k=10 neighbours per query, deduplicating 1 M documents against the existing corpus takes on the order of 2-4 hours on an RTX 4090 (including embedding computation). If the embedder is self-hosted with vLLM, the cost per token is negligible against compute time.

Policy on what to do with a duplicate

Detection is not enough, you have to decide. Three common policies, in order of complexity:

Drop: discard the more recent one, keep the older one. Simple, with no extra lineage.
Replace: discard the old one, index the new one. More volatility but it reflects the update.
Merge with provenance: mark the new one as a “shadow” of the old one, keep both in lineage but index only one. Better for regulated audit.

The policy has to be explicit and applied uniformly, not an ad-hoc decision per document.

Layer 3 — PII anonymisation with measured precision/recall

This layer is the one that most easily turns into theatre. The typical mistake: install Presidio, run it over the corpus, assume the output is clean. Without measuring the detector’s precision and recall against an annotated golden set, you know nothing.

A PII detector can fail in two ways:

False negative (low recall): it does not detect a national ID written as “12345678-A” because your model was trained on the 12345678A format without a hyphen. The RAG serves personal data unredacted.
False positive (low precision): it redacts a number from a configuration manual thinking it is a phone number. The RAG loses useful information.

Both are problems; regulation (GDPR, ENS, NIS2) penalises the first, and user experience degrades with the second. The acceptable ratio depends on the domain. In medical data, practically zero false negatives is non-negotiable; in internal technical documentation you can tolerate more recall in exchange for less precision.

The standard metric is F1 over an annotated golden set:

\[ \text{precision} = \frac{TP}{TP + FP}, \quad \text{recall} = \frac{TP}{TP + FN}, \quad F_1 = 2 \cdot \frac{\text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}} \]

To build the PII golden set, annotate around 200 documents by hand with every entity marked (national ID, IBAN, email, phone, address, personal name). Then run the detector and compute the metrics per category, not just aggregated, because a global F1 of 0.90 can hide a recall of 0.55 on IBANs.

2026 stack for this layer:

Microsoft Presidio (MIT, Microsoft): the most complete OSS option. Configurable detectors, recognises around 50 entities by default, extensible with your own regex patterns or with fine-tuned NER models.
spaCy NER (MIT, Explosion AI): a base for custom detectors; useful when Presidio does not cover a domain entity.
Llama Guard 4 (Llama Community License, Meta): a safety classifier that also detects PII in one pass, an option when you already have a GPU for inference and prefer a single pass.
DataFog (Apache 2.0): a more recent alternative, specialised in streaming pipelines.

Recommended hybrid pattern: Presidio for rule-based + regex detection (fast, deterministic) → Llama Guard as a second pass over what Presidio did not flag (an ensemble that raises recall without killing throughput). This is measured and reported as aggregate and per-category F1 on every release of the detector.

Common fallacy: trusting that a detector with F1 0.95 “is very good”. If you have 1 M documents and each contains on average 1 PII entity, F1 0.95 means 50,000 mishandled entities (between false positives and negatives). With sensitive data you have to design so that false negatives go to human quarantine, not to the public corpus.

Layer 4 — Anti-contamination against the golden eval set

If the RAG corpus contains fragments of the golden eval set, Eval metrics measure memorisation. The model returns the exact answer because it has it literally in its context, not because it generalised anything. Deploy then promotes models that shine in the exam and fail in production.

This layer is the easiest to implement and the easiest to forget:

The golden eval set has its own versioned hash.
Before indexing any new document into the RAG corpus, run a token-by-token overlap check (or by shingles, similar to MinHash) against the golden set.
If overlap exceeds a threshold (typically ≥ 30% of 5-token n-grams), the document is not indexed. It stays in quarantine flagged as “contamination risk vs golden_v12”.
A human reviews the rejections. Sometimes they are false positives (a short quotation, a boilerplate sentence). Sometimes they are real contamination that a supplier introduced without realising.

The deeper reason: the RAG corpus and the golden set are enemies by design. The golden set measures how well the system generalises to questions it has not seen. If those questions are in the RAG, the system “sees” them on every query. The metric stops measuring generalisation.

This check is computationally trivial, a hash join over n-grams. The difficulty is keeping it up: every time the golden set changes (monthly or quarterly), the full corpus has to be re-validated against the new golden set. Without that discipline, contamination sneaks in through the back door when someone updates the golden set with real cases that the RAG was already serving.

Layer 5 — End-to-end lineage: from document to trace

The last layer is the one that closes the auditable chain. Every chunk indexed in the vector store carries metadata that lets you answer the forensic question:

“The system generated this answer on 14 March at 16:23. Which exact document did the quoted fragment come from? When did that document enter the corpus? Which version of the embedder processed it? Who approved its ingest?”

Without lineage, that question is unanswerable. With lineage done properly, it is four queries.

The canonical pattern:

Every indexed chunk carries in its metadata: source_system, source_id, document_version, chunk_index, embedder_version, dataset_hash, ingested_at, ingested_by, schema_version.
Every RAG answer in production emits a trace span that includes the retrieved chunk_id values.
The central tracing system (Langfuse, Phoenix or OpenLLMetry) joins chunk_id → chunk metadata → document metadata → corpus dataset_hash → embedder version → and so on.

The tools that standardise this glue are OpenLineage (Apache 2.0, LF AI & Data) and Marquez (Apache 2.0, its server implementation). They define a lineage event schema that interoperates between systems; an ingest job emits an event “produced corpus_v12.3 from source X with embedder bge-base-v1.5”; a retrieval job emits “consumed corpus_v12.3 with query Q produced response R”. The graph is reconstructed automatically.

This layer is the only way to pass real audits under regulations such as the EU AI Act, GDPR or ENS, where traceability of which data went into which answer is a requirement, not an option. Without it, the answer “we do not know which document this came from” is not acceptable, and it is the default answer if lineage is not built from day one.

The maths that matter

Beyond dedup thresholds and PII F1 scores, there are three mathematical pieces that any serious team ends up using.

Chunk size vs retrieval quality. Chunk size affects retrieval quality in a non-monotonic way: chunks that are too small fragment ideas (retrieval returns a piece without context), and chunks that are too large dilute the signal (the embedding mixes several topics and similarity drops). The empirical sweet spot for technical text in 2026 is between 256 and 768 tokens per chunk, with 15-25% overlap between contiguous chunks to preserve continuity.

Numerically, for a corpus of 1 M documents with a mean length of 2,000 tokens, chunking at 512 tokens with overlap 100 gives $\frac{2000}{512 - 100} \approx 5$ chunks per document, a total of roughly 5 M indexed chunks. With 768-dimensional embeddings in float32, that occupies $5 \cdot 10^6 \cdot 768 \cdot 4 \approx 15$ GB of vector memory, manageable in any modern vector store.

PII golden set coverage. To know whether the annotated PII golden set is representative enough, compute the proportion of categories covered: if your golden set of 200 documents has 5 IBAN examples and production has 12,000 IBANs per day, the measured F1 on IBANs is statistical noise. Rule of thumb: a minimum of 30 examples per category for per-category metrics to mean anything.

Cost of re-embedding when rotating the model. Changing the embedder invalidates the whole index. For a corpus of 5 M chunks with a model such as BAAI/bge-base-en-v1.5 (768 dim, around 110 M parameters) served in vLLM on 1× H100, throughput is on the order of 8,000-15,000 chunks/second. Full re-embedding: about 5-10 minutes. For a larger embedder (bge-large, 1024 dim, around 335 M parameters), a factor of 3× worse, around 15-30 minutes. The bottleneck is usually vector store I/O, not GPU compute. The dual-index pattern, keeping the old index serving while the new one is built and doing an atomic swap at the end, avoids downtime and allows rollback.

Applied to typical on-premise hardware

For an on-premise deployment that keeps all curation in-house without sending data to external APIs:

RTX 4090 (24 GB): covers layer 1 (the schema check with Great Expectations is CPU-bound), layer 2 levels A and B (hash + MinHash are CPU-bound), and layer 2 level C semantic dedup with a bge-base-class embedder (8-15k chunks/s, enough for corpora of up to 5-10 M chunks in hours). For Presidio in NER mode (layer 3) it runs comfortably. It is the sensible GPU for the whole curation pipeline on a mid-size corpus.
Generic 4×H100 SXM configuration (320 GB total, NVLink): only needed if the corpus exceeds around 50 M chunks or if you want frequent re-embeddings with large models (bge-large, e5-mistral). In practice, two GPUs serve the embedder at TP=2 with throughput above 50k chunks/s, and the other two go to the PII judge (Llama Guard 4) or to serving the main inference model. Capacity for corpora of hundreds of millions of chunks.

The stubborn arithmetic: with a 4090, corpus curation is an overnight job; with 4×H100, it is minutes. The decision depends on corpus size and on how often you rotate the embedder or the PII rules.

The seven traps that kill this stage

Trap 1 — No validated schema at ingest. Malformed documents reach the chunker, the chunker slices them into nonsense, garbage embeddings enter the index. The RAG answer quotes incoherent text and nobody knows why.

Trap 2 — Dedup only at exact hash level. The corpus fills up with paraphrases and translations of the same document. Retrieval’s top-k returns the same source 3 times. The LLM reads it as three confirmations.

Trap 3 — PII detector without precision/recall measurement. Everyone assumes Presidio “works”. IBANs in non-standard formats slip through. The RAG serves personal data.

Trap 4 — Golden eval set contaminated with the corpus. Eval metrics measure memorisation. Promotions approve models that fail in real production.

Trap 5 — No lineage down to the chunk. The question “where did this quotation come from?” has no answer. Regulatory audit fails. Incidents cannot be investigated.

Trap 6 — Maintenance as a one-off event. The corpus is curated once when the system is initialised, and afterwards everyone assumes it is fine. Six months later the documents are out of date, the new PII rules are not applied retrospectively, dedup has not been re-run after adding new sources. The corpus degrades silently.

Trap 7 — Quarantine without human review. Rejected documents go into a table nobody looks at. False positives pile up, real contamination cases go uninvestigated, the team’s trust in curation erodes and the pressure to “relax the thresholds” begins.

All seven are operational, not technical. Corpus curation does not break because of an algorithm bug: it breaks because discipline slackens. It is the exact equivalent of the kind of degradation that kills Eval suites, and in both cases the symptom is the same: the metrics improve or hold steady while the real experience gets worse.

What we have not covered (upcoming posts)

Vector store versioning proper: an embedding index is not versioned like a raw dataset because it depends on the embedding model. Changing the embedder rewrites the whole index. It is a different animal with its own patterns (index branching, selective reembedding, recall-aware ANN parameters).
Streaming corpus updates with CDC: when the corpus has to be updated in near-real time from an OLTP system. The Postgres + Qdrant ingestion post covers the mechanics; still pending is the pattern for selectively invalidating chunks that depend on deleted rows.
Multi-tenant corpus isolation: how to set up a shared corpus versus one with per-tenant namespaces, with ACLs over individual chunks. Especially relevant for multi-customer RAG under data sovereignty constraints.
Federated corpus: corpora distributed across silos that the system queries without centralising the content. An emerging pattern for companies with several sites and cross-border restrictions.
Reranking-aware curation: how curation discipline changes when there is a reranker (Cohere Rerank, ColBERTv2, BGE-Reranker) reordering the top-k after retrieval. Some duplicates you would tolerate without a reranker become intolerable when the reranker pushes them up the ranking.

References

Presidio: https://microsoft.github.io/presidio/ — official docs, list of supported entities, guide to extending it with custom NER.
OpenLineage: https://openlineage.io/ — event schema spec and libraries per language.
Marquez: https://marquezproject.ai/ — server implementation of OpenLineage.
datasketch (MinHash + LSH): https://ekzhu.com/datasketch/ — the reference Python library for near-duplicate deduplication at scale.
Great Expectations: https://docs.greatexpectations.io/ — declarative data quality suites.
Unstructured: https://docs.unstructured.io/ — parsing and normalisation of heterogeneous documents (PDF, HTML, DOCX, eml) before chunking.
Argilla: https://docs.argilla.io/ — human annotation UI for building the PII golden set and other calibration sets.
Llama Guard 4: Meta technical paper, multimodal safety classifier — useful as a second PII detection layer.
GDPR, EU AI Act, ENS, NIS2 — the regulatory frameworks whose compliance depends, in practice, on the discipline of layers 3 (PII) and 5 (lineage). The CEN/CENELEC technical standards for conformity assessment of GenAI systems under the EU AI Act are still pending final publication.

Evals for LLMs: the layer after tracing that decides whether your model performs or only looks like it does

Mon, 25 May 2026 07:30:00 +0200

This is stage 3 of the LLMOps pipeline. If you arrive without context on the full journey, the six-stage LLMOps pipeline describes where Eval sits between Tune and Deploy, and the anatomy of a production LLM request shows a real eval blocking the promotion of an adapter.

TL;DR

Tracing is not evaluation. Tracing tells you what happened; eval tells you whether what happened is any good. The two layers live in tools that overlap visually (Langfuse does both), but the discipline is different: tracing is continuous capture over real traffic; eval is controlled execution against a stable dataset, with metrics that have to fail CI if they drop below a threshold. Without eval, the Tune → Deploy loop closes blind: adapter v8 goes to production because the engineer “saw it answering well” on five examples. With eval done properly, v8 only gets in if it beats v7 across a battery of 500 curated cases, evaluated by a mix of heuristics, embeddings, an LLM judge calibrated against humans, and a sample of real traffic with human annotation. This post takes apart the mechanism, the mathematics that keep you honest, the real tools in 2026 and the pitfalls that turn it into theatre.

The analogy: the academic examination board

A doctoral candidate defends a thesis. They do not defend it before a single distracted professor: they defend it before a mixed board, with a syllabus fixed in advance (not improvised on the day), and with an explicit pass mark that separates pass from fail. The board is not a single expert: it is a panel combining fast readers (the heuristics: does it have the requested format? does it include the citation?), semantic reviewers (the embeddings: does it resemble the expected answer?), an external assessor trained on the subject (the LLM judge: is it faithful to the context, is it relevant, does it sound coherent?), and human jury members on a sample of the most sensitive cases. If the candidate does not reach the mark, they are not promoted: they go back and prepare the defence again.

That analogy has three edges worth holding on to from the first minute:

The syllabus is fixed, but it is actively updated when a new topic appears in the real world that the candidate must know. If it is not updated, the exam measures something increasingly distant from what happens outside.
The board has to be trained: an LLM judge not calibrated against humans is an examiner making up the criteria.
The pass mark is published beforehand: it is not decided after seeing the result, because then it is not a pass mark, it is a justification.

These three ideas run through the rest of the post. Every tool and every piece of mathematics that follows is, at bottom, a way of operationalising them.

The mechanism itself: four layers of evaluators

An eval suite in 2026 is made up of four layers that coexist. None replaces the others; each one measures what the rest cannot measure well, and leaves unmeasured what the rest do measure well.

Layer 1 — Deterministic heuristics. Rules that return true or false with no ambiguity: the output matches a regex, contains a specific entity, does not exceed a length, follows a valid JSON schema, respects a requested format (markdown, tool_call function, mandatory citation). They are cheap, extremely fast, need neither judge nor embeddings, and catch the most frequent class of bug: the model answered in the wrong format. Their limit is obvious, in that they do not know whether the answer is correct, only whether it is well formed.

Layer 2 — Semantic metrics with embeddings. They compare the model’s output with an expected answer by computing cosine similarity between their embeddings, or by measuring whether a claim in the output is entailed by the retrieved context. They are cheap, deterministic given the embedding model, and very useful for detecting answers that drift in meaning but not in form. Their limit is equally clear: two answers can have high cosine similarity and say the opposite of each other (“the customer can cancel at any time” vs “the customer cannot cancel at any time” share 80% of their tokens).

Layer 3 — LLM-as-judge. A model, or an ensemble, evaluates the output of the model under test with a prompt designed to produce a score on a rubric. The canonical methods in 2026 are G-Eval (chain-of-thought prompting with a calibrated numeric score), Prometheus (open-source judge trained specifically for evals, reporting 0.897 correlation with humans in its v2.5 release from late 2025), and panel-of-judges setups that average votes from three heterogeneous models to reduce bias. This layer captures nuances the previous two cannot see: is it faithful to the context? is it useful? is it safe? is it complete? Its limit is cost and the need for calibration, treated as a whole section below.

Layer 4 — Humans. Trained annotators who evaluate a sample of the eval set, not all of it. They are the only layer with ultimate authority over the rubric: the LLM judge is calibrated against them, not the other way round. They are expensive (≈ 0.50–2.00 € per annotated sample when the domain is technical) and slow (a competent annotator produces 60–120 quality annotations per working day). The mistake teams make again and again is dropping this layer “because we have an LLM judge”; without humans there is no calibration, and without calibration the judge measures whatever it feels like.

Normal operation of an eval gate combines all four: heuristics remove malformed outputs before spending judge tokens, embeddings filter out the manifestly irrelevant, the judge scores the rest, and humans annotate a sample every N runs to keep the judge calibrated.

The golden dataset: a versioned syllabus

The golden dataset is the most undervalued artefact in the pipeline. It is the exam. If it is badly built, everything else, the best-calibrated judge in the world, the strictest gates, the fastest suite, measures noise. Covered at the top level in the data versioning post as one of the four artefacts to version differently; here we go into detail from the Eval perspective.

The three properties a golden dataset has to satisfy are:

Stratified representativeness. The dataset has to cover the system’s real input space in proportions that reflect production. If 30% of real traffic is in German, 30% of the golden set has to be in German; if 12% of questions are about subscription cancellation, that category cannot be 60% of the eval set just because it was easy to annotate. Stratification stays auditable: each example carries tags (lang=de, category=cancellation, tenant_type=enterprise, difficulty=hard) and the suite reports metrics per segment, not just the aggregate.

Strict holdout, no contamination with training. This rule is so obvious that almost every team believes it is following it, and almost every team breaks it without noticing. If the golden eval set gets mixed with the fine-tuning dataset, because somebody did a badly built random_split, because a purchased dataset is used by other vendors for training, because the LLM judge saw it during pretraining, then the metric stops measuring generalisation and starts measuring memorisation. The eval set hash is versioned separately (covered in the continuous fine-tuning post) and a routine leakage check is run: if an example in the golden set matches one in training token for token, it raises an alert.

Reasoned sample size, not aspirational. How many examples are needed? There is a mathematical minimum for telling two models apart with confidence. If model A is right 80% of the time and model B 85% on the same set, the 95% confidence interval for that difference of proportions (unpaired) is:

\[ \Delta p \pm 1{.}96 \cdot \sqrt{\frac{p_A(1-p_A) + p_B(1-p_B)}{n}} \]

To distinguish 80% from 85% with 95% confidence (an interval that does not cross zero), you need n ≈ 700 examples. To distinguish 90% from 91% under the same criterion, the calculation gives n ≈ 6,500. A “we have 50 examples in the golden set and we see adapter v8 scoring 90%” means nothing statistically: the confidence interval is ±8 points. The field’s practical rule in 2026 is a minimum of 300 examples to detect coarse differences, ideally 500–1,500 if you want to detect fine improvements, and to start by stratifying properly before obsessing over sample size.

On top of this comes active maintenance: the golden set is enriched with production incidents (covered in detail in the retrain post), so that every real complaint that ends in a bug becomes a curated example the next deploy candidate will have to pass. The golden set is not static: it is a living record of errors the system has already made and must not make again.

LLM-as-judge: how you calibrate an examiner

Layer 3 is the one most teams misuse. The typical mistake is: “we use GPT-4 as judge because it is the most capable”. The judge is not chosen for nominal capability; it is chosen for agreement with humans on the specific rubric you are measuring. A judge with 60% agreement is useless even if it is GPT-5; a Prometheus 7B fine-tuned for your domain with 88% agreement is worth more.

The standard metric for measuring agreement between two annotators (human vs judge, or two humans with each other) is Cohen’s kappa, which corrects for the agreement expected by chance:

\[ \kappa = \frac{p_o - p_e}{1 - p_e} \]

where $p_o$ is the observed proportion of agreement and $p_e$ is the proportion expected by chance under each annotator’s marginal distributions. The accepted interpretations in the literature are:

κ < 0.40: poor agreement. The judge says whatever it likes.
κ ∈ [0.40, 0.60]: moderate. Acceptable for coarse signals (is it toxic?), dreadful for nuance (is it faithful to the context?).
κ ∈ [0.60, 0.80]: substantial. Useful in production for most metrics.
κ > 0.80: almost perfect. The judge can be treated as a substitute for the human for that specific kind of judgement.

Numerically, consider a binary rubric (faithful / not faithful) over 200 examples annotated by a human and by a judge. If the human said “faithful” in 150 cases and the judge in 140, and they agree on 175 of the 200, then $p_o = 0{.}875$; the marginals are $p_h = 0{.}75$, $p_j = 0{.}70$, and $p_e = 0{.}75 \cdot 0{.}70 + 0{.}25 \cdot 0{.}30 = 0{.}600$. Kappa comes out at 0.6875: substantial but not excellent, usable, with an eye kept on the rubric.

Calibrating the judge means an explicit process:

Build a calibration set — 100–300 examples annotated by trained humans, with written guidelines. Inter-annotator kappa between the humans is also measured; if the humans themselves cannot agree, the rubric is badly written before you even start talking about a judge.
Iterate the judge prompt until judge agreement with the humans passes the accepted threshold (typically κ ≥ 0.7 for sensitive metrics).
Pin the judge version (claude-3-5-sonnet-20251022, gpt-4o-2024-11, prometheus-2-7b@sha256:…): any change invalidates the calibration.
Recalibrate periodically — every time the judge, the judge prompt, or the rubric changes. The frequency the field recommends in 2026 is quarterly at minimum, monthly if the rubric is new.
Persist everything in lineage — a score of “faithfulness 0.87” with no traceability of which judge, which prompt, which calibration set and which human validated it, is decorative.

The prompt versioning post covers how judge prompt versioning is materialised. The practical consequence is that the judge is versioned like any other model: your eval suite has judge_id = prometheus-2.5@v3 just as it has adapter_id = customer_support_v7.

The two cadences: CI gate and platform regression

Eval suites live in two places and run on two different cadences. Teams that conflate them turn one of the two into theatre.

CI gate (pre-merge, blocking). It runs on every pull request that modifies prompts, adapters, RAG configuration, or any artefact that could move the model’s output. It runs against the golden dataset versioned at the hash that is on main. The gate fails the merge if:

the critical metric drops more than X percentage points in absolute terms against the baseline (typically X = 2);
some safety metric (toxicity, PII leakage) crosses a hard threshold (typically tox > 0.02);
some strategic segment (a language, an enterprise-type tenant) drops more than Y points even if the aggregate improves.

This cadence has to be fast (ideally < 10 minutes over 500 examples) and cheap (LLM judge in batch mode, cached embeddings, heuristics run locally). The CI gate is not exhaustive: it is the low-latency line of defence.

Platform regression (post-deploy, continuous). It runs on a schedule (typically nightly or weekly) over sampled production traffic, not over the static golden set. It detects drift: the model has not changed, the golden set has not changed, but the users have, and quality over real traffic drops. This cadence is more expensive (judge over thousands of samples, human annotation over hundreds), tolerates latencies of hours, and its main consumer is not CI but the observability dashboard and the humans on the product team who decide whether to open a retrain cycle.

Both cadences persist results in the same store (Langfuse, MLflow, or equivalent) and connect them by model_id, prompt_id, dataset_hash and judge_id. Without that glue of identifiers, the metric that passes CI cannot be correlated with the one that fails in production three weeks later.

The minimum mathematics that matter

Beyond kappa and the interval for the difference of proportions, both already covered above, there are three more mathematical pieces that any team running evals seriously ends up using.

Confidence interval for a continuous metric. If your metric is a continuous score (faithfulness ∈ [0, 1]) and you measure the sample mean $\bar{x}$ over n examples with standard deviation s, the 95% confidence interval for the population mean is:

\[ \bar{x} \pm 1{.}96 \cdot \frac{s}{\sqrt{n}} \]

For n = 300 and s ≈ 0.2 (typical of a 0-1 score with non-degenerate variance), the margin is ±0.023. This means that differences below 2 hundredths cannot be distinguished from noise at that sample size. If your team is chasing “+0.5 pp” improvements over 100 examples, it is optimising noise.

Judge cost as a function of sample size and rubric. The cost of one eval pass with LLM-as-judge over n examples, with m metrics evaluated in a single call per example, and per-token price $c_{in}, c_{out}$ on the judge model, is:

\[ C \approx n \cdot (t_{in} \cdot c_{in} + t_{out} \cdot c_{out}) \]

where $t_{in}$ is the number of input tokens (includes context, output of the model under test, full rubric) and $t_{out}$ the output tokens (includes the judge’s CoT + score). For n = 500, $t_{in}$ ≈ 4,000, $t_{out}$ ≈ 300, a GPT-4o judge at May 2026 prices, one pass costs on the order of 8–15 USD per suite. If the pass fires on every PR and there are 30 PRs/day, that is 240–450 USD/day on CI gates alone. Multiplied by continuous regression, teams that do not control this spend four figures a month on judge tokens without realising. The canonical mitigation is a mix of layers: heuristics and embeddings filter first, the judge is only invoked on what the cheap layers cannot resolve, and for platform regression a self-hosted open-source judge is used (Prometheus 7B on your own GPU plane) instead of a commercial model.

Distinction between aggregate metric and per-segment metric. The classic eval fallacy is the hidden mean. If your suite reports faithfulness = 0.87 and the team reads it as “up 2 points against the previous adapter”, this may be what is happening: the new adapter gains 4 points in English (where 70% of the eval set is) and loses 6 points in German (where 30% is). The aggregate mean improves, the German experience gets worse. Any serious suite reports a breakdown by strategic segment (language, tenant type, question category, context length). The CI gate can also have per-segment thresholds, not just aggregate ones.

The 2026 stack: dominant tools

Tool	Main layer	Licence	Maintainer	When to pick it
DeepEval	CI gate	Apache 2.0	Confident AI	“Evals as pytest” — assertions in Python code, trivial integration with GitHub Actions. A reasonable default.
Promptfoo	CI gate	MIT	Promptfoo Inc.	Declarative YAML, matrix of prompts × providers × assertions, diff vs baseline. DevOps-friendly.
RAGAS	RAG-specific metrics	Apache 2.0	Exploding Gradients	Faithfulness, context relevancy, answer relevancy. The canonical piece if your system is RAG.
Inspect AI	Safety/capability evals	MIT	UK AI Safety Institute	Suite focused on safety and capability. Useful for regulatory gates under the EU AI Act.
Langfuse Evals	Platform regression	MIT (OSS) / EE	Langfuse GmbH	Integrated with tracing — datasets, runs and scores in the same UI as production traces.
MLflow GenAI Evals	Registry + evals	Apache 2.0	Databricks/LF AI	Good when you already have MLflow for classic models; “GenAI dashboard” since 3.10.
Phoenix (Arize)	Eval + visual drift	Elastic License 2.0	Arize AI	Focused on visual debugging of embeddings and drift; a complement to Langfuse, not a replacement.
Prometheus 2.5	Self-hosted OSS judge	Apache 2.0	KAIST + LG AI	Fine-tuned judge, high correlation with GPT-4 at zero cost per token when self-hosted.
G-Eval / JudgeLM	Prompting methods	— (techniques)	academic	Prompting frameworks for LLM-as-judge — they apply on top of any judge model.

The OSS catalogue by stage goes into detail entry by entry; the parallel OSS vs hyperscalers catalogue compares against Bedrock Evaluations, Vertex AI Eval Service and Azure AI Evaluation.

The canonical pattern in 2026 is hybrid: DeepEval or Promptfoo for CI gates + Langfuse Evals for platform regression + Prometheus 2.5 as a self-hosted judge + human annotation on Argilla or Label Studio for the calibration set. Swapping any of these pillars for equivalents (W&B Weave instead of Langfuse, Inspect instead of DeepEval) is style, not functionality; what matters is that the four functions are present and connected.

Applied to typical on-premise hardware

For an on-premise deployment that wants to avoid sending sensitive outputs to a commercial LLM judge (for data sovereignty, ENS, NIS2 or equivalents), the judge is hosted on your own GPU plane. Reference figures for May 2026, based on Prometheus 2.5 (Llama-3.1-8B fine-tuned as a judge) served on vLLM:

RTX 4090 (24 GB, Ada Lovelace): viable for small suites (< 200 examples) and for running the judge offline during development. Latency per evaluation ≈ 1.5–3 s with BF16; aggregate throughput on the order of 25–40 evaluations/min with batching. Useful for a developer’s local CI gates, not for platform regression.
Generic 4×H100 SXM configuration (320 GB total, NVLink): runs the judge in parallel at TP=2 across two GPUs, leaving two free to serve the model under test. Aggregate throughput on the order of 200–350 evaluations/min, a full 500-example suite in 2–3 min. This allows PR gates with no perceptible waiting and nightly platform regression over thousands of samples with no per-token cost.

The comparative cost arithmetic is stubborn: self-hosting Prometheus 2.5 pays off an H100 in roughly 6 months if the team fires ≥ 30 PRs/day with eval gates. Below that volume, the commercial judge still wins unless sovereignty is a requirement, and under ENS / NIS2 it is.

The seven pitfalls that kill this stage

Pitfall 1 — Aged golden dataset. It is not enriched with production incidents. After months it measures a world that no longer exists. The metric climbs comfortably while real users complain more.

Pitfall 2 — Contaminated or uncalibrated judge. The LLM judge evaluates with criteria it invents itself. Without a human calibration set as reference, there is no way to know whether its 0.89 is generous, severe or random.

Pitfall 3 — Insufficient sample size. A 50-example suite, 1-point differences the team treats as significant. The confidence interval is ±10 points. They are measuring noise and making real decisions on it.

Pitfall 4 — Runaway cost. A GPT-4 batch-mode judge fired on every PR over 1,000 examples, with no prior filtering by cheap layers. The eval bill overtakes production serving. It happens more often than people admit.

Pitfall 5 — Aggregate metric hiding segments. Global mean improves 2 points, German drops 6, enterprise tenants drop 3. With no explicit per-segment breakdown, the gate approves what it should not.

Pitfall 6 — Judge with a floating version. Judge model updated without recalibrating the rubric. Thresholds lose their statistical meaning. Last month’s regressions are not comparable with those from two months ago.

Pitfall 7 — Eval gate that is not enforced. The gate exists in the documentation but not in the real workflow: the suite takes 30 minutes, developers skip it with --no-verify, managers ask for one-off exceptions that become the norm. An eval gate without enforcement is ornament.

All seven are operational, not technical. The Eval layer does not break because the mathematics are wrong: it breaks because the discipline slackens. It is the same thing that happens with unit tests in any project that grows, except that here, without the discipline, the system improves its metrics while getting worse, and that makes the degradation invisible until it is already ungovernable.

What we have not covered (upcoming posts)

Guardrails and online safety: the inline eval layer that filters outputs in real time, not in CI. Conceptually a cousin of Eval, but with very different latency constraints. Covered in detail in the guardrails post: four lines of defence (input, retrieval, tool, output), OWASP LLM Top 10 (2025), OSS catalogue and deployment patterns.
Judge ensembles and agreement between multiple judges: how to reduce bias by combining three heterogeneous models as a panel, which aggregation function works (simple majority, trimmed mean, calibrated meta-judge).
Meta-eval: how you evaluate the suite itself. If the eval improvement from adapter v7 to v8 does not translate into a real improvement for the user, the suite is wrong, and that is also measured, by correlating eval metrics against product metrics.
Metamorphic testing: evaluating robustness against input perturbations (typos, paraphrasing, alternative language) as an extra gate. Beyond nominal agreement, it measures consistency.
Adversarial evals with red teaming: introducing prompt injection and jailbreak attacks as part of the gate.
Privacy in the judge: how to stop sensitive outputs from the model under test travelling to an external judge when regulation forbids it — homomorphic judge, judge in a TEE, or simply a self-hosted judge.

References

G-Eval: Liu et al., “G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment” — the reference paper for the chain-of-thought LLM judge method.
Prometheus: Kim et al., “Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models” — OSS judge with a reported 0.897 correlation vs humans.
RAGAS: Es et al., “RAGAS: Automated Evaluation of Retrieval Augmented Generation” — the paper that standardises faithfulness, context relevancy and answer relevancy.
Cohen’s kappa: Cohen, J. (1960). “A Coefficient of Agreement for Nominal Scales.” Educational and Psychological Measurement — the classic metric for inter-annotator agreement, still the operational reference.
DeepEval docs: https://docs.confident-ai.com/
Promptfoo docs: https://promptfoo.dev/docs/
Langfuse Evals: https://langfuse.com/docs/scores
Inspect AI: https://inspect.ai-safety-institute.org.uk/
EU AI Act, relevant articles on mandatory evaluation of high-risk systems — pending publication of the CEN/CENELEC technical standards on conformity assessment for GenAI.

The OSS catalogue for LLMOps in six stages: tool by tool, what each one does and when to choose it

Sat, 23 May 2026 07:30:00 +0200

TL;DR

For each of the six LLMOps stages (Data, Tune, Eval, Deploy, Observe, Retrain) and the two cross-cutting components (prompt + data versioning), the open source ecosystem has canonical pieces that this blog has been citing again and again. This post gathers them in one place with cards of roughly 150 words per core tool: what it does, how it differs from the alternatives inside the same bucket, its licence and governance model, and a typical gotcha that you only learn in production. Plus more alternatives as bullets, a decision matrix per stage according to the case (small / large corpus, single tenant / multi-tenant…), a diagram of the connected OSS stack and a master table of licences / EE offerings. The intention: that the reader closes the post knowing what is available, which company maintains it, what gap each piece fills, and when to choose it. This is not opinion: it is a curated catalogue.

You are here: every stage, but by OSS column

This post shares a map with the two previous ones in the series, the six stages and the two cross-cutting pieces are all active, but it zooms in on the open source column.

The analogy: the electrician’s toolbox

A professional electrician arrives at an installation with a box organised into compartments. Nothing is improvised: for every type of cable there is a specific wire stripper, for every screw a screwdriver of the exact gauge, for every measurement a multimeter and a clamp meter, for every connection the right terminal block or connector. The difference between a professional electrician and a handyman is not that one knows more theory, the handyman has often read the manuals, it is that the right tool is within arm’s reach and he knows when to use each one. The day the specific stripper is missing, improvising with a box cutter breaks the insulation, leaves a badly terminated cable, and the panel comes back under warranty two months later.

The OSS LLMOps stack works the same way. For every canonical problem, versioning a dataset, indexing a corpus for retrieval, serving tokens with dynamic batching, propagating trace_id end-to-end, managing prompts with a production label, orchestrating retraining pipelines, there is a canonical piece of the open source ecosystem that solves it, maintained by a serious community or foundation, with a clear licence and a well documented gotcha. The consultant who knows which tool to use for each job builds a robust system in weeks; the one who improvises with “whatever the team already knows” pays later in operations, usually once the system is carrying real load and any replacement is expensive.

This post opens the toolbox and shows each card. It is not a user manual, that is what the deep-dive posts linked at the end are for; it is the curated catalogue.

Diagram of the connected reference OSS stack

The catalogue makes sense once you see how the pieces connect into a single coherent architecture, the one this blog has been describing throughout the series. The boxes do not float; they talk to each other over stable contracts (HTTP, gRPC, OTel, Kafka, S3/MinIO API).

Solid arrows mark data / control flow; the dashed blue ones are OTel traces. The K8s plane holds everything up. The control plane at the bottom is where the retraining pipelines, the evals in CI, the versioned prompts and the lineage live. The data plane on the left feeds both the serving side (RAG, configs) and the control plane (datasets, lineage). The observability plane receives from serving and from everything else.

Now we go stage by stage. Each one opens with a context paragraph, then cards for the core tools (about 150 words each), bullets with relevant alternatives, and a specific decision matrix at the end.

Stage 1 — Data + cross-cutting Data versioning

The Data stage solves three distinct problems that beginners confuse: versioning datasets (so that (dataset_id, version, hash) exists and propagates), storing and serving the operational corpus (object store + vector index + structured text), and moving it between systems with CDC and stable schemas. Covered in detail in the posts on data versioning with DVC and lakeFS, PostgreSQL + Qdrant in ingestion and RAG over Kafka.

DVC (Data Version Control)

DVC puts datasets under version control with the same discipline that git applies to code. The .dvc pointers live in git (plain text, about 200 bytes per dataset), the bulk content lives in a remote object store (S3, MinIO, Azure Blob, GCS). Each dvc add computes a SHA-256 hash of the dataset, uploads it to the remote and stores the pointer. The key line: the dataset_hash becomes the luggage tag that travels to the trainer, to experiment tracking and to lineage. The same dataset retrained twice produces the same hash, and therefore reproducible experiments. DVC integrates with MLflow and W&B as an input artefact. Gotcha: it works well for datasets that change by replacement (I swap train.jsonl for a new version) and worse for datasets with thousands of small files that change individually. For that case, combine it with lakeFS. Licence Apache 2.0, maintained by Iterative.ai since 2017. There is DVC Studio (managed) and dvc data (pure CLI) on different planes.

lakeFS

lakeFS brings git semantics (branch, commit, merge, rollback) to an entire S3/MinIO/ADLS bucket. Where DVC versions individual files as pointers in git, lakeFS versions the whole bucket: you can branch the corpus, ingest new data into the branch, validate that it passes checks (recall@10 over golden queries for embeddings, completeness for a tabular corpus), and only then merge into main. It is the piece that makes continuous RAG safe: the production corpus is always on main, updates are tested on branches. It offers hooks (pre-merge, pre-commit) that fire automatic validations, and time-travel to reproduce the state of the bucket at a past date. Gotcha: the manifest overhead on huge buckets (hundreds of millions of objects) deserves sizing; lakeFS keeps metadata in its own Postgres, not in the bucket. Licence Apache 2.0, maintained by Treeverse since 2020. Managed offering: lakeFS Cloud.

MinIO

MinIO is the S3-compatible object store that fills the “S3 on-premise” gap without surprises. API identical to S3 (the AWS SDKs work by pointing them at a different endpoint), its own CLI client (mc), erasure-coded mode for fault tolerance, bucket-to-bucket replication, encryption at rest. It is the base on which the other data plane components are built: DVC remote, lakeFS underlying storage, Postgres snapshots, MLflow artefacts, eval datasets, saved models, distributed KV cache fabric. In small deployments it runs single-node multi-disk; in serious ones, distributed clusters. Gotcha: the licence changed to AGPLv3 in 2021 (it was Apache 2.0 before), which means distributing software connected to MinIO obliges you to open the code that connects to it. For internal on-premise use this is not a problem; for a vendor packaging MinIO inside a commercial product, it is. Maintained by MinIO Inc. with a SUBNET enterprise offering and a community fork called AIStor launched in 2025.

Qdrant

Qdrant is the OSS vector database most aligned with this blog’s “RAG corpus per tenant with strict ACLs” pattern. Written in Rust, it exposes a REST + gRPC API, indexes with HNSW + scalar/binary quantisation to cut memory, supports efficient payload filtering (it is not post-filtering: it integrates the filter into the HNSW search), and allows collections isolated per tenant. For the multi-tenant chatbot scenario, Qdrant is where the tenant_<id>_kb_v3 collections with strict ACL live. It scales well horizontally (sharding by payload) and vertically (millions of chunks on a node with 64GB RAM). Gotcha: binary quantisation is aggressive, it cuts VRAM by 32× but degrades recall by 10-20 %; enabling it without re-tuning the threshold breaks retrieval silently. Licence Apache 2.0, maintained by Qdrant Solutions GmbH (Germany). There is Qdrant Cloud (managed) and EU-only support for ENS cases.

PostgreSQL + pgvector

Postgres 18 with the pgvector extension is the stack’s “hidden vector database”: when the corpus is small (under a million embeddings) and Postgres is already in production for operational data, running a separate Qdrant is expensive operations. pgvector adds a vector(dim) type, HNSW and IVF indexes, and the operators <->, <#>, <=> for cosine, L2 and dot product. Combined with tsvector (Postgres full-text search) it allows hybrid search, dense + sparse, in a single SQL query. Version 0.8 (2025) introduced halfvec and bit support to reduce size by 4×-8×. Gotcha: HNSW in pgvector consumes a fair amount of RAM to build the index (roughly 2× the size of the embeddings) and blocks inserts during the build; in production you build on a secondary, promote it, and discard the primary. Licence PostgreSQL License (permissive BSD-style) for both the core and pgvector. Maintained by the PostgreSQL Development Group plus pgvector by Andrew Kane + Crunchy Data + Neon.

Apache Kafka + Debezium

Kafka is the event bus where “everything that happens in the company is a stream” becomes real. For LLMOps in production it serves two functions: CDC from source systems (Debezium captures changes in Postgres / MySQL / MongoDB and publishes them as topics) and LLM event buffer (every request, every piece of feedback, every eval result ends up in a topic with the trace_id propagated). As the post on RAG over Kafka explains, the RAG corpus is kept fresh by capturing changes from the CMS / source system as CDC, running the embedding in Flink streaming, and ingesting into Qdrant continuously. Gotcha: a badly sized Kafka with long retention plus multi-client topics turns into a fast disk sink; measuring throughput per topic and key cardinality before production is mandatory. Kafka licence Apache 2.0 (ASF project); Debezium Apache 2.0 (project incubated by Red Hat). Kafka-compatible drop-in alternative: Redpanda (BSL, restricted commercial use).

Apache Flink (brief mention)

Flink processes streams with sub-second latency and exactly-once semantics. On the LLM plane it is used to: run embeddings in streaming (over CDC topics), aggregate online metrics, materialise features for retraining. Licence Apache 2.0, ASF. Common alternative: Spark Structured Streaming (also ASF, micro-batch latency).

More options for Data, mentioned on the blog:

Ceph — object store for large clusters with geo-distributed replication. LGPL/Apache licence, Red Hat / IBM.
Milvus — C++ vector database, an alternative to Qdrant; better for corpora of billions. Apache 2.0, Zilliz.
Karapace — Confluent-compatible OSS Schema Registry. Apache 2.0, Aiven.
DataHub / Apache Atlas / OpenMetadata — catalogue + lineage. Apache 2.0, Acryl Data / ASF / Collate respectively.
OpenLineage — cross-system lineage event standard. Apache 2.0, Linux Foundation AI&Data.

Decision matrix — Data:

If your case is	Choose
Corpus < 1M embeddings, you already have Postgres	pgvector (one component fewer)
Corpus 1M-100M, multi-tenant with ACL	Qdrant (integrated filtering, ACLs per collection)
Corpus > 100M, aggressive sharding	Milvus (scales linearly better into the billions)
Training datasets + experiment tracking	DVC over MinIO + MLflow integration
RAG corpus with controlled releases	lakeFS over MinIO + pre-merge hooks
You want both	DVC + lakeFS as complements (the blog’s recommendation)

Stage 2 — Tune

The Tune stage produces a new model_id, model_version, typically a LoRA adapter over a stable base, with lineage back to the dataset and experiment tracking to reproduce it. Detail in the post on continuous fine-tuning.

HuggingFace Transformers + PEFT

transformers is the canonical library for loading and training models of the decoder-only family (Llama, Mistral, Qwen, Gemma…) and encoder-decoder. peft (Parameter-Efficient Fine-Tuning) is the complement that adds declarative support for LoRA, QLoRA, IA3 and assorted adapters. Together they form the mandatory core of the OSS Tune stack: any higher-level framework (Axolotl, LLaMA-Factory) uses them underneath. PEFT allows you to train an adapter of about 280 MB (order of magnitude) instead of a full model of about 140 GB, with a functionally equivalent result on most style / domain adjustment tasks. Gotcha: PEFT with a badly configured target_modules trains an adapter that covers only Q and V of the attention, leaving out key, output proj and MLP. The result looks trained but performs poorly; adding target_modules=["all-linear"] fixes it (at the cost of a larger adapter). Licence Apache 2.0, maintained by Hugging Face SAS (a French company); open governance model with active external maintainers.

bitsandbytes

bitsandbytes implements weight quantisation to 8-bit and 4-bit with NF4 for models loaded with transformers. It reduces the 140 GB of Llama 3 70B FP16 to about 40 GB in NF4, allowing QLoRA training on a single H100 80GB. The trick is that the weights stay quantised in memory while the sensitive computations (attention, gradient updates in the adapter) are done in FP16/BF16 with on-the-fly dequantisation. Ideal for fine-tuning on limited hardware and for serving with vLLM when you want to cut VRAM. Gotcha: NF4 quantisation is lossy; on small models (< 7B) the quality degradation is noticeable. For production serving of models under 7B, INT8 is preferred (more memory, less loss) or FP8 if the hardware supports it (the H100 does). Licence MIT, maintained by Tim Dettmers (originally at U. Washington, now with support from Anthropic and HuggingFace).

MLflow Tracking

MLflow is the reference OSS experiment tracking tool: each trainer run records parameters (lr, batch size, epochs, target_modules), metrics (loss curves, eval scores), artefacts (model, tokeniser, configs) and, crucially, input artefacts (dataset_id, dataset_hash, parent_run). The model registry associates each model_version with a reproducible run_id. The line of continuity between Tune and Deploy passes through here: the deployment reads the model to serve from the registry, with its lineage made explicit. MLflow 2.x integrates MLflow Prompts (prompt registry) and MLflow Tracing (OTel-compatible spans), cutting the number of components needed. Gotcha: the default backend store is SQLite, which works for personal experiments and breaks on a shared cluster. In production: Postgres as the backend store + MinIO/S3 as the artefact store. Licence Apache 2.0, maintained by LF AI & Data (donated by Databricks in 2020).

Axolotl

Axolotl wraps transformers + PEFT + bitsandbytes + DeepSpeed + FSDP in a declarative YAML configuration: instead of writing a script of about 300 lines to configure a fine-tuning run, you define a config.yml with base model, dataset path, LoRA config and training hyperparameters, and run it in one line. It supports Llama, Mistral, Qwen, Gemma, Phi… loads. It keeps compatibility with the HuggingFace Hub to download models and datasets, and with MLflow / W&B for tracking. It is the convenience framework this blog cites when it talks about “productive fine-tuning without reinventing the wheel”. Gotcha: the pace of change in the community is fast; a config.yml that worked six months ago can break with a current version because of internal refactors. Pinning the exact Axolotl version in the environment mitigates this. Licence Apache 2.0, maintained by the OpenAccess AI Collective (community-driven). A very similar alternative, more widely used in China: LLaMA-Factory (Apache 2.0, Beihang U.).

Ray Train

Ray Train scales fine-tuning to multiple nodes by distributing workers across a Ray cluster. While DeepSpeed and FSDP are intra-job parallelism (several GPUs collaborating on one job), Ray Train is the orchestration plane that builds the cluster, launches workers, manages checkpoints, recovers from node failures, and integrates with Slurm or Kubernetes. For training runs beyond 8 GPUs on changing clusters, Ray Train avoids the operational burden of “manually launching N torchrun processes with NCCL”. It combines with MLflow for tracking. Gotcha: the Ray learning curve is real; for a single node with 4-8 GPUs, torchrun or Hugging Face Accelerate are simpler. Ray Train shines when there are N changing nodes. Licence Apache 2.0, maintained by Anyscale Inc. (commercial backer) + community. A more K8s-native alternative: Kubeflow Training Operator (Apache 2.0, LF AI & Data).

More options for Tune:

DeepSpeed — ZeRO parallelism in 3 stages, mixed precision, CPU/NVMe offload. MIT, Microsoft.
FSDP (Fully Sharded Data Parallel) — native PyTorch parallelism, an alternative to DeepSpeed. BSD, Meta.
LLaMA-Factory — equivalent to Axolotl with a focus on the Llama family. Apache 2.0, Beihang University.

Decision matrix — Tune:

If your case is	Choose
Fine-tune on 1 GPU with 24GB (RTX 4090)	QLoRA with bitsandbytes NF4 + Axolotl
Fine-tune on 1 H100 80GB, models < 13B	LoRA bf16 + Axolotl
Fine-tune on 4-8 GPUs, single node	transformers + PEFT + Accelerate + MLflow
Multi-node fine-tune on a K8s cluster	Kubeflow Training Operator or Ray Train
Reproducible tracking is mandatory	MLflow + DVC input artefact
You want the minimum viable	Axolotl + MLflow

Stage 3 — Eval + Guardrails

Eval validates candidates pre and post promotion against a golden set with operational metrics; Guardrails runs safety online. Detailed in the posts on evals and guardrails.

DeepEval

DeepEval is the “pytest-style” OSS eval suite: you define tests with assertions over faithfulness, answer relevancy, contextual precision, hallucination rate, summarisation quality… and run them in CI. Each metric is an evaluator: some rule-based, others LLM-as-judge with auditable prompts. The philosophy is “evals as unit tests”: parameterisable by dataset, failable in CI, integrable with GitHub Actions. Gotcha: LLM-as-judge metrics vary between judge model versions, so if the judge moves up a version, the thresholds lose their earlier statistical meaning. Explicit pinning of the judge model in config plus periodic threshold recalibration is mandatory discipline. Licence Apache 2.0, maintained by Confident AI (a company); parallel commercial SaaS offering. Comparable: TruLens (MIT, TruEra) and G-Eval (academic).

RAGAS (RAG Assessment)

RAGAS specialises in evaluating RAG pipelines. It defines four canonical metrics: faithfulness (the answer is supported by the retrieved chunks), answer relevancy (the answer addresses the query), context precision (the retrieved chunks are relevant), context recall (all the relevant chunks were retrieved). Each metric is computed with LLM-as-judge over a dataset of (query, context, expected answer). For a RAG system, RAGAS is the evaluator that measures whether retrieval is aligned with generation. It integrates with Langfuse and MLflow to store results. Gotcha: RAGAS works well with golden sets of under 1000 examples; over huge golden sets the cost of the LLM judge per evaluation explodes, and the practice is to sample. Licence Apache 2.0, maintained by Exploding Gradients (the authors’ company).

Promptfoo

Promptfoo is the declarative, CI-oriented evaluator: in promptfooconfig.yaml you define a set of prompts and a set of assertions (contains text X, does not contain Y, faithfulness > 0.8, judge approves…), point at a provider (OpenAI compatible, vLLM, Ollama…), and promptfoo eval runs the matrix of prompts × providers × assertions, returns a diff against the baseline and fails CI if anything regresses. It is the most “DevOps-friendly” piece of the evals ecosystem: it integrates trivially with GitHub Actions, GitLab CI or Jenkins. Gotcha: assertion thresholds have to be calibrated with real data; starting at a default > 0.5 produces false positives that erode the team’s trust. Calibrate after the first week. Licence MIT, maintained by Promptfoo, Inc. (a company); a commercial Promptfoo Cloud SaaS offering exists but the OSS version is complete.

NeMo Guardrails

NeMo Guardrails is NVIDIA’s framework for defining and enforcing policies in LLM systems through a DSL called Colang. It lets you express rules such as “if the user asks about topic X, answer with template Y” or “if the model tries to do Z, block it” in a conversational-script syntax rather than in Python. It runs as middleware between the app and the model: input rails (validate what comes in), output rails (validate what goes out), dialog rails (control the flow). Designed for complex multi-turn systems where the policies are nontrivial. Gotcha: Colang adds latency per turn (roughly 50-200 ms depending on the policy graph); for high-throughput conversational chat, dialog rails are disabled and only input + output remain. Licence Apache 2.0, maintained by NVIDIA.

Microsoft Presidio

Presidio is the most mature OSS PII (Personally Identifiable Information) detector in the ecosystem. It detects DNI, NIE, IBAN, phone numbers, emails, physical addresses, credit card numbers, personal names, dates of birth… with recognisers based on regex + NER (spaCy) + custom validators. It supports redaction (replace with placeholders), masking (asterisks) or deterministic anonymisation (repeatable hash). For ENS/NIS2 scenarios, it is the piece placed in front (on input) and behind (on output) the LLM to guarantee that no PII is processed or emitted. Gotcha: the built-in recognisers cover English well and the rest badly; for Spanish, Catalan and Basque you have to add custom recognisers, which is disciplined work but doable. Licence MIT, maintained by Microsoft.

More options for Eval:

Phoenix Arize OSS — combines tracing + evals, an alternative to Langfuse Evals. ELv2, Arize AI.
lm-eval-harness — academic suite with standard benchmarks (MMLU, HellaSwag…). MIT, EleutherAI.
HELM — holistic academic evals. Apache 2.0, Stanford CRFM.
Guardrails AI — a pythonic alternative to NeMo Guardrails. Apache 2.0, Guardrails AI Inc..
LlamaGuard / PromptGuard / ShieldGemma — safety models, not frameworks. Open weights, Meta / Google.

Decision matrix — Eval + Guardrails:

If your case is	Choose
Eval in CI, “pytest for LLMs” style	Promptfoo + GitHub Actions
Eval specific to a RAG pipeline	RAGAS + Langfuse datasets
General eval with custom metrics	DeepEval + MLflow dataset
Dialog policy with declarative rules	NeMo Guardrails (Colang)
PII redaction in/out only	Presidio (you do not need NeMo)
Open safety model in Spanish	LlamaGuard 3 or ShieldGemma

Stage 4 — Deploy

Deploy serves tokens to the user with predictable throughput and latency, adapter hot-swap and multi-tenancy where it applies. Covered in the posts on vLLM on K8s, LLM operators, multi-tenant cluster, KV cache, PagedAttention and disaggregated serving.

vLLM

vLLM is the reference OSS inference engine. It implements PagedAttention (KV cache paging in the style of virtual memory, avoiding fragmentation), continuous batching (requests join the batch as they arrive, rather than waiting for the next batch), prefix caching (common prefixes, system prompts, do not recompute the KV cache), LoRA hot-swap (--enable-lora allows loading and unloading adapters without restarting the engine), an OpenAI-compatible API, and disaggregated prefill/decode support since 2025. It covers almost everything from Llama 3 / Mistral / Qwen / DeepSeek. Gotcha: maximum throughput is only reached with --max-num-seqs and --gpu-memory-utilization tuned for the specific model and hardware; the defaults are conservative. The initial tuning session pays off: 2-3× throughput. Licence Apache 2.0, originated at UC Berkeley, today maintained by the vLLM Project / LF AI & Data plus a broad community (Red Hat, NVIDIA, AWS, IBM contribute). Serious alternatives in the same bucket: TGI (Apache 2.0, Hugging Face), SGLang (Apache 2.0, LMSys), TensorRT-LLM (Apache 2.0, NVIDIA, requires conversion).

KServe

KServe is the Kubernetes operator for serving ML models, LLMs included, in a declarative pattern: you define an InferenceService YAML with the model and predictor (which can be vLLM, TGI, Triton, or a custom container) and KServe handles scheduling onto GPU nodes, autoscaling (scale-to-zero included), traffic splitting for canary, and model registry integration. It is the layer that standardises “how a model is deployed on K8s” across multiple engines, instead of inventing engine-specific YAML. It supports multi-model with Inference Graphs (chaining preprocessor → model → postprocessor) and integrates with KEDA/Karpenter for GPU pool autoscaling. Gotcha: scale-to-zero on GPU works badly in practice because the warm-up (loading weights into VRAM) takes tens of seconds; minReplicas: 1 is better. Licence Apache 2.0, maintained by Kubeflow / LF AI & Data. Alternatives: KubeRay (Apache 2.0, Anyscale), llm-d (Apache 2.0, CNCF), KAITO (MIT, Microsoft Azure).

Triton Inference Server

Triton serves heterogeneous models on a single backend: LLMs (via the vLLM or TensorRT-LLM backend), traditional models (ONNX, TorchScript, TensorFlow), custom models. For systems that mix LLM inference with traditional classifiers, embedding encoders, reranking models, OCR and so on, Triton avoids having N different engines in N pods. It supports ensemble models (chaining models in a single request), dynamic batching, model versioning, model warmup. Gotcha: Triton is flexible but heavy to operate; for systems serving only LLMs, vLLM directly is simpler and more optimised. Triton shines when there is real heterogeneity. Licence BSD-3-Clause, maintained by NVIDIA.

Envoy AI Gateway

Envoy AI Gateway is the CNCF ecosystem’s “LLM-aware API gateway”. Built on Envoy Proxy, it adds knowledge of the OpenAI-compatible APIs (chat completions, embeddings, etc.), routing across multiple backends (local vLLM + OpenAI + Anthropic + Bedrock), token-based rate limiting (limits by tokens per minute, not by requests), intelligent retries, fallback between providers, and built-in OTel observability. It is the piece that makes “AI Gateway” real as an architectural category. Gotcha: integration with authentication (OIDC, JWT) is flexible but requires detailed Envoy configuration; an “out of the box” AI Gateway with no configuration produces an Envoy that passes everything through. Licence Apache 2.0, maintained by CNCF since the initial donation from Tetrate. Alternatives: LiteLLM Proxy (MIT, BerriAI), Portkey (MIT, Portkey AI), Kong AI Gateway (Apache 2.0 base + EE, Kong Inc.).

llama.cpp

llama.cpp serves LLMs on CPUs (and Apple Silicon, GPUs via Vulkan/Metal/CUDA) with very aggressive quantisation (GGUF format, down to 2-bit). It is the canonical option for inference on hardware without a dedicated GPU: edge devices, workstations, development machines. It covers everything from small models (Phi-3, Gemma 2B) to Llama 70B on hardware with enough RAM. Gotcha: latency on CPU is orders of magnitude worse than on a dedicated GPU, so it is useful for offline evals, drift checks and local development, not for production serving under real load. Licence MIT, maintained by Georgi Gerganov + community.

More options for Deploy:

TensorRT-LLM — maximum optimisation on NVIDIA Hopper/Ada. Apache 2.0, NVIDIA.
SGLang — good for workloads with structured generation and JSON. Apache 2.0, LMSys.
TGI — a mature alternative, focused on the HuggingFace ecosystem. Apache 2.0, HuggingFace.
NVIDIA Dynamo — multi-node disaggregated serving. Apache 2.0, NVIDIA.
llm-d — K8s operator specific to LLMs. Apache 2.0, CNCF.

Decision matrix — Deploy:

If your case is	Choose
Production serving on NVIDIA H100/A100	vLLM (the safe default)
Absolute throughput squeezing on Hopper	TensorRT-LLM + vLLM plugin or standalone
Edge / local dev without GPU	llama.cpp
Multi-model (LLM + classifiers + encoders)	Triton with the vLLM backend
Declarative K8s with autoscaling	KServe + vLLM as predictor
AI Gateway with token rate limiting	Envoy AI Gateway
Multi-node disaggregated GPU cluster	NVIDIA Dynamo over vLLM

Stage 5 — Observe

Observe propagates trace_id end-to-end, emits runtime metrics, runs an LLM judge over sampling and detects drift. Detailed in tracing with AgentSight, MCP observability with OTel and eBPF + drift.

OpenTelemetry Collector

The OTel Collector is the agent that receives traces, metrics and logs in OTel format (or in any other via receivers), processes them (filters, sampling, attribute enrichment, PII redaction), and routes them to one or several backends (Tempo, Jaeger, Prometheus, Loki, Langfuse…). It is the piece that decouples the apps from the observability backend: moving from Tempo to Jaeger means changing the Collector’s exporter, not the app. For LLMOps it matters especially because the OTel GenAI semantic conventions spec defines the attributes gen_ai.request.model, gen_ai.prompt.version, gen_ai.response.tokens and so on, which stitch the trace_id to the system’s lineage. Gotcha: the Collector configuration tends to grow; without discipline and periodic review it ends up as an 800-line YAML that nobody understands. Modularising with extensions helps. Licence Apache 2.0, maintained by the CNCF / OpenTelemetry Project.

Tempo (traces) + Jaeger

Grafana Tempo is the distributed trace backend optimised for cost: it uses an object store (S3/MinIO) instead of Elasticsearch, deduplicates by trace_id, and integrates natively with Grafana for visualisation. For LLMOps, where a real request generates 10-30 spans (gateway, prompt pull, RAG retrieval, prefill, decode N times, scoring), Tempo copes with high volumes at a reasonable cost. Jaeger is the more established CNCF alternative, better for cases under 100k traces per day, worse for native object store. Gotcha: Tempo has no traditional indexing; searches like “traces that took more than 5s and touched tenant X” require TraceQL + Grafana and are not as fast as Jaeger with Elasticsearch. For immediate ad-hoc diagnosis it is worth keeping a parallel Jaeger with aggressive sampling. Licences AGPL 3.0 (Tempo) and Apache 2.0 (Jaeger), maintained by Grafana Labs and CNCF respectively.

Prometheus + Grafana

Prometheus is the time-series metrics foundation of the ecosystem. Pull model (it scrapes /metrics endpoints), PromQL for queries, exporters for everything (Postgres, Kafka, NVIDIA GPU via dcgm-exporter, vLLM natively). Grafana visualises Prometheus + Tempo + Loki on a single plane. For LLMOps, the critical metrics are gpu_utilization, kv_cache_usage_pct, tokens_per_second, prefill_latency_p95, decode_latency_p95, queue_depth, aggregated by tenant. Gotcha: Prometheus is very good up to about 1M active series; above that it is worth moving to Thanos or Mimir for long retention and horizontal scalability. For the blog’s typical LLM cluster (4-8 H100), Prometheus alone is enough. Licences Apache 2.0 (Prometheus, CNCF) and AGPL 3.0 (Grafana 10+, Grafana Labs).

Langfuse

Langfuse is the LLM-specific OSS observability + prompt management tool. It captures spans with LLM semantic conventions (input, output, model, tokens, latency, score, user_id, session_id), visualises them as conversational traces (not just span trees), manages prompts versioned with a production label and allows curated datasets + evals from the same UI. For serious LLMOps, Langfuse fills the gap that neither Tempo nor Jaeger covers: a tracing UI designed to be LLM-first. Gotcha: Langfuse maintains its own store (Postgres + ClickHouse for high volume); on large clusters the operational side of ClickHouse deserves attention. To get started, Postgres alone copes. Licence MIT for the OSS core, EE Enterprise Edition with additional features (SSO, audit logs, advanced RBAC). Maintained by Langfuse GmbH (Berlin, German). There is Langfuse Cloud (SaaS).

Phoenix Arize OSS

Phoenix is Arize AI’s OSS for LLM observability + evals, an alternative to Langfuse with a different emphasis: more oriented to evaluation and visual debugging (embedding drift, cluster analysis), less to prompt management. A good pairing with Langfuse when you want a dual approach: Langfuse for “production conversational traces”, Phoenix for “exploratory investigation of model behaviour”. Gotcha: Phoenix duplicates functionality with Langfuse and with MLflow; having all three in production multiplies the operational burden. Pick one as the main tool and the others as complements. Licence Elastic License 2.0 (not strictly OSI), maintained by Arize AI.

Cilium Tetragon + Hubble

Tetragon (eBPF runtime security observer) and Hubble (eBPF network observer) are the low-level pieces that give real runtime visibility into the cluster: which processes run in which pods, what syscalls they make, what network connections they open, in real time. For ENS/NIS2 environments that demand “prove what ran in production”, Tetragon is the irrefutable audit layer: every process execution with its parent, its capabilities, its K8s context. Hubble visualises network flows by pod, namespace, service. Gotcha: the volume of events generated is high; without kernel-side filtering (which Tetragon supports with TracingPolicy), it saturates the observability plane fast. Discipline in policies. Licence Apache 2.0 for both, maintained by Cilium / CNCF / Isovalent.

Evidently AI

Evidently is the OSS library for drift detection: it compares input and output distributions between two time windows (training vs production, current week vs previous week), applies statistical tests (KS, PSI, Wasserstein, chi-square) and generates HTML reports. For LLMOps it detects when the distribution of prompts changes (new topics, new lengths, new languages) or when the model starts answering shorter / longer / differently. Gotcha: Evidently is oriented to tabular data and embeddings; for raw text it is best combined with an encoder embedder that produces vectors before applying the tests. Licence Apache 2.0, maintained by Evidently AI (a company). Alternatives: NannyML (Apache 2.0, NannyML BV), Alibi Detect (Apache 2.0, Seldon).

More options for Observe:

Loki — Prometheus-style logs backend for Grafana. AGPL 3.0, Grafana Labs.
Pixie — auto-instrumented eBPF observability. Apache 2.0, CNCF.

Decision matrix — Observe:

If your case is	Choose
Minimum viable stack	OTel Collector + Tempo + Prometheus + Grafana + Langfuse
Traces with strong ad-hoc search	Add Jaeger with aggressive sampling
ENS / NIS2 compliance runtime audit	Tetragon + Hubble + mandated retention
Exploratory investigation of the model	Phoenix Arize OSS alongside Langfuse
Statistical drift detection	Evidently over embeddings + inputs
Cluster beyond 1M Prometheus series	Mimir (Grafana Labs) or Thanos

Stage 6 — Retrain + cross-cutting pieces

Retrain closes the loop feedback → triage → enriched dataset → new adapter. Prompt versioning and data versioning stitch lineage across stages. Detailed in retrain, prompt versioning and data versioning.

Apache Airflow

Airflow is the most established OSS DAG scheduler. You define workflows as Python code (DAGs), each DAG with tasks (operators) that run according to declared dependencies plus a cron schedule. For retraining: a weekly DAG that extracts feedback from Postgres, triages it with an LLM-as-classifier, enriches the enriched dataset in DVC, launches the fine-tuning job on Kubernetes, runs evals against the golden set, and promotes if it passes the gates. There is a huge ecosystem of operators for everything (S3, Postgres, Kafka, Slack, K8s, Spark…). Gotcha: Airflow 2.x improved a great deal over the chaos of 1.x, but the scheduler is still a component that deserves operational attention (Postgres backend, executor pool, sidecar workers); for simple pipelines it is over-engineering. Licence Apache 2.0, maintained by the ASF.

Argo Workflows

Argo Workflows is the K8s-native equivalent of Airflow: each step is a container, the DAGs are defined as K8s YAML, the executor is Kubernetes itself. For environments where everything is K8s, Argo fits without an extra component to maintain. Long tasks (a six-hour fine-tuning run) execute as Pods that survive control plane failures. It integrates trivially with Kubeflow Pipelines (which is built on top). Gotcha: Argo’s YAML syntax is verbose; for complex DAGs, Argo feels less productive than Airflow in Python. Solutions: Hera (Python DSL for Argo, a DataBricks contribution) or Argo + custom CRDs. Licence Apache 2.0, maintained by CNCF.

Kubeflow Pipelines

Kubeflow Pipelines is the layer above Argo Workflows oriented specifically to ML: artefact tracking, experiment tracking, reusable pipeline templates, versioned components. Built on Argo, it adds the ML conceptual model (input artefact, output artefact, metrics) that raw Argo does not have. For cyclical retraining on a K8s cluster, it is the most “ML-ready” option in the OSS ecosystem. Gotcha: Kubeflow as a full suite is heavy (10+ components); many organisations install only Pipelines + Training Operator + Katib and skip Notebook Server / legacy KFServing. Licence Apache 2.0, maintained by CNCF / LF AI & Data.

Feast

Feast is the most widely used OSS feature store. It defines feature views over batch sources (BigQuery, Postgres, Parquet) and online sources (Redis, DynamoDB, Postgres with an extension), exposes a consistent API for read-during-training and read-during-inference (point-in-time correctness), and guarantees that the features of the model in production are the same ones it was trained with. For LLMOps where the model needs consistent user / session / context features (latest plan, tenure as a customer, recent tickets), Feast provides the discipline. Gotcha: for many pure LLM systems (a RAG chatbot with no complex features), Feast is over-engineering, Postgres is enough. When there are real features (recommendation, scoring, ranking), Feast shines. Licence Apache 2.0, maintained by LF AI & Data.

Argilla

Argilla is the OSS annotation + HiL (human-in-the-loop) platform most aligned with modern LLMOps. It creates annotation projects with templates (classification, ranking, span annotation, RLHF preference, free-form text), connects to HuggingFace datasets, and integrates with Langfuse to import traces from production as cases to annotate. It supports multiple annotators with reconciliation, kappa scoring and quality control. To enrich retrain datasets with cases from the “blunt tone” cluster of the Retrain post, Argilla is the front end. Gotcha: Argilla requires Elasticsearch for production performance; for small experiments SQLite is enough. Licence Apache 2.0, maintained by Argilla, Inc. (acquired by Hugging Face in 2024). Alternative: Label Studio (Apache 2.0, HumanSignal), more generalist, less LLM-first.

Langfuse Prompts + MLflow Prompt Registry

Langfuse Prompts manages prompts as versioned entities with labels (production, staging, experiment). The client reads the active prompt from Langfuse on the request path (with a local cache of a few seconds) and propagates prompt_id, prompt_version to the OTel span, exactly as the forensic post does. MLflow Prompt Registry does the same with a slightly different conceptual model (no labels-as-pointers; it uses stages, like the Models registry). Both are valid; the choice depends on which tracking tool is already in place. Gotcha (Langfuse): labels are mutable, so changing production points at another version with no explicit audit; prompts are better deployed via PR against the configs repo, not manually in the UI. Licences and governance covered above.

More options for Retrain + cross-cutting pieces:

Prefect — “modern” Python DAGs, an alternative to Airflow. Apache 2.0, Prefect Tech.
Dagster — DAGs with a strong focus on data assets. Apache 2.0, Dagster Labs.
Label Studio — generalist annotation. Apache 2.0, HumanSignal.
OpenLineage — cross-system lineage event standard. Apache 2.0, LF AI & Data.
DataHub / Apache Atlas / OpenMetadata — catalogue + lineage with a UI. Apache 2.0.

Decision matrix — Retrain + cross-cutting pieces:

If your case is	Choose
Simple pipelines with a catalogue of operators	Airflow
Everything is K8s, minimal components	Argo Workflows
ML pipelines with artefact tracking	Kubeflow Pipelines
HiL annotation for LLM retraining	Argilla + Langfuse integration
Features shared between training and inference	Feast
No complex features, only prompts + LLM	Skip Feast
Lightweight prompt registry	Langfuse Prompts
MLflow is already centralised	MLflow Prompt Registry

Master table: licence, governance and enterprise offering

Tool	Licence	Governance / maintainer	Commercial EE / SaaS
DVC	Apache 2.0	Iterative.ai	DVC Studio
lakeFS	Apache 2.0	Treeverse	lakeFS Cloud
MinIO	AGPL v3	MinIO Inc.	SUBNET / AIStor
Qdrant	Apache 2.0	Qdrant GmbH	Qdrant Cloud
pgvector	PostgreSQL License	Andrew Kane + community	— (built-in Postgres clouds)
PostgreSQL	PostgreSQL License	PostgreSQL Global Dev Group	several managed (Crunchy, Neon, Aiven, EDB)
Apache Kafka	Apache 2.0	ASF	Confluent Cloud
Debezium	Apache 2.0	Red Hat / ASF	Debezium Server / Confluent Connectors
Apache Flink	Apache 2.0	ASF	Ververica Platform, Aiven
HF Transformers	Apache 2.0	Hugging Face SAS	HF Inference Endpoints / Enterprise Hub
PEFT	Apache 2.0	Hugging Face SAS	— (part of the HF offering)
bitsandbytes	MIT	Tim Dettmers + community	—
MLflow	Apache 2.0	LF AI & Data	Databricks MLflow
Axolotl	Apache 2.0	OpenAccess AI Collective	—
Ray (Train)	Apache 2.0	Anyscale + community	Anyscale Platform
DeepSpeed	MIT	Microsoft	—
DeepEval	Apache 2.0	Confident AI	Confident AI SaaS
RAGAS	Apache 2.0	Exploding Gradients	—
Promptfoo	MIT	Promptfoo, Inc.	Promptfoo Cloud
NeMo Guardrails	Apache 2.0	NVIDIA	NeMo Microservices
Presidio	MIT	Microsoft	—
Phoenix (Arize)	Elastic v2	Arize AI	Arize Platform
vLLM	Apache 2.0	vLLM Project / LF AI & Data	several (Red Hat, AWS, IBM, NVIDIA)
TGI	Apache 2.0	Hugging Face SAS	HF Inference Endpoints
SGLang	Apache 2.0	LMSys + community	—
TensorRT-LLM	Apache 2.0	NVIDIA	NVIDIA AI Enterprise
llama.cpp	MIT	Georgi Gerganov + community	—
Triton Inference Server	BSD-3	NVIDIA	NVIDIA AI Enterprise
KServe	Apache 2.0	LF AI & Data (Kubeflow)	—
Envoy AI Gateway	Apache 2.0	CNCF / Tetrate	Tetrate Service Bridge
LiteLLM	MIT	BerriAI	LiteLLM Cloud
OpenTelemetry	Apache 2.0	CNCF	several vendors (Honeycomb, Datadog, Grafana)
Tempo	AGPL 3.0	Grafana Labs	Grafana Cloud Tempo
Jaeger	Apache 2.0	CNCF	—
Prometheus	Apache 2.0	CNCF	Grafana Cloud, AMP, GCP Managed Prom, Azure
Grafana	AGPL 3.0	Grafana Labs	Grafana Cloud, Grafana Enterprise
Loki	AGPL 3.0	Grafana Labs	Grafana Cloud Loki
Langfuse	MIT (core) / EE	Langfuse GmbH	Langfuse Cloud
Tetragon	Apache 2.0	Cilium / CNCF / Isovalent	Isovalent Enterprise
Hubble	Apache 2.0	Cilium / CNCF	Isovalent Enterprise
Evidently AI	Apache 2.0	Evidently AI	Evidently Cloud
Apache Airflow	Apache 2.0	ASF	Astronomer, MWAA, Cloud Composer
Argo Workflows	Apache 2.0	CNCF	—
Kubeflow Pipelines	Apache 2.0	CNCF / LF AI & Data	—
Feast	Apache 2.0	LF AI & Data	Tecton (commercial)
Argilla	Apache 2.0	Hugging Face	HF Hub features
OpenLineage	Apache 2.0	LF AI & Data	—
DataHub	Apache 2.0	Acryl Data	Acryl Cloud

The pattern to watch for when reading the table: AGPL 3.0 and Elastic v2 are the ones that add the most friction in companies with strict licensing policies (legal asks for a specific review). The Apache 2.0 ones pass compliance without argument. Those with an “EE Enterprise” tier or equivalent hide a decision: the OSS version is functionally complete for production, but team features (SSO, audit, advanced RBAC) live in the commercial version. For ENS customers under a HIGH declaration, the EE features (SSO with corporate SAML/OIDC, immutable audit logs) are usually mandatory, so it is worth knowing the price beforehand.

When to move from the “minimum stack” to the “full stack”

The whole catalogue can be intimidating. But you do not build all of it on day one. There is a reasonable order that this blog has been validating in posts throughout the series. The minimum viable stack that serves an LLM API with acceptable discipline:

Serving: vLLM on Kubernetes plus an Envoy AI Gateway in front.
Data: Postgres + pgvector (no Qdrant), MinIO for object store, no Kafka.
Tune: Axolotl + MLflow, no Ray Train.
Eval: Promptfoo in CI, no RAGAS and no judge in production.
Observe: OTel Collector + Prometheus + Grafana + Langfuse, no Phoenix and no Tetragon.
Retrain: feedback in Postgres + crontab scripts, no Airflow.
Versioning: prompts in Langfuse + datasets in DVC over MinIO, no lakeFS.

That is about 8-10 components and it serves a reasonable LLM system for a single tenant with moderate traffic. As the system grows, there are identifiable moments where adding each piece pays off:

Trigger	Component to add
Multi-tenant with isolated corpora	Qdrant (collections per tenant, ACL)
The corpus is refreshed often and periodically breaks	lakeFS (branches with pre-merge hooks)
The embedding pipeline needs streaming	Kafka + Debezium + Flink
Retraining moves from monthly to weekly	Airflow or Argo Workflows
Shared features appear (customer profile, scoring)	Feast
Annotation outgrows informal capacity	Argilla
RAG eval needs specific metrics	RAGAS + Langfuse datasets
ENS compliance demands runtime audit	Tetragon + Hubble
Drift is invisible and shows up late	Evidently
A single stack stops covering multi-model	Triton or KServe with several predictors
Multiple simultaneous multi-tenant adapters	vLLM Production Stack + a dedicated Operator

Each jump adds 1-2 components and is worth the cost only when the trigger is clear. Adding Kafka “just in case” when the corpus is updated once a month is net negative work.

What we have not covered (yet)

Some pieces still deserve a post of their own:

Schema Registry for LLM data and prompts (Confluent OSS, Karapace, JSON Schema Registry).
Catalogue + lineage in depth: DataHub vs Atlas vs OpenMetadata + OpenLineage taken seriously.
Federated learning over OSS (Flower, FedML) for scenarios where the data is not centralised.
OSS MCP Servers and their place in the stack as a tools / actions layer.
“Agentic” evals specific to multi-step systems with tool use.
Upgrade best practices for each component (vLLM every 6 weeks, Kafka major every 18 months, etc.).

References

vLLM · vLLM Production Stack · TGI · SGLang · TensorRT-LLM · llama.cpp — OSS inference engines.
Triton Inference Server · KServe · Envoy AI Gateway · LiteLLM — orchestration and AI gateway.
Qdrant · pgvector · Milvus — vector databases.
DVC · lakeFS · MinIO — versioning and object store.
Apache Kafka · Debezium · Apache Flink — streams and CDC.
Hugging Face Transformers · PEFT · bitsandbytes · Axolotl — fine-tuning.
MLflow · Ray Train · Kubeflow Training Operator — training orchestration.
DeepEval · RAGAS · Promptfoo · NeMo Guardrails · Presidio — evals and guardrails.
OpenTelemetry · OTel GenAI semconv · Tempo · Prometheus · Grafana · Loki — observability foundation.
Langfuse · Phoenix Arize · Evidently AI — LLM observability and drift.
Cilium · Tetragon · Hubble — eBPF runtime.
Apache Airflow · Argo Workflows · Kubeflow Pipelines · Feast · Argilla — orchestration + retrain + annotation.

The parallel catalogue: the six LLMOps stages in open source and on the hyperscalers (AWS, GCP, Azure)

Sat, 23 May 2026 07:00:00 +0200

TL;DR

The previous forensic post used a single request to walk the six stages of the LLMOps pipeline and the two cross-cutting components. This post walks the same stages but crosses them with three extra columns: how each stage is built in open source on-premise, and what the equivalent services are on AWS, GCP and Azure. It is not a migration guide nor a cost benchmark: it is a catalogue of equivalences with its gaps. The general pattern you will see: OSS gives you control, sovereignty and free composition in exchange for expensive operations; the hyperscalers give you integration and time-to-market in exchange for lock-in on margins, data contracts and political dependency. For scenarios subject to ENS / NIS2 with critical customer data, OSS wins by default; for discovery projects where time-to-market is the deciding metric, the hyperscaler wins by default. The interesting part is in the middle. As a concrete thread, at the end we take the multi-tenant chatbot from the previous post and port it to AWS piece by piece to show what disappears, what appears, and where the lock-in materialises.

You are here: the same six stages, but by column

This post shares a map with the previous one, the six stages and the two cross-cutting pieces are all active, but it changes the cut: instead of following a request horizontally, it makes the vertical cut and shows which tools live in each stage depending on the deployment model.

The analogy: your own bakery and the franchise

A baker opens a business. There are two possible models.

He can open his own bakery: rent the premises, buy the oven, choose the flour suppliers, hire his master baker, write his recipes, set the prices, dress the shop window. The day he wants to launch an organic rye sourdough loaf, he asks nobody for permission. The day the price of flour goes up, he finds another supplier. But he pays for all of it: the initial investment, the risk, the daily operations, the months when he misreads the neighbourhood. The bakery is his.

Or he can join a franchise: the franchisor hands over the turnkey premises, the oven with a maintenance contract, the suppliers already negotiated, the operating manuals, the written recipes, the centralised marketing, the loyalty app, the POS system. The learning curve is weeks, not years. But the recipes belong to the franchisor, so do the suppliers, the price of bread is in the catalogue, and the day the croissant formula changes he gets an informational email, not a business decision.

Both bakeries turn out bread. Both meet health regulations and generate income. The operational difference is enormous and it is not about technology: it is about ownership, control and timescale.

The parallel with LLMOps is direct. The on-premise OSS stack is the bakery you own. The managed stack on the hyperscalers is the franchise. The pieces that appear in each stage are functionally equivalent, at the end of the day both solve the same technical problem, but the governance model, the operational cost, the lock-in and the compliance guarantees are different. This post builds the parallel catalogue so that the choice is not made by default.

Quick recap of the previous post

In the forensic post we followed one specific request: a premium-es user of an insurance company asking “How do I cancel my premium subscription?” to the multi-tenant support chatbot of the SaaS provider that hosts it. The route crossed the six stages of the LLMOps pipeline, Data, Tune, Eval, Deploy, Observe, Retrain, plus the two cross-cutting components, prompt versioning and data versioning, over an on-premise infrastructure: RKE2 with Cilium BGP, a 4×H100 SXM cluster, an RTX 4090 for development, vLLM on Kubernetes, Langfuse + OTel + Prometheus + Tempo, Postgres + Qdrant, DVC + lakeFS + MinIO, Kafka and MLflow. The system meets ENS / NIS2 and keeps trace_id propagated end to end.

What comes now is that same system, piece by piece, showing for each box which tool does the job if you are on public cloud, because the integrator’s question is rarely “OSS yes or no?”: it is “what do I lose and what do I gain if I take this box managed?”. And the answer differs box by box.

Stage 1 — Data

The problem. There are three sub-problems the Data stage solves, frequently confused with one another. First, versioning and identity of the corpus and of the training datasets (making sure a dataset_id, dataset_version exists and propagates). Second, storage and serving of the operational corpus (object store + vector index + structured text). Third, streams and ingestion from source systems with CDC, transformation and stable schemas (Schema Registry).

Reference OSS stack. Versioning lives in DVC (pointers in git, content in an object store) combined with lakeFS for branch/merge semantics over data. The post on data versioning goes into the functional difference. The object store is MinIO or Ceph. The vector index is Qdrant or Milvus for large corpora (millions of chunks) and pgvector on Postgres 18 for small cases where having one component fewer pays off operationally. The stream layer is Kafka (pure Apache or Redpanda) with a Schema Registry (Confluent or Karapace OSS), CDC with Debezium or Flink CDC, transformation with Flink or Spark Structured Streaming. The catalogue / lineage is DataHub, Apache Atlas or OpenMetadata with OpenLineage events between systems. The post on PostgreSQL + Qdrant ingestion and the post on RAG over Kafka cover the detailed operations.

Hyperscaler equivalents. On AWS, the corpus lives in S3 (with versioning enabled, which is the cheap substitute for serious data versioning), tabular queries in Athena or Redshift, the vector index in Amazon OpenSearch with the vector plug-in or in Amazon Aurora pgvector. The stream layer is MSK (managed Kafka) or Kinesis Data Streams, CDC with AWS DMS, transformation with Glue Streaming or MSK Connect. The catalogue is AWS Glue Data Catalog + AWS Lake Formation for data governance. And for the RAG case there is also Amazon Bedrock Knowledge Bases, the managed shortcut: you give it S3, it indexes into OpenSearch or Aurora pgvector, exposes a retrieval API and the operations end there, in exchange for paying per indexed chunk and per query.

On GCP, the corpus lives in Cloud Storage (with object versioning), the analytical warehouse is BigQuery (with BigQuery Vector Search already integrated), the dedicated vector store is Vertex AI Vector Search (formerly Matching Engine). The stream layer is Pub/Sub + Dataflow, CDC with Datastream. Catalogue and lineage are Dataplex (which in 2024-2025 absorbed Data Catalog and added automatic lineage). The managed equivalent of Knowledge Bases is Vertex AI Search (formerly Discovery Engine).

On Azure, the corpus lives in ADLS Gen2, tabular queries in Microsoft Fabric / Azure Synapse, the vector index in Azure AI Search (vector mode) or Azure Cosmos DB for PostgreSQL with pgvector. The stream layer is Event Hubs + Stream Analytics or Microsoft Fabric Real-Time Intelligence, CDC with Azure Data Factory. The catalogue is Microsoft Purview, which covers catalogue, lineage and data governance integrated with Entra ID.

Summary table — Data stage.

Functional piece	OSS on-premise	AWS	GCP	Azure
Object store	MinIO, Ceph	S3	Cloud Storage	ADLS Gen2
Dataset versioning	DVC, lakeFS	S3 Versioning (limited), Lake Formation	GCS Versioning, Dataplex	ADLS versioning, Purview
Vector index	Qdrant, Milvus, pgvector	OpenSearch, Aurora pgvector, Bedrock KB	Vertex Vector Search, BigQuery VS	Azure AI Search, Cosmos pgvector
Stream + CDC	Kafka + Debezium + Flink	MSK / Kinesis + DMS + Glue	Pub/Sub + Datastream + Dataflow	Event Hubs + ADF
Schema Registry	Karapace, Confluent OSS	Glue Schema Registry	Pub/Sub schemas	Schema Registry (Event Hubs)
Catalogue + lineage	DataHub, Atlas, OpenLineage	Glue Catalog + Lake Formation	Dataplex	Purview
Managed end-to-end RAG	— (you build it)	Bedrock Knowledge Bases	Vertex AI Search	Azure AI Studio Knowledge

Where the names mislead. S3 Versioning is not DVC. It keeps object versions but has no notion of a dataset (which objects together make up version 3 of the enriched set?), does not propagate dataset_hash to the trainer, does not integrate with experiment tracking, and does not fail a CI run if a dataset breaks schema. Covering that properly on AWS requires combining it with Lake Formation, Glue Data Catalog and your own records in SageMaker Experiments. The same applies on GCP with Dataplex and on Azure with Purview. The gap is real and is paid for in operations or in broken lineage.

Stage 2 — Tune

The problem. Producing a new model_id, model_version, typically a LoRA adapter over a stable base, as the post on continuous fine-tuning explains, with lineage back to the dataset that trained it and experiment tracking that lets you reproduce it six months later.

OSS stack. Technical core: HuggingFace Transformers + PEFT (LoRA, QLoRA), bitsandbytes for quantisation, DeepSpeed or FSDP for parallelism. Experiment tracking: MLflow (authoritative) or Weights & Biases self-hosted. Convenience frameworks: Axolotl and Llama Factory wrap the machinery above in declarative configuration. Distributed orchestration: Kubeflow Training Operator or Ray Train. On small infrastructures, direct scripts with Slurm or K8s Jobs over GPU pools. The lineage chain dataset → run → model is closed by registering the dataset as an MLflow input artefact.

Hyperscaler equivalents. On AWS, SageMaker Training Jobs covers most workloads, SageMaker HyperPod covers large training runs with resilience to node failures, and SageMaker JumpStart offers click-to-train fine-tuning over a catalogue of pre-curated models. For fine-tuning Bedrock models (hosted Claude, Llama, Mistral) there is Bedrock Custom Models: you upload the dataset to S3, Bedrock trains, and it returns a private endpoint with provisioned throughput. The equivalent experiment tracking is SageMaker Experiments or managed MLflow in SageMaker (yes, AWS has officially hosted MLflow since 2024).

On GCP, Vertex AI Custom Training runs any container with GPUs or TPUs; Vertex AI Tuning is the managed API for fine-tuning Gemini and Model Garden models. Experiment tracking in Vertex AI Experiments (with MLflow compatibility).

On Azure, Azure ML Training Jobs on your own clusters or managed compute; Azure OpenAI fine-tuning for fine-tuning GPT and the o-series; Azure ML Experiments with MLflow natively integrated since 2022.

Summary table — Tune stage.

Functional piece	OSS on-premise	AWS	GCP	Azure
Training framework	HF Transformers + PEFT	SageMaker SDK	Vertex AI SDK	Azure ML SDK
Quantisation / parallelism	bitsandbytes, DeepSpeed, FSDP	SageMaker libs + HF support	Vertex + HF support	Azure ML + HF support
Managed fine-tuning (black box)	—	Bedrock Custom Models, JumpStart	Vertex Tuning (Gemini)	Azure OpenAI fine-tuning
Distributed on a cluster	Kubeflow, Ray Train, Slurm	SageMaker HyperPod	Vertex AI Training (multi-node)	Azure ML compute clusters
Experiment tracking	MLflow, W&B self-hosted	SageMaker Experiments, managed MLflow	Vertex Experiments	Azure ML + MLflow
Access to the base model	Whatever you download (Llama, Mistral, Qwen)	Bedrock catalog + HF Hub	Vertex Model Garden + HF Hub	Azure ML model catalog + HF Hub

Where the names mislead. The managed fine-tunings (Bedrock Custom, Vertex Tuning, AOAI fine-tuning) are a black box: you do not choose hyperparameters beyond a handful, you do not see the trainer’s detailed logs, and you cannot inspect the dataset once it is inside their pipeline. The experiment tracking they offer is not comparable to MLflow sitting next to the trainer, where you can capture any metric and artefact. For scenarios where operationally you do not need inspection this is liberating; for ENS / NIS2 scenarios where you have to prove what trained what, the black box fails by construction.

Stage 3 — Eval

The problem. Validating candidates before and after promotion against a golden set, with operational metrics (faithfulness to the RAG, tone, format compliance, toxicity, jailbreak resistance, PII leakage) run as gates in CI and as online sampling. Covered in the post on evals and the one on guardrails.

OSS stack. Eval suites: DeepEval, RAGAS (specialised in RAG), Promptfoo (declarative, ideal for CI), lm-eval-harness (academic), HELM. Evals integrated with tracing: Langfuse Evals, Phoenix Arize OSS. LLM-as-judge judges: any local OSS model; in serious systems, two different judges to reduce bias. Safety and guardrails: NeMo Guardrails (NVIDIA), Guardrails AI, LlamaGuard + PromptGuard (Meta), ShieldGemma (Google, open weights), PII detectors such as Presidio (Microsoft) on-prem.

Hyperscaler equivalents. On AWS, Bedrock Model Evaluation offers automatic evals (toxicity, accuracy, robustness) and human-in-the-loop, Bedrock Guardrails covers the safety layer (denied topics, PII, prompt injection, contextual grounding check), and SageMaker Clarify adds bias and explainability over general models.

On GCP, Vertex AI Evaluation Service runs evals with automatic metrics and an LLM judge, Vertex AI Model Armor and the safety filters built into the Gemini API cover the guardrails layer. Vertex AI Studio exposes interactive Eval for prompt iteration.

On Azure, the Azure AI Evaluation SDK runs offline evals against datasets, Azure AI Content Safety covers safety (Prompt Shields against jailbreak, Groundedness detection, content categories, PII detection). All accessible from Azure AI Foundry.

Summary table — Eval stage.

Functional piece	OSS on-premise	AWS	GCP	Azure
Automatic eval suite	DeepEval, RAGAS, Promptfoo	Bedrock Model Evaluation	Vertex AI Evaluation Service	Azure AI Evaluation SDK
LLM-as-judge	Any OSS model	Bedrock judge models	Vertex judge (Gemini)	Azure OpenAI judges
Golden set management	Langfuse datasets, manual	SageMaker Ground Truth datasets	Vertex Datasets	Azure ML Datasets
Guardrails (jailbreak, PII, prompt injection)	NeMo Guardrails, LlamaGuard, Presidio	Bedrock Guardrails	Vertex Model Armor + Gemini safety	Azure AI Content Safety (Prompt Shields, Groundedness)
Eval in CI	Promptfoo + GitHub Actions	Bedrock Eval API + CodeBuild	Vertex Eval API + Cloud Build	Azure AI Eval + Azure Pipelines

Where the names mislead. Managed guardrails are convenient but opaque: the Bedrock Guardrails rules are configurable but the detection implementation cannot be inspected; the same goes for Azure AI Content Safety. In OSS, NeMo Guardrails shows you the Colang graph and Presidio shows you the recognisers, auditable and modifiable. For regulated systems where an auditor asks “how exactly does it detect PII?”, OSS answers with code; the cloud answers with documentation.

Stage 4 — Deploy

The problem. Serving tokens to the end user with predictable latency and throughput, a decent cost / token ratio, hot-swap adapter support, and multi-tenancy if the business demands it. Covered in the posts on KV cache, PagedAttention, disaggregated serving, vLLM on K8s, LLM operators and multi-tenant cluster.

OSS stack. Inference engine: vLLM (PagedAttention, prefix caching, LoRA hot-swap, OpenAI-compatible API) as the reference, TensorRT-LLM for maximum optimisation on Hopper / Ada, SGLang for workloads with heavy prompt restructuring, TGI (Hugging Face) as a mature alternative, llama.cpp for edge and CPUs, NVIDIA Dynamo for multi-node disaggregated serving on large clusters. Orchestration on Kubernetes: KServe, KubeRay, dedicated operators such as llm-d, vLLM Production Stack and KAITO. Gateway / control plane: Envoy AI Gateway, LiteLLM Proxy, Portkey AI Gateway, Kong AI Gateway. Triton Inference Server covers mixed workloads (LLM + traditional) where a single backend matters.

Hyperscaler equivalents. On AWS, two different routes. The managed-by-model route is Amazon Bedrock: a catalogue of hosted models (Claude, Llama, Mistral, Cohere, Titan), pay per token or Provisioned Throughput with an SLA, Bedrock Prompt Caching as the conceptual equivalent of vLLM’s prefix caching, and integrated Bedrock Agents and Bedrock Knowledge Bases. The managed-by-infrastructure route is SageMaker Endpoints (real-time, async, serverless, batch) with Inference Components to densify multiple models on one instance. Own hardware: AWS Inferentia and Trainium via the Neuron chip, an alternative to NVIDIA with a better cost / token on stable workloads if you compile your model.

On GCP, Vertex AI Prediction Endpoints runs your containers or models from the Model Garden, the Gemini API via Vertex AI offers the managed Gemini models, and Cloud TPU v5e / v5p / Trillium (v6) is the in-house hardware competing with the H100 for training and inference. For sovereignty there is Google Distributed Cloud air-gapped, which brings Vertex AI to a certifiable on-premise rack.

On Azure, Azure OpenAI Service serves OpenAI models (GPT-4.1, o-series, GPT-image), Azure ML Managed Online Endpoints runs any model (OSS included, via container), and Azure AI Foundry models absorbed in 2025 the catalogue of open models served as-a-service. Hardware: Azure ND H100 v5, ND H200 v5, ND GB200 v6 and the in-house bet Microsoft Maia 100 for internal inference.

Summary table — Deploy stage.

Functional piece	OSS on-premise	AWS	GCP	Azure
Inference engine	vLLM, TensorRT-LLM, SGLang, TGI	Bedrock (managed model), SM Endpoints (your container)	Vertex Prediction, Gemini API	Azure OpenAI, Azure ML Endpoints
Prefix / prompt caching	vLLM native	Bedrock Prompt Caching	Vertex AI context caching	Azure OpenAI prompt caching
Adapter hot-swap (LoRA)	vLLM `--enable-lora`, S-LoRA	Bedrock Custom Models endpoints	Vertex Tuning endpoints	Azure OpenAI fine-tuned deployments
Disaggregated serving	NVIDIA Dynamo, vLLM PD-disagg	— (managed internally, not exposed)	— (managed internally, not exposed)	— (managed internally, not exposed)
Accelerator hardware	NVIDIA H100/H200/B200, AMD MI300	Inferentia, Trainium, NVIDIA	TPU v5/v6, NVIDIA	Maia, NVIDIA
AI Gateway / proxy	Envoy AI Gateway, LiteLLM, Portkey, Kong	API Gateway + Bedrock	Vertex AI + Apigee	Azure API Management + AOAI
K8s orchestration	KServe, KubeRay, llm-d, KAITO	EKS + SageMaker Operators	GKE + Vertex AI	AKS + KAITO

Where the names mislead. Bedrock Prompt Caching and Vertex context caching sound equivalent to vLLM’s prefix caching, but operationally they are different: the cache lives on the hyperscaler’s plane, its eviction policy is opaque, its cost is billed separately, and you cannot easily see the hit ratio per tenant. In vLLM you see the hit ratio in Prometheus metrics and you decide the policy. The same applies to disaggregated serving: the hyperscalers implement it internally to cut their own cost of serving, but they do not expose prefill/decode control to the user. If you need your workload’s TTFT to be controlled separately from TPS, that lever is not available.

Stage 5 — Observe

The problem. End-to-end LLM traces with trace_id propagated through every component, runtime metrics per tenant, online scoring (LLM judge over sampling), statistical drift, and safety / guardrails monitoring. Covered in the posts on AgentSight tracing, MCP observability with OTel and eBPF + drift.

OSS stack. Base standard: OpenTelemetry (specification + collector + SDKs) with the gen_ai semantic conventions that stabilised in 2025. Backends: Tempo or Jaeger for traces, Prometheus for metrics, Loki for logs, Grafana as the common UI. LLM-specific layer: Langfuse (self-hosted with an optional EE licence) and Phoenix Arize OSS. eBPF layer for low-level observability: Pixie, Hubble, and Cilium Tetragon for runtime security. Drift: Evidently AI, NannyML, Alibi Detect.

Hyperscaler equivalents. On AWS, CloudWatch (metrics + logs) + AWS X-Ray (traces) form the base, CloudWatch Application Signals adds OTel-compatible APM, and Amazon Managed Prometheus and Amazon Managed Grafana serve the plane if you want to keep Prom + Grafana without operating them. Bedrock logging integrates with CloudWatch and S3. ADOT (AWS Distro for OpenTelemetry) is the official collector.

On GCP, Cloud Monitoring + Cloud Logging + Cloud Trace + Cloud Profiler form the quintet, all OTel compatible. Vertex AI Model Monitoring offers drift detection (feature skew, prediction drift) integrated with runs.

On Azure, Azure Monitor + Application Insights + Log Analytics cover the APM stack with native OTel, Azure ML Model Monitor adds drift and data quality, and Azure OpenAI diagnostic logs enrich the traces with token and model metadata.

Summary table — Observe stage.

Functional piece	OSS on-premise	AWS	GCP	Azure
Traces (OTel)	OTel + Tempo / Jaeger	X-Ray + ADOT, App Signals	Cloud Trace	App Insights + Azure Monitor
Metrics	Prometheus + Grafana	CloudWatch + AMP / AMG	Cloud Monitoring	Azure Monitor Metrics
Logs	Loki, ELK	CloudWatch Logs	Cloud Logging	Log Analytics
LLM-specific (prompt, scores, sessions)	Langfuse, Phoenix Arize OSS	Bedrock logging + CW + custom	Vertex AI tracing + custom	App Insights + AOAI logs + custom
Drift detection	Evidently, NannyML, Alibi Detect	SageMaker Model Monitor	Vertex AI Model Monitoring	Azure ML Model Monitor
eBPF / runtime	Pixie, Hubble, Tetragon	— (no direct equivalent)	GKE Dataplane v2 / Cloud Service Mesh	Azure CNI + Defender for Cloud

Where the names mislead. The classic cloud APM tools (X-Ray, Cloud Trace, App Insights) do not understand prompt versioning or adapter id as native concepts. They accept the gen_ai.* attributes as dimensions, but the UIs do not prioritise those views. Langfuse and Phoenix do, because they are designed for LLM. In cloud, the usual pattern is to send dual: APM to the managed service for infrastructure + self-hosted Langfuse / Phoenix for the LLM plane. That pays off.

Stage 6 — Retrain + cross-cutting pieces

The problem (Retrain). Closing the loop feedback → triage → enriched dataset → new adapter, with a mixed cadence (quarterly + incident-driven). Covered in the Retrain post.

OSS Retrain stack. Orchestration: Apache Airflow, Prefect, Dagster or Argo Workflows and Kubeflow Pipelines for K8s-native. Feature store where applicable: Feast. Annotation and human-in-the-loop: Argilla, Label Studio, Trubrics. Structured feedback capture: your own Postgres table + Langfuse scores + Phoenix annotations. Closed-loop lineage: OpenLineage tying dataset → run → model → deployment → feedback → next dataset.

Hyperscaler Retrain equivalents. On AWS, SageMaker Pipelines orchestrates the cycle, SageMaker Ground Truth and A2I (Augmented AI) manage annotation and HiL, and SageMaker Model Monitor fires alerts that can invoke retrain pipelines. AWS Step Functions serves as a more general alternative orchestrator.

On GCP, Vertex AI Pipelines (based on Kubeflow Pipelines, compatible) orchestrates, Vertex AI Data Labeling Service annotates, Vertex AI Feature Store manages features, and Workflows or Cloud Composer (managed Airflow) act as orchestration alternatives.

On Azure, Azure ML Pipelines orchestrates, Azure ML Data Labeling annotates, Azure ML Feature Store manages features.

The problem (cross-cutting: prompt + data versioning). Making sure prompt_id, prompt_version and dataset_id, dataset_version propagate through the whole system and show up in spans, runs and metrics. Covered in the posts on prompt versioning and data versioning.

Prompt versioning equivalents. OSS: Langfuse Prompts, MLflow Prompt Registry. AWS: Bedrock Prompt Management (catalogue, versions, labels, integrated A/B testing) and SageMaker Prompt Hub. GCP: Vertex AI Prompt Management inside Vertex AI Studio. Azure: Azure AI Foundry Prompt flow and prompt versioning in Azure OpenAI deployments.

Data versioning equivalents. OSS: DVC + lakeFS (already covered under Data). AWS: S3 Versioning + Lake Formation + Glue Catalog (they are not DVC but together they cover part of it). GCP: Cloud Storage versioning + Dataplex (same). Azure: ADLS Gen2 versioning + Purview (same). The real gap here is that no hyperscaler offers DVC natively, so dataset-as-first-class-citizen operations still require a layer of your own.

Summary table — Retrain stage + cross-cutting pieces.

Functional piece	OSS on-premise	AWS	GCP	Azure
ML pipeline orchestration	Airflow, Dagster, Argo, Kubeflow	SageMaker Pipelines, Step Functions	Vertex AI Pipelines, Cloud Composer	Azure ML Pipelines
Feature store	Feast	SageMaker Feature Store	Vertex AI Feature Store	Azure ML Feature Store
Annotation / HiL	Argilla, Label Studio	SageMaker Ground Truth, A2I	Vertex Data Labeling	Azure ML Data Labeling
Feedback capture	Postgres + Langfuse scores	Bedrock + custom + Ground Truth	Vertex + custom	App Insights + custom
Prompt versioning	Langfuse Prompts, MLflow Prompts	Bedrock Prompt Management	Vertex Prompt Management	Azure AI Foundry Prompt flow
Data versioning	DVC + lakeFS + OpenLineage	S3 Versioning + Lake Formation	GCS + Dataplex	ADLS + Purview
Cross-system lineage	OpenLineage + DataHub	SageMaker Lineage Tracking	Dataplex lineage	Purview

The chatbot from the previous post ported to AWS

To stop the catalogue being abstract, we take the full scenario from the previous post, the multi-tenant support chatbot for insurers on an on-premise OSS stack, and describe it component by component with an AWS stack. It is not an executable migration; it is the map of what disappears, what appears and where the lock-in shows up.

The network plane. Edge LB and WAF: AWS WAF + CloudFront. Ingress to the cluster: AWS Load Balancer Controller over EKS. What was Cilium BGP + RKE2 is replaced by EKS with VPC CNI (or Cilium on EKS, which is possible). The conceptual equivalent of Tetragon is Amazon GuardDuty for EKS + optional Falco. Moderate lock-in: network control couples to VPC.

The chat gateway and auth. What was a bespoke API gateway with JWT verification materialises as Amazon API Gateway + Amazon Cognito (or IAM Identity Center if it is B2B). The gateway’s AI-aware routing is covered with Bedrock + per-customer tags, or with AWS API Gateway custom authorizers invoking a Lambda for tenant resolution. High lock-in on the identity layer if Cognito is chosen.

The inference engine. Three different options, with a clear trade-off.

Bedrock with a managed model (Claude / Llama / Mistral): all the operations of vLLM, K8s Operators, KV cache and disaggregated serving disappear. You move to Provisioned Throughput for a latency guarantee. You gain time-to-market; you lose control over prefill/decode, over custom LoRA adapters (Bedrock accepts Bedrock-managed fine-tunes but not arbitrary LoRAs), and you enter model lock-in (moving from Claude to Llama means changing API).
SageMaker Endpoints with your own vLLM container: you keep vLLM and its optimisations, but K8s disappears and SageMaker replaces it as the orchestration plane. Inference Components allow you to densify multiple adapters. KV cache, prefix caching and LoRA hot-swap work the same. Moderate lock-in on the SageMaker SDK and on the Inference Components format.
EKS with vLLM (the minimalist option): basically the OSS stack but with EKS instead of RKE2 and EBS/EFS instead of Ceph. Low lock-in, limited benefit from the cloud.

Data layer. The corpus moves to S3 with versioning, the embeddings to Amazon OpenSearch Service or to Aurora pgvector. The radical managed option is Bedrock Knowledge Bases: you upload documents to S3, it indexes them, it exposes a retrieval API. We remove Qdrant, we remove manual embedding pipelines, we remove part of Kafka + Flink. But control over custom reranking, fine-grained per-chunk ACL and the possibility of re-embedding with a new encoder of your own disappears. Bedrock KB uses the Titan or Cohere embedders available in Bedrock, and changing them means changing the whole index. ENS compliance: you have to validate that the buckets and the index live in EU regions and that the embedding model does too.

Stream + CDC. Kafka + Debezium is replaced by MSK + MSK Connect or by Kinesis + DMS. Schema Registry: Glue Schema Registry. The events remain functionally equivalent. Moderate lock-in if you go to Kinesis (Kinesis is not Kafka), low if you go to MSK (Kafka compatibility).

Data versioning. Here the gap is clear. S3 Versioning + Lake Formation + Glue Catalog is not DVC. To preserve the discipline of the previous post, (dataset_id, dataset_version, sha256_hash) propagated as an input artefact to the trainer, you can keep DVC over S3 (DVC works perfectly with S3 as a remote) or accept the limitation and register lineage manually in SageMaker Lineage Tracking. The first option preserves the practice; the second accepts degradation.

Tune stage. The LoRA adapter customer_support_v7 is trained with SageMaker Training Jobs on ml.p5.48xlarge instances (8× H100), using a standard HuggingFace + PEFT container. MLflow managed by SageMaker or your own MLflow on EC2 covers tracking. Alternative: if the black box is acceptable, Bedrock Custom Models with a dataset in S3 produces a fine-tuned Bedrock model without instantiating a GPU manually, at the cost of not being able to inspect the run.

Eval stage. Promptfoo + RAGAS in CI runs the same over CodeBuild. Bedrock Model Evaluation replaces a good part of the automatic eval suite. Bedrock Guardrails replaces NeMo Guardrails + Presidio + LlamaGuard, with the loss of transparency noted earlier.

Deploy stage. If Bedrock was chosen as the engine, this stage vanishes: Bedrock serves. If SageMaker Endpoints + vLLM was chosen, KServe is replaced by SageMaker Operators (or KServe is kept over EKS). The AI Gateway that in OSS was Envoy AI Gateway or LiteLLM becomes API Gateway + Bedrock or API Gateway + Lambda + SageMaker.

Observe stage. The OTel Collector remains the standard. Traces to AWS X-Ray + CloudWatch Application Signals. Metrics to Amazon Managed Prometheus. Logs to CloudWatch Logs + optionally OpenSearch for search. Langfuse is hosted on ECS Fargate or EKS because the cloud has no native equivalent of integrated prompt + traces + scores. Drift: SageMaker Model Monitor replaces Evidently / NannyML. eBPF (Pixie / Hubble / Tetragon) has no direct equivalent in managed AWS, so Falco or installing Tetragon on EKS is still the route.

Retrain stage. SageMaker Pipelines orchestrates the quarterly cycle. SageMaker Ground Truth + A2I replace Argilla. The feedback_signals table in Postgres stays as it is (RDS Postgres) or moves to DynamoDB for large scales.

How much the lock-in weighs. The component with the highest lock-in is Bedrock + Bedrock Knowledge Bases + Bedrock Guardrails: leaving that requires rewriting the inference plane and reindexing the whole RAG. Next is the SageMaker SDK (Pipelines, Endpoints, Training), where leaving costs but means rewriting scripts, not data. Data in S3 is portable (S3 → MinIO with rclone works). OTel observability is portable at almost no cost if the collector is kept as the abstraction. The auth gateway is the other high lock-in point if it uses Cognito.

What you gain. A dramatic reduction in GPU infrastructure operations, K8s patching, CUDA driver management, prefill/decode sizing, Ceph / MinIO management. A very short start-up curve: one request served in less than a sprint versus several weeks of bring-up for the OSS stack. Explicit provider SLAs.

What you lose. Contractual data sovereignty (the data stays in EU regions if configured that way, but the operator is a US third party under the Cloud Act). Visibility of the whole stack (Bedrock is a black box from the model downwards). Roadmap independence (the decision to discontinue a model, raise prices or change guardrails is not controlled by the customer). Fine-grained cost-per-token optimisation (the levers are the ones the provider exposes). For ENS customers under a HIGH declaration or NIS2 essential category, several of these points are non-compliance, not preference.

Master table: the entire parallel catalogue

Stage / component	OSS on-premise (the blog’s reference)	AWS	GCP	Azure
Data	DVC + lakeFS + MinIO + Qdrant + Kafka + Debezium	S3 + Lake Formation + OpenSearch / Aurora pgvector + MSK + DMS	GCS + Dataplex + Vertex Vector Search + Pub/Sub + Datastream	ADLS Gen2 + Purview + Azure AI Search + Event Hubs + ADF
Data versioning (cross-cutting)	DVC + lakeFS + OpenLineage	S3 Versioning + Lake Formation + Glue Catalog	GCS Versioning + Dataplex lineage	ADLS versioning + Purview
Tune	HF Transformers + PEFT + bitsandbytes + MLflow + Ray/Kubeflow	SageMaker Training + HyperPod + Bedrock Custom + SM Experiments	Vertex AI Training + Vertex Tuning + Vertex Experiments	Azure ML Training + Azure OpenAI fine-tuning + Azure ML + MLflow
Eval	DeepEval + RAGAS + Promptfoo + Langfuse Evals + NeMo Guardrails	Bedrock Model Evaluation + Bedrock Guardrails + SageMaker Clarify	Vertex AI Evaluation Service + Model Armor + Gemini safety	Azure AI Evaluation SDK + Content Safety (Prompt Shields, Groundedness)
Deploy	vLLM + KServe + LLM Operators + Envoy AI Gateway	Bedrock + SageMaker Endpoints (+ Inferentia / Trainium)	Vertex AI Prediction + Gemini API (+ TPU)	Azure OpenAI + Azure ML Endpoints (+ Maia)
Observe	OTel + Tempo + Prometheus + Loki + Langfuse + Phoenix + Hubble	CloudWatch + X-Ray + ADOT + AMP/AMG + SM Model Monitor	Cloud Monitoring + Cloud Trace + Vertex Model Monitoring	Azure Monitor + App Insights + Azure ML Model Monitor
Retrain	Airflow / Argo / Kubeflow Pipelines + Argilla + Feast	SageMaker Pipelines + Ground Truth + A2I + SM Feature Store	Vertex AI Pipelines + Data Labeling + Vertex Feature Store	Azure ML Pipelines + Data Labeling + Azure ML Feature Store
Prompt versioning (cross-cutting)	Langfuse Prompts + MLflow Prompt Registry	Bedrock Prompt Management + SM Prompt Hub	Vertex AI Prompt Management	Azure AI Foundry Prompt flow

When to choose each side — the real decision

The right question is not “OSS or cloud?”. It is stage by stage.

The OSS side wins by default when there is:

Data subject to ENS HIGH category, NIS2 essential sectors or equivalents (identifiable health data, regulated banking, critical infrastructure). Here provider traceability and the processing contract are not negotiable; using a service whose operator is subject to the Cloud Act, FISA 702 or equivalent compromises the legal basis.
A requirement for auditable inspection of the model, the guardrails and the whole pipeline. If a regulator asks “how exactly does it detect PII?” and an answer that ends in open code is mandatory.
Large volumes with stable workloads. Above a certain threshold of tokens per month, the cost of Bedrock / AOAI / Vertex diverges from the amortised cost of your own GPU cluster. The threshold depends on the workload, but it typically sits between 5 and 50 billion tokens per month for models in the Llama 70B range.
Roadmap independence as a priority. The day the provider discontinues a model or raises the price by 40 %, the organisation has to be able to ignore it.

The hyperscaler side wins by default when there is:

Critical time-to-market, an MVP in weeks. The operational weight of the OSS stack is too much for a project that has not yet proved product-market fit.
A small team with no SREs / MLEs specialised in GPU inference. Operating KServe + vLLM + KV cache + multi-tenancy is not trivial; if the team cannot sustain it, hosting is the way.
Variable workloads / unpredictable spikes. Bedrock on-demand and SageMaker serverless charge for what you use; your own cluster pays for the GPU whether it is busy or not.
A need for specific proprietary models (Claude, GPT-4.1, Gemini Pro) with no acceptable OSS equivalent for the case.

Mixed stages are frequent and reasonable. In practice, a common pattern in 2026 is: data, observe and retrain in self-hosted OSS (lineage and sovereignty), tune in OSS on your own cluster, eval in OSS + managed guardrails depending on the safety profile, deploy managed for proprietary models and self-hosted for open models. The question to ask for each stage is: “if the provider raises prices by 50 % or discontinues a component tomorrow, how much does it cost to move it?”. The parallel catalogue in this post gives the answer for each box.

What we have not covered (yet)

Some pieces still deserve a post of their own:

OpenAI / Anthropic API directly (not through Bedrock or AOAI): another level of managed, another contract.
Serious hybrids: AWS Outposts, GCP Distributed Cloud, Azure Stack HCI / Azure Local, the hyperscaler in your own room.
Per-tenant cost accounting compared OSS vs cloud: how the bill is built and where attribution breaks.
A real migration OSS → cloud or cloud → OSS: steps, scripts, gotchas.
Concrete European sovereignty: GAIA-X, EuroHPC, the European cloud offering (OVHcloud, Scaleway, IONOS, Aruba), compared with the big three for ENS / NIS2 cases.
AWS Inferentia / Trainium, GCP TPU v6 Trillium, Azure Maia: in-house chips and how they change the cost / token calculation.

References

vLLM documentation and vLLM Production Stack — the reference OSS inference engine.
Amazon Bedrock documentation — AWS managed model catalogue, Knowledge Bases, Guardrails and Prompt Management.
Amazon SageMaker AI — training, endpoints, pipelines, model monitoring.
Google Vertex AI documentation — training, prediction, evaluation, model monitoring.
Azure AI Foundry documentation — Microsoft’s unified plane for AI applications.
Azure OpenAI Service documentation — OpenAI models hosted on Azure.
OpenTelemetry GenAI semantic conventions — the standard that stitches observability across the OSS / cloud boundary.
Langfuse documentation and Arize Phoenix — the reference OSS LLM observability tools.
DVC and lakeFS — OSS data versioning.
NeMo Guardrails — OSS safety + dialog policy.
ENS (Esquema Nacional de Seguridad) and NIS2 (Network and Information Security Directive 2) — the compliance frameworks that have the last word in the OSS vs cloud choice for regulated EU customers.

Data versioning for LLMOps: DVC, lakeFS and the reproducible golden dataset challenge

Fri, 22 May 2026 11:00:00 +0200

TL;DR

The Data stage of the six-stage LLMOps pipeline has a silent link on which everything else depends: versioning the datasets with the same discipline the code is versioned with. It is not optional. An LLM system in production consumes at least four distinct types of dataset (training/fine-tuning, RAG corpus, golden eval set, enriched dataset from the Retrain loop) and each has its own demands. Git solves code but fails on data for two technical reasons (size and a useless binary diff) and one operational one (it does not propagate lineage as far as the weights bucket of the trained model). The two dominant OSS tools, DVC and lakeFS, unified in November 2025 under a single organisation with a roadmap oriented towards LLM training and RAG datalakes; they are still complementary projects (file-level vs whole-bucket branching) but now under common governance. The productive pattern the market has consolidated on: identify each artefact with an immutable (dataset_id, version), propagate the pair as far as the experiment tracking (MLflow / W&B), version the dataset’s schema too (not just its content), apply a strict holdout to the golden eval set so you are not measuring memorisation, and maintain bidirectional traceability dataset_version ↔ model_version ↔ deployment ↔ trace_id. Without this, the promise of “we can audit which model answered what” collapses at the first serious incident.

You are here: Data (with a cross-cutting effect on Tune, Eval and Retrain)

This post goes into the detail of the versioning link inside stage 1 · Data. Versioning belongs operationally to Data, but the artefacts it produces travel to Tune (training set), Eval (golden set) and Retrain (enriched dataset). That is why the diagram marks Data as active and shows a cross-cutting band indicating end-to-end lineage.

The master analogy: batch traceability in a serious factory

A serious pharmaceutical factory does not produce without batch traceability. Every box of pills carries a printed batch number; that batch is associated with manufacturing dates, with the specific batches of each raw material used, with the quality tests it passed, and with the technicians who signed off each step. If a patient reports an adverse effect, the factory can rewind within hours: this package → this batch → these raw materials → this shift → this production line → this quality control result. Without that chain, the incident is a permanent mystery.

A serious LLM system works the same way. The “package” is the answer a user saw in production. The “batch” is the combination of model, adapter, prompt, context and configuration that generated it. And the “raw materials” are the datasets: the training set the base model was trained on, the adapter’s fine-tuning dataset, the RAG corpus that feeds retrieval, the golden eval set that validates promotion. If a customer says “what data was the model trained on that answered X to my question Y on 14 March?”, without batch traceability the answer is “we do not know”. And that, with a customer under compliance pressure, kills the contract.

Git versions the recipe (the code). Data versioning versions the ingredients. Without both, there is no auditable factory.

The four artefacts worth versioning (with differentiated demands)

Not all datasets are versioned the same way or at the same frequency. The typical LLM system in production handles four artefacts worth governing separately.

Artefact	What it is	Typical size	New-version frequency	Who consumes it
Training / fine-tuning dataset	Input/output pairs (or conversations) that train the adapter or the model.	10⁴ – 10⁷ examples · 1 – 100 GB	Per Tune experiment	Trainer (Axolotl, TRL, Unsloth)
RAG corpus	Indexed documents that feed retrieval.	10⁵ – 10⁹ chunks · 10 GB – 10 TB	Almost continuous (streaming ingest)	Indexer + vector store
Golden eval set	Curated examples with an expected answer to measure quality.	10² – 10⁴ examples · MB	Per product release	Eval gates in CI
Enriched retrain dataset	Cases where the system failed + the human correction.	Hundreds to thousands per quarter	Per retrain cycle	Next Tune

All four have common requirements (immutable identity, lineage, schema) and relevant differences:

The training set tends to be large, stable per experiment, and the cost of an error is a lost experiment (expensive but bounded).
The RAG corpus is enormous, in continuous change, and its versioning is handled through periodic snapshots of the index (not of the raw text). Usually lakeFS or bucket branches; DVC is not the best fit.
The golden eval set is small but critical: errors here contaminate the whole promotion chain. Here the rigidity of the versioning matters more than anywhere else.
The enriched retrain dataset is incremental by nature: each Retrain cycle contributes a delta over the previous one. The new version does not overwrite; it inherits and adds.

Confusing them, treating the RAG corpus as if it were the training set, or the golden eval as if it were just another dataset, is the source of half the operational problems in data versioning.

Why Git is not enough

The obvious question: if Git already solves code, why does it not solve data too? Three reasons, two technical and one operational.

Reason 1: size. A Git repository with a 50 GB dataset becomes unmanageable. git clone pulls the entire history; git status walks every file; the pack file in .git/objects bloats to up to twice the size of the dataset. Git LFS solves the first part (the binary leaves the pack) but introduces its own complexity without addressing the other two reasons.

Reason 2: useless binary diff. Git assumes text diffs are useful. When a column changes in an 8 GB parquet, the diff is opaque, because the file is binary, compressed and columnar. You cannot do code review on a dataset change the way you do on a function change. You need a semantic diff: how many rows changed, which columns changed, which distribution moved. No native Git gives you that.

Reason 3: lineage that crosses repository boundaries. This is the most important and the most subtle. The training dataset lives in a bucket. The trainer code lives in a Git repo. The trained model is published to a model registry. Inference in production generates traces in an observability system. Connecting dataset_v3 → adapter_v7 → deployment_d2 → trace t_x9 requires propagating identifiers across four different systems, not inside one repo. Git has no opinion on this.

Data versioning tools (DVC, lakeFS, Pachyderm, Quilt) exist because they solve the three problems at once: they hang the data outside the Git repo, they offer some form of semantic diff, and they expose stable identities that can be propagated towards experiment tracking and the model registry.

DVC vs lakeFS before the unification

Until November 2025, the two dominant OSS tools coexisted as complementary approaches.

Axis	DVC	lakeFS
Mental model	“Git for data”	“Branching for the data lake”
Granularity	Individual file	Whole bucket (with per-branch namespacing)
Storage	Remote-agnostic (S3, GCS, Azure, MinIO, SSH)	S3-compatible (S3, MinIO, Ceph)
Workflow	`dvc add` + `dvc push` + `dvc.yaml` pipelines	`lakectl commit` + branches/merges over the bucket
Diff	File hash + external metadata	Object-level diff + commit log
Strong cases	Discrete training datasets, model files, reproducible pipelines	Large RAG corpora, branching a shared data lake, parallel experiments without duplicating data
Git integration	Deep (the `.dvc` files are committed to Git)	Tangential (lakeFS lives alongside)
Who operates it	MLE team	Data engineering team

In practice, many teams used them at the same time: DVC for the discrete datasets that fed an experiment (they fit in a Git repo thanks to the indirection of the .dvc pointers), and lakeFS for the large RAG corpus bucket they wanted to branch without duplicating terabytes.

What changed with the November 2025 acquisition

lakeFS acquired DVC in November 2025. The operational consequence as of May 2026 is modest but relevant:

There is (as yet) no technical merger of the projects. DVC is still DVC and lakeFS is still lakeFS. The current CLIs, formats and workflows have not changed.
An explicit combined roadmap towards LLM training and RAG datalakes. The merged organisation has stated specific priorities: consistent branching between the dataset and the trained model, native integrations with MLflow / W&B / Langfuse, support for the typical LLM formats (jsonl, parquet with embedded tokenisation), and branch-aware vector indexing.
Convergence expected in 2026-2027. The market anticipates a single registry with two operating modes (file-level + bucket-branching) under a unified CLI. As of today, teams still combine both.

The practical reading for 2026: adopt DVC for discrete training/eval datasets and lakeFS for the RAG corpus, but design the lineage so that a future unified registry can absorb both without re-versioning everything. Concretely: use stable identifiers (dataset_id, version, commit_hash) that can be propagated regardless of the tool.

The operational pattern: four-hop lineage

Once you accept that datasets have to be versioned, the question is not “which tool” but “which chain of identifiers connects production to the source data”. The pattern the market has consolidated on has four hops:

(dataset_id, dataset_version)
│ versioned in DVC or lakeFS
▼
(model_id, model_version)
│ registered in MLflow / W&B with the dataset as input
▼
(deployment_id, prompt_version)
│ registered in the model registry + prompt registry
▼
(trace_id)
│ emitted by the inference engine with OTel
▼
answer visible to the user

Each arrow is a metadata write that crosses the boundary between two systems. If a single arrow is missing, the lineage breaks and the promise of auditability evaporates.

A concrete example of the flow, using DVC + MLflow:

# Data stage: version the dataset
dvc add data/finetune_v3.jsonl
git add data/finetune_v3.jsonl.dvc data/.gitignore
git commit -m "data: finetune dataset v3"
dvc push # uploads the binary to the remote (MinIO/S3)

# Tune stage: train while recording lineage
mlflow run train.py \
 -P dataset_id=finetune \
 -P dataset_version=v3 \
 -P dataset_hash=$(dvc get-url data/finetune_v3.jsonl | sha256sum)
# The run records: input dataset + model output

# Eval stage: validate while recording lineage
mlflow run eval.py \
 -P model_id=adapter_customer_v7 \
 -P golden_set_id=customer_support \
 -P golden_set_version=v12

# Deploy stage: the deployment inherits the dataset + golden ids
# Every trace in Observe carries model_version + prompt_version
# which rewind back to dataset_version

The equivalent version with lakeFS over the RAG corpus:

# Branch for the new corpus embeddings
lakectl branch create lakefs://corpus/embed-2026q2 --source main

# Index the corpus on that branch
python index_corpus.py --branch embed-2026q2

# Validate before merging to main
python eval_retrieval.py --branch embed-2026q2 \
 --metric recall@10 --threshold 0.78

# If it passes, merge (changes the corpus that serves production)
lakectl commit lakefs://corpus/embed-2026q2 -m "embed: corpus 2026q2"
lakectl merge lakefs://corpus/embed-2026q2 lakefs://corpus/main

The virtue of the second flow: while the new corpus is being validated, the production system keeps serving from main with no interference. The parallel branch works as a real staging area over the whole bucket.

Schema contracts: data versioning without them is an illusion

Versioning a dataset’s content without versioning its schema is a frequent mistake. The problem: a versioned dataset with an implicit schema still breaks silently when a producer (the ingestion team, the annotation team, an ad-hoc script) changes a field.

A concrete case: a customer support golden eval set, 1000 examples, an expected_output field originally string. Someone decides they need to capture several valid answers and changes the field to list[string]. The eval loader accepts both formats by chance (Python is lax) but the downstream LLM judge receives a different object. The eval keeps passing but now it measures something else.

Productive pattern: the dataset is versioned with DVC/lakeFS and its schema is versioned with a Schema Registry (Confluent or Apicurio) or, in less mature systems, with a JSON Schema embedded alongside the dataset. CI blocks any PR that breaks the contract without a version bump.

A minimal schema for a golden eval entry (illustrative):

$schema: https://json-schema.org/draft/2020-12/schema
$id: https://example.org/schemas/golden_eval_entry/v3.json
type: object
required: [example_id, input, expected_outputs, rubric, segment]
properties:
 example_id: {type: string, format: uuid}
 input:
 type: object
 required: [user_query, retrieved_context]
 properties:
 user_query: {type: string}
 retrieved_context: {type: array, items: {type: string}}
 expected_outputs:
 type: array
 minItems: 1
 items: {type: string}
 rubric:
 type: object
 required: [must_include, must_not_include, format]
 properties:
 must_include: {type: array, items: {type: string}}
 must_not_include: {type: array, items: {type: string}}
 format: {enum: [text, json, markdown]}
 segment: {type: string}
 difficulty: {enum: [easy, medium, hard]}
 added_at: {type: string, format: date-time}
 curated_by: {type: string}

Operational rules:

Explicit forward/backward compatibility: adding an optional field is backward-compatible; removing a required one is breaking. The policy is enforced with a compatibility check in CI.
The schema version embedded in every row of the dataset (a _schema_version field). The loader validates that the version matches what the consuming code expects.
The schema registry as the single source of truth, not as an optional copy of the JSON Schema in four repos.

Without this level of discipline, “we have data versioning” means “we store the bytes but we do not control what they mean”.

Golden eval set: the most critical version

Of the four artefacts, the golden eval set is the one that demands the most rigour. A failure here contaminates the whole promotion chain: if the eval lies, the gates approve models that should not pass.

Three extra disciplines over the golden set:

Annotation with measured quality. Each example is labelled by a human, and a percentage (10-20 %) is annotated by two independent people. The inter-annotator agreement (Cohen’s kappa or pairwise F1) is measured and published; a golden set with kappa < 0.7 is measuring human noise, not model behaviour. Argilla and Label Studio provide the mechanics; what matters is the discipline, not the tool.

A strict holdout against contamination. The golden set must never enter the training set. The concrete mechanism: hash each golden set input (sha256 normalised by lowercasing + stripping trivial punctuation) → check in CI against all the training set hashes. If there is an intersection, CI blocks until it is resolved. Without this check, the model passes the eval by memorisation, not by capability. The consequence in production is disastrous: the “validated” model fails on cases analogous to the golden set that it had not memorised.

Additive versioning, never destructive. When the golden set grows (each retrain cycle adds cases), golden_v3 = golden_v2 ∪ new_examples. Never golden_v3 = a different new set. Only that way can you compare two models trained months apart over the same base + the new delta. If you rewrite the golden set, you cannot tell whether the March model was worse than the May one or whether you were simply measuring different things.

Summary table of the discipline per artefact:

Practice	Training set	RAG corpus	Golden eval set	Enriched retrain
Immutable versioning	Yes	Yes (snapshots)	Yes, critical	Yes
Schema with a contract	Yes	Recommended	Yes, critical	Yes
Double annotation	No	Not applicable	Yes (10-20 %)	Yes (10-20 %)
Holdout vs other datasets	N/A	N/A	Yes, hash check	Yes (vs golden)
Drift check vs previous version	Recommended	Yes	Recommended	Yes
Lineage to deployment	Yes	Yes	Yes	Yes

Promotion gates: the dataset is promoted like the model

A candidate dataset (a freshly enriched golden_v13, an enriched_retrain_2026_q2 resulting from the Retrain cycle) does not enter production just by being in the bucket. It goes through gates equivalent to the model’s or the prompt’s:

Schema validation — the contract is met. Blocked in CI if not.
Quality validation — a random 5-10 % sample reviewed by a human with a quality score ≥ 4/5. Blocked if the sample fails.
Holdout segregation check — for golden sets and enriched datasets, a hash check against all the other active datasets. Blocked if there is overlap.
Drift check vs previous version — a KS test over the distribution of the input embeddings, or simpler metrics (mean length, segment distribution, the ratio of each label). A warning if the drift is high with no documented cause; a block if it is very high.
Lineage check — the dataset explicitly declares which version it inherits from and what changed. Without that metadata, it does not go in.

Only when all five gates pass is the dataset tagged production-ready and the downstream pipelines that depend on it unblocked (the next Tune, the next product release, the next eval cycle).

The on-premise stack applied

On a generic infrastructure with an RTX 4090 (24 GB VRAM, a development / small-batch profile) and a 4×H100 SXM cluster (80 GB VRAM each, NVLink, training and production inference), data versioning fits without a dedicated GPU for the versioning itself, since versioning lives on CPU + storage, but it does touch the GPU for the drift checks that require embeddings.

Typical topology:

┌────────────────────────────────────────────────────────────┐
│ Object store (MinIO or Ceph) │
│ buckets: /training-sets /corpus-rag │
│ /golden-evals /enriched-retrain │
└────────────────────────┬───────────────────────────────────┘
│
┌─────────────────┼──────────────────┐
│ │ │
┌───▼────┐ ┌────▼────┐ ┌────▼─────┐
│ DVC │ │ lakeFS │ │ MLflow │
│ remote │ │ branches│ │ Tracking │
└───┬────┘ └────┬────┘ └────┬─────┘
│ │ │
└─────────────────┴──────────────────┘
│
┌──────▼──────┐
│ CI/CD gates │
│ (Forgejo / │
│ GitLab) │
└──────┬──────┘
│
┌──────────┴───────────┐
│ │
┌─────▼──────┐ ┌─────▼─────┐
│ RTX 4090 │ │ 4×H100 │
│ (drift │ │ (training │
│ embeds, │ │ + │
│ validates)│ │ serving) │
└────────────┘ └───────────┘

Operational notes:

The object store (MinIO or Ceph) serves both as the DVC remote and as lakeFS storage. One storage plane, two views.
The schema checks and holdout hashing are fast CPU-bound tasks; the CI runner executes them without a GPU.
The embedding drift check needs an encoder; the RTX 4090 handles this without touching the production cluster. A small encoder (BGE-small, E5-small, ~100M parameters) processes 10⁴ examples in a few minutes.
The H100 cluster stays free for training and serving, with no contamination from versioning jobs.

When do you NOT need DVC/lakeFS?

There is an opposite position defended with figures in the continuous fine-tuning post: for small systems with a single team, datasets < 1 GB and a handful of adapters, Postgres + pgvector + an S3 bucket + hashed filenames are enough. The operational complexity of DVC/lakeFS does not pay for itself.

The dividing line is reasonable:

You do not need DVC/lakeFS: a single team, small datasets, few adapters, no multiple products sharing data.
You do need it: multiple teams, datasets > 10 GB, several products sharing a golden eval set, external compliance demanding batch traceability, or an institutionalised quarterly retrain cycle.

Adopting DVC + lakeFS before you need them is overhead. Adopting them six months late is losing six months of lineage irrecoverably.

Seven pitfalls that turn data versioning into theatre

Versioning the data but not the schemas. The content is versioned, the contract changes silently, the system breaks without the versioning capturing it. A Schema Registry is not optional; it is half the problem.
The same S3 path overwritten. “Upload training.jsonl to the bucket” and the next experiment rewrites the file. S3 versioning (if it is enabled) saves your skin, but without an immutable identifier propagated to MLflow you cannot rewind. The correct pattern: training_v3.jsonl or training/2026q2/<sha>.jsonl, never the same name.
A golden eval set without a strict holdout. Without a hash check against training, the model memorises the eval and passes without having learned. It is the LLM equivalent of an exam the teacher announces in advance: everyone passes, nothing has been measured.
Not recording dataset → model lineage. When an incident requires knowing what data a certain model was trained on, the correct answer is a query to MLflow / W&B. If the answer is “let’s ask whoever trained it” (assuming they are still on the team), the lineage does not exist.
DVC added six months late. Adopting versioning in month 1 = a nuisance. Adopting it in month 6 = an irrecoverable loss of six months of datasets that can no longer be reconstructed. The curse of “we’ll add it later”.
lakeFS with branches that never get merged. Parallel branches over the corpus are useful for experimenting; kept indefinitely without merging, operations turn into a graveyard of half-updated branches. Explicit policy: merge or destroy within N weeks.
Schema validation only in production. The contract is validated when the dataset is already in production and the model already trained. By then, the incident has already happened. The validation has to be in CI, before the merge, over the delta the PR introduces.

A dataset’s cycle on one screen

┌─────────────────────────────────────────────────────────────┐
│ Producer (ingest / annotation / retrain loop) │
└────────────────┬────────────────────────────────────────────┘
│
▼ (commit a candidate version)
┌─────────────────────────┐
│ CI gates │
│ - Schema validation │
│ - Quality sampled │
│ - Holdout hash check │ ── fails → PR blocked
│ - Drift vs previous │
│ - Lineage declared │
└────────────┬────────────┘
│ passes
▼
┌─────────────────────────┐
│ DVC tag or lakeFS commit│
│ + MLflow registry │ ← immutable version
│ + Schema Registry │
└────────────┬────────────┘
│
▼
┌─────────────────────────┐
│ Downstream pipeline │
│ Tune / Eval / Deploy │
└────────────┬────────────┘
│
▼
┌─────────────────────────┐
│ Production trace │
│ → rewinds to dataset │
└─────────────────────────┘

What we have not covered

At top level, this post leaves out:

Vector store versioning proper: an embedding index is not versioned like a raw dataset because it depends on the embedding model. Changing the embedder rewrites the whole index. It is a different animal and deserves separate treatment (recall, ANN parameters, index branching vs full re-embedding).
Standardised lineage tooling (OpenLineage, Marquez): how to emit and consume lineage events interoperably between systems.
Data quality frameworks (Great Expectations, Soda, Deequ): how to write suites of “expectations” over a dataset and enforce them on every version.
Privacy-preserving versioning: federated learning without centralising the dataset, differential privacy applied to the version that gets distributed.
Contamination between third-party golden sets (HumanEval, MMLU, etc.) and the training datasets of open models: the problem of “the model passes HumanEval because HumanEval is in its pretraining”.

Each one is worth a post of its own when the field justifies it.

References

DVC documentation — versioning workflows, pipelines and remotes.
lakeFS documentation — branching, merging and commits over the bucket.
lakeFS acquires DVC, November 2025 — announcement and combined roadmap.
Confluent Schema Registry and Apicurio — schema contracts for streaming data.
OpenLineage and Marquez — open standard for lineage events.
Great Expectations — data quality expectations in CI.
MLflow Tracking — input datasets as first-class artefacts since MLflow 2.4.
Pachyderm and Quilt — historical alternatives to DVC/lakeFS.
On eval set contamination: “Stop Uploading Test Data in Plain Text” (Magar & Schwartz, 2022) and later work on detecting contamination in pretraining corpora.

Retrain: closing the loop between the production incident and the adapter that fixes it

Fri, 22 May 2026 07:45:00 +0200

TL;DR

The Retrain stage of the six-stage LLMOps pipeline is the one that closes the cycle. Without it, the deployed system is a project that ends; with it, it is a living practice that improves every quarter. The mechanics at the top level fit into five sequential sub-processes: capture feedback (explicit via thumbs + implicit via latency, abandonment, retries), triage incidents by root cause (model issue, retrieval issue, prompt issue, infra issue), enrich the dataset with the cases where the system failed plus the correct answer labelled by a human, decide the cadence (quarterly scheduled by default + incident-driven when a pattern crosses a threshold), and promote the candidate by taking it through Tune → Eval → Deploy with gates against the model in production. The tools the market has consolidated on in 2026: Langfuse for feedback collection in the UI, Argilla and Label Studio for human annotation of the enriched dataset, MLflow stages for promotion. The most lethal trap, and the most common, is the open loop: having all the pieces but no structured channel connecting them, so the Retrain stage boils down to “we will retrain when we need to” and therefore never.

You are here: Retrain (closes the cycle back to Data)

This post goes into the detail of stage 6 of the LLMOps pipeline. What follows takes apart the five Retrain sub-processes at complete top level, without descending into the internal mechanics of Tune (covered in the continuous fine-tuning post) or into the implementation of the eval suites (covered in the evals post).

The master analogy: the hospital mortality committee

A serious hospital holds periodic morbidity & mortality (M&M) meetings: the doctors review, without blame but without omitting anything, the cases where a patient died or had a serious complication. They look for root cause, identify patterns, adjust protocols, and leave a record. The committee does not meet when someone “remembers”; it is scheduled and mandatory. And when there is a catastrophic incident outside the cycle, an extraordinary M&M is convened within 48 h.

The Retrain stage is exactly that for an LLM system:

The morbidity is the minor incidents: answers the user marked with a thumbs-down, sessions where they retried the same question three times, examples where the eval score dropped but not below the alert threshold.
The mortality is the serious incidents: the system gave a dangerous answer, a key customer cancelled over a string of errors, the agent executed a tool it should not have.
The periodic meetings are the quarterly scheduled retrain: you look at the accumulated feedback, prioritise, and decide what goes into the enriched dataset for the next training run.
The extraordinary M&Ms are the incident-driven retrains: faced with a problematic pattern that crosses a threshold, a mini-cycle is fired off outside the cadence.

Without this discipline, incidents are anecdotes that get forgotten and the system does not learn.

Sub-process 1 — Feedback capture

The first link in the loop is observing what the system does badly. There are two families of feedback, and they complement each other.

Explicit feedback

The user tells you directly that the answer was bad. Mechanisms:

Thumbs up/down in the UI: the classic, low latency (1 click). Coverage: typically 1-5 % of traffic. Bias: users vote more when they are annoyed than when they are happy.
Annotation by power users: internal customers or experts who leave structured comments (“the answer is correct but the format does not follow our style guide”). Far lower coverage but high quality.
“What went wrong?” forms when the thumbs-down is clicked: predefined options (hallucination, format, tone, incomplete, off-topic) + optional free text. Enables automated triaging.
Re-editing: if the system writes a draft (an email, some code) and the user edits it before sending, that edit is rich feedback. The diff between what was generated and what was sent is an explicit signal of the failure.

All explicit feedback travels labelled with trace_id, prompt_version, model, user_id (anonymised where required), timestamp, and enters the feedback store. Langfuse, Phoenix and LangSmith have built-in UI for this; what matters is that every thumbs-down materialises as a row in a table, not as an event that gets lost.

Implicit feedback

The user tells you nothing but their behaviour gives the problem away. Typical signals:

Anomalous latency: the system’s TTFT was 8 s when the average is 800 ms. Indicates overload, heavy retrieval, an unexpectedly large prefill. Covered at top level in evals and ebpf+drift.
Abandonment rate: the user leaves the session before reading the full answer. If the ratio goes from 5 % to 15 % in one segment, something is wrong.
User retries: the user asks the same question (or a very similar one) 2-3 times. Indicates the first answer was no use to them.
Aborted sessions: the user closes the chat before the model finishes generating. In streaming, a high abort ratio is a strong indicator.
Workflow exit: in an agent, the user cancels the plan before execution. The agent’s trajectory did not convince them.
Statistical drift in the distribution of inputs or outputs (KS test, PSI, embedding-space shift). Covered at top level in eBPF + drift.

Implicit signals are noisier but cover 100 % of traffic, not the 1-5 % of explicit feedback. Combining them with explicit feedback gives the full picture.

Typical storage pattern

All the feedback, explicit and implicit, ends up in a common table with a minimal schema:

CREATE TABLE feedback_signals (
 signal_id UUID PRIMARY KEY,
 trace_id UUID NOT NULL,
 request_id UUID NOT NULL,
 signal_type VARCHAR NOT NULL, -- 'thumbs', 'retry', 'abandon', 'drift', ...
 signal_value JSONB, -- feedback payload (thumbs-down text, latency, etc.)
 prompt_id VARCHAR,
 prompt_version INT,
 model VARCHAR,
 user_segment VARCHAR, -- tenant, plan, geo
 occurred_at TIMESTAMPTZ NOT NULL,
 triaged BOOLEAN DEFAULT FALSE,
 triage_label VARCHAR -- filled in by sub-process 2
);

Postgres is more than enough for reasonable volumes (millions of rows a month). Langfuse uses Postgres underneath. For high volumes you can divert to ClickHouse or BigQuery, but it is rarely worth the extra complexity.

Sub-process 2 — Triage by root cause

Having feedback is not enough. You have to categorise each incident by its root cause before deciding what to do with it. Without triage, the enriched dataset is a junk drawer and the next retrain fixes nothing in particular.

The four canonical categories:

Category	Means	Typical action
Model issue	The model answered badly to something that was within its theoretical capability.	Candidate case for the enriched dataset for the next Tune.
Retrieval issue	The RAG did not retrieve the right context. The model answered reasonably from poor context.	Adjust reranker, chunking, indexing — the Data stage, not Tune.
Prompt issue	The system prompt does not cover the case, or covers it badly.	New version of the prompt (the cross-cutting stage of prompt versioning).
Infra issue	Latency, timeout, 5xx error, overload.	Adjust capacity / autoscaler — the Deploy stage.

Triage can be done:

Manually: a human (typically: the MLE team / data scientist) reviews the feedback in the Langfuse / Phoenix / LangSmith UI, looks at the complete trace, labels it. Cost: 2-5 min per incident. Sustainable up to around 50-100 incidents/week per person.
Assisted by LLM-as-classifier: an LLM classifies the incident into one of the four categories with a structured prompt. 80-90 % coverage automated, the rest escalated to a human. State of the art in 2026: GPT-5, Claude 4, Llama 3 70B-instruct with a careful prompt give F1 > 0.85 against calibrated internal rubrics.
Heuristic rules for the obvious ones: a 5xx error is always infra; latency > 5σ is always infra; a thumbs-down over RAG with context_relevance < 0.3 is retrieval. Captures 30-50 % of the volume at zero cost.

The productive pattern is: rules → LLM classifier → human, cascaded, escalating only what the previous level does not resolve with confidence.

New feedback
│
▼
[heuristic rules]
│
├── high confidence → automatic label
│
▼ (rest)
[LLM-as-classifier]
│
├── high confidence → suggested label
│
▼ (rest, or disagreement with the rules)
[human review]
│
└── final label → feedback_signals.triage_label

Sub-process 3 — Dataset enrichment

Once the incidents are triaged with a model issue label, those cases are candidates to enter the enriched dataset that will feed the next Tune. But they do not go in as they are: you need the correct answer labelled by a human.

How an enriched case is built

Each enriched case is a minimal tuple:

case_id: enrich-2026-05-22-0142
source_trace_id: trace-xyz
prompt_input:
 system: "You are a support assistant..."
 user: "I cancelled my order on Tuesday but I can still see the charge"
prompt_version_at_failure: customer_support_v3@v2
model_at_failure: llama-3-70b-instruct
failure_response: "Sorry for the inconvenience. The charge should be reversed within 5-7 business days."
human_corrected_response: "Sorry for the inconvenience. I have checked your account and I can see the refund was processed on Wednesday. It will show in your account within a further 24-48 h depending on your bank. Here is the refund ID: ABC123."
labeler: "support_agent_M3"
labeled_at: "2026-05-22T09:30:00Z"
quality_score: 4 # 1-5, eval by a second human before promoting to the dataset
notes: "The model gave a generic answer without checking the real state of the refund. It needs the account_lookup tool."

What matters is that the enriched case has enough context to be reproduced: original prompt, prompt version, model, failed answer, correct answer. Without this, the case is a loose data point, useless for training.

Annotation tools

Three dominant options in 2026:

Argilla (OSS, maintained by Hugging Face since 2024). Designed specifically for LLM datasets: annotation of (input, output) pairs, preference data (DPO/RLHF), instruction tuning. Python-friendly UI. Native integration with HuggingFace datasets and with MLflow.

Label Studio (OSS from Heartex). More generalist, also works for LLM. Rich, configurable, multi-modal UI. Better when the team already uses it for other tasks.

Langfuse built-in UI. Lets you annotate existing traces directly with thumbs + text + categorical labels. Useful for lightweight feedback; for building serious preference or instruction tuning datasets, Argilla and Label Studio are a better fit.

Typical pattern: Langfuse for traffic feedback + Argilla to build the formal enriched dataset that goes into the Tune pipeline. The traces marked as candidates in Langfuse are exported periodically to Argilla, where a human produces the correct answer and validates quality.

Quality validation before promoting

Not every annotated case enters the dataset. Good discipline requires:

Double annotation on at least 10-20 % of the critical cases (two independent annotators; if they disagree, a third resolves it).
Quality score per case (1-5 or equivalent): only cases with a score ≥ 4 enter the dataset.
Dataset versioning with DVC + lakeFS or equivalent, just like the rest of the Data stage datasets.
Reserved holdout: a portion of the enriched dataset is set aside to evaluate the retrained adapter, without it entering training. If the dataset is enriched with cases where the model failed and the same dataset is used to evaluate, you are measuring memorisation, not learning.

Sub-process 4 — Cadences: scheduled vs incident-driven

Once enriched dataset accumulates, it remains to decide when the retrain is launched. There are two complementary cadences.

Scheduled retrain (quarterly by default)

A process fixed in the calendar. Every quarter, in a specific week, the team:

Closes the accumulated feedback capture cycle.
Counts the enriched cases available (typically tens to hundreds per quarter).
Launches the fine-tuning pipeline with the aggregated dataset (golden dataset + the quarter’s enriched cases).
Evaluates the candidate against the full suite + the enriched holdout.
Promotes it if it passes the eval gates.

Advantages: predictable capacity planning, a closed budget, controlled risk, a team that is not burnt out. The default.

Incident-driven retrain

When a serious incident crosses a threshold, a mini-cycle fires off outside the cadence. Typical triggers:

Drift detected in the input/output distribution over threshold (KS p-value < 0.01, PSI > 0.25, embedding-space shift > 2σ).
A failing segment: a cluster of users or a type of question shows a 3× error rate over baseline for > 48 h.
A successful prompt injection or jailbreak attack that crosses the severity threshold (covered in guardrails).
External domain change: the customer changes policy, a new regulation comes out, and so on. The model trained against the old version stops being valid.

A typical mini-cycle: feedback from the last 7-14 days, a dataset focused on the problematic segment, fast fine-tuning on top of the existing adapter (not a full retrain), a segment-specific eval gate, canary deployment, promotion if it passes.

Cost: around 3-7 days of team work depending on severity. It is not optional for critical cases: if the failing segment is regulatory or reputational, the cost of not responding quickly is far higher than that of the mini-cycle.

Anti-pattern: “we will retrain when we need to”

The most lethal phrase in LLMOps. Without explicit scheduling, the scheduled retrain never arrives; without explicit thresholds, the incident-driven one never fires either. The system accumulates silent debt until a catastrophic incident forces a retrain that is already late.

The minimum discipline: a date in the calendar for the next scheduled run + 3-5 explicit incident-driven thresholds in writing. Without this, the Retrain stage is theatre.

Sub-process 5 — Promotion: the candidate enters production

Once the candidate adapter exists, it does not go straight into production. It goes through the same flow as any release: Tune → Eval → Deploy with gates.

Candidate adapter (from Tune)
│
▼
[full eval suite]
- historical golden dataset
- the quarter's enriched holdout
- regression vs production
│
passes? → no → block + alert
│
yes
▼
[non-regression eval gate]
- make sure it does not degrade
segments that already worked
│
passes? → no → block + alert
│
yes
▼
[canary deployment]
- 5-10% of traffic to the new
adapter for 24-72 h
- online metrics vs production
│
metrics OK? → no → rollback
│
yes
▼
[full promotion]
- move the label in the model registry
- MLflow stages: Staging → Production
- The previous one goes to Archived (preserves
historical reproducibility)

The registry tools:

MLflow Model Registry stages (Staging, Production, Archived) is the canonical pattern. Promotion is an API call: mlflow.models.transition_stage(name, version, "Production"). Audited, revertible.
A private Hugging Face Hub with a repo per adapter is the “Git for models” equivalent: versioned by commit hash, branches for staging/production, deploy via PR.
vLLM multi-LoRA hot-swap (described in continuous fine-tuning) loads the new adapter without restarting the server, so the physical promotion takes seconds.

Applied to typical on-premise hardware

Retrain as a stage does not need big hardware. The calculation:

Feedback collection: a small table in Postgres. Trivial on any node.
Manual / assisted triage: the LLM-as-classifier runs on the same inference engine that serves production, during low-demand hours, at spot priority. Tens of thousands of incidents a month consume on the order of minutes of GPU per day.
Dataset enrichment: human annotation, no GPU cost. Negligible storage.
Tune (mini-cycle or quarterly): here there is a cost. Fine-tuning a LoRA adapter over Llama 3 70B with a dataset of a few thousand examples costs on the order of 2-8 hours on a single H100. On 4 H100s with tensor parallel: 30-90 min. It fits comfortably into any low-demand overnight window.
Full eval suite: minutes on an engine with prefix caching active (covered in pagedattention deep-dive).
Canary deployment: zero additional cost, since the new adapter coexists in the same engine via multi-LoRA hot-swap.

For an RTX 4090 serving Llama 3 8B with a small team: a monthly or quarterly scheduled retrain overnight, an enriched dataset of 50-100 cases per cycle, annotation with self-hosted Argilla on the same node. Plenty.

For a 4×H100 SXM cluster serving several tenants: an enriched dataset segregated by tenant (each with its own holdout and eval suite), a retrain pipeline orchestrated with Argo Workflows or equivalent, a centralised MLflow registry, multi-LoRA hot-swap per tenant.

Common operational traps

The open loop. The system captures feedback, saves it to a table, and there it dies. Nobody triages, nobody enriches, nobody retrains. The deployed model ages silently. Fix: an explicit internal SLO (for example, “any feedback more than a week old without triage gets reported in standup”), with an assigned owner.

Human feedback that gets lost. Thumbs-downs with no structured capture (the event is logged but the reason is not), or the reason is logged but nobody indexes it for queries. Fix: an explicit schema like the one above, a weekly dashboard of “top thumbs-down reasons”.

Undefined cadence. “We will retrain when we need to” — never. Fix: a date in the calendar + 3-5 written thresholds.

No holdout test set. The enriched dataset is mixed with the golden dataset both to train AND to evaluate. The adapter looks like it improved because it “memorised” the enriched cases, but it generalises badly to new similar cases. Fix: a holdout reserved before training; the eval against the holdout is the metric that decides promotion.

Ad-hoc triage by one person. The senior data scientist triages when they can; while they are on holiday it piles up; they come back and give up because there are 400 incidents waiting. Fix: automate 70-80 % with an LLM-as-classifier, leave only the hard part to a human; rotate the “triage oncall” so one person does not get saturated.

Promotion without canary. The adapter passes the offline eval and is deployed to 100 % directly. A regression in production takes a while to be detected, until the online metrics show it, and by then the damage is done. Fix: a mandatory 5-10 % canary for 24-72 h.

No reproducibility of the original incident. The team goes to investigate why the model failed in the 22 May incident and discovers the prompt was different (it was changed two weeks ago), the model was too, and the logs did not keep the RAG context. Fix: strong traceability (covered in prompt versioning and MCP observability). Without reproducibility, retrain is guesswork.

The enriched dataset contaminates the Eval data. The team confuses “cases where it failed” (which go into the enriched training set) with “the golden regression dataset” (which has to stay stable to detect drift). Mixing them invalidates the eval. Fix: two different datasets, two different routes.

Recommended operational pattern: the quarterly cycle on one screen

A serious team with Retrain properly implemented has this flow every 3 months:

Week 1 (cycle close): new capture is frozen for the cycle, a snapshot is taken of the accumulated feedback. Automated report: how many thumbs-downs, how many incidents triaged, the distribution by category, top patterns.

Week 2 (triage and annotation): the MLE team plus annotators process the untriaged model issue cases. Human annotation in Argilla. Cross-validation on samples.

Week 3 (training and eval): the pipeline is launched with dataset = golden + this_quarter’s_enriched - holdout. Fine-tuning the adapter overnight. Eval against the full suite + holdout. If it passes the gates, candidate v_new.

Week 4 (canary and promotion): deploy the candidate as an alternative adapter in vLLM, route 5-10 % of traffic to the candidate for 48-72 h. Online metrics: latency, complaint rate, implicit eval in production. If everything is OK, full promotion; if not, rollback and analysis.

Week 5+ (next cycle): the v_new adapter is now production. Feedback capture for the next quarter begins. The previous v_old goes to Archived but stays accessible for historical reproducibility.

Quarterly, that cycle plus whatever incident-driven mini-cycles show up in between. Operational, predictable, auditable.

What we have not covered (upcoming posts)

Online DPO and continuous on-policy learning: how the cycle shortens to hours or days (Fast-Slow Chasing, iterative RLOO). 2026 state of the art, still emerging in production.
Machine unlearning for GDPR: when a user exercises the right to be forgotten and their interactions formed part of the enriched dataset of an adapter in production. Negative LoRA, selective retraining.
Constitutional AI runtime: continuous alignment that replaces or complements periodic retraining.
Eval gates with metamorphic testing: evaluating robustness against input perturbations (typos, paraphrasing, language) as part of the promotion gate.

References

Argilla documentation, Building Datasets for LLM Fine-Tuning: https://argilla.io/docs.
Label Studio documentation, LLM Annotation: https://labelstud.io/templates/llm.
Langfuse documentation, User Feedback and Dataset Management: https://langfuse.com/docs/scores/user-feedback.
MLflow Model Registry stages: https://mlflow.org/docs/latest/model-registry.html.
Ethayarajh et al., KTO: Model Alignment as Prospect Theoretic Optimization (2024) — reference for the feedback cycle as an alignment signal.
Google Cloud, Continuous Training and MLOps for GenAI (2025).
DataRobot, MLOps Best Practices: Closing the Loop (2025).
Eugene Yan, Feedback Loops in LLM Systems (blog, 2025).

Prompt versioning: the contract that stops a five-word change sinking your system

Fri, 22 May 2026 07:30:00 +0200

TL;DR

In a classic software system, the most dangerous line a team can change is a SQL migration. In an LLM system, it is a line of prompt. The prompt determines the output as much as the model or more, it does not show up in unit tests, it does not appear in the logs by default, and if it is changed without leaving a trail there is no way to know which version generated which response. Prompt versioning is the discipline that turns the prompt into a first-class artefact: with a unique identifier, history, deployment labels, an associated eval suite, and per-request traceability. The field has settled on three primitives (immutable version, mutable label, read cache) and two dominant tools (Langfuse OSS with a built-in UI, MLflow Prompts integrated into the registry since MLflow 3.10). This article covers the pattern at first level: why it matters, how it is implemented, which tool to choose, and how it fits with Eval, Deploy and Observe.

You are here: cross-cutting (touches Data, Tune, Eval, Deploy and Observe)

Prompt versioning does not live in one stage, it cuts across five. It appears as a cross-cutting component in the master map of the six-stage LLMOps pipeline precisely for that reason: the prompt version is necessary metadata at every stage, not the responsibility of a single one.

The master analogy: the prompt is an invisible SQL migration

A serious backend team would never accept someone modifying a column directly in production without going through a versioned migration. Even if the change “works” at the time, without a migration there is no way to:

Reproduce the previous state if something fails.
Know who applied the change and when.
Apply the same change in staging before prod.
Test the new version against an automated suite before promoting.
Know, two months later, why the table has the shape it has.

The LLM prompt occupies exactly that position in an inference system. Changing "You are a helpful assistant." to "You are a helpful, concise assistant. Answer in fewer than 3 sentences." can:

Cut the average cost per response by 30 % (the responses are shorter).
Or degrade quality in a segment where concision breaks necessary nuance.
Or change the distribution of tools the agent decides to invoke.
Or alter the behaviour of the downstream LLM judge that assumes a certain length.

And most importantly: if the change is made by editing a constant in the app code and deploying, when two weeks later someone asks “why did the complaint rate go up in the finance segment?”, there is no way to know which prompt was being served at each moment. The logs store the response and, with luck, the model invoked; the prompt is rarely stored explicitly.

Prompt versioning solves the same problem Flyway/Liquibase/Alembic solved for SQL: turning an invisible change into an auditable artefact.

The three primitives of the pattern

Whatever the tool, the systems that work in 2026 share three operational primitives worth fixing before looking at products.

1. Immutable version

Every time the prompt content changes (template, system message, available variables, recommended model parameters such as temperature), a new version is generated with a unique identifier. The version is immutable: once created, it is not overwritten; if something needs changing, v+1 is created.

prompt_id: customer_support_v3
versions:
v1 (2026-03-12): "Eres un asistente de soporte..."
v2 (2026-04-08): "Eres un asistente de soporte... formato JSON..."
v3 (2026-05-21): "Eres un asistente de soporte... formato JSON... 3 frases máx..."

Immutability is what allows a trace from two months ago to be reproduced: if the trace says “customer_support_v3@v2 was served”, version v2 exists literally and can be reloaded.

2. Mutable label (deployment alias)

Versions are immutable, but which version is in production changes. That decision is materialised in labels: pointers with a semantic name (production, staging, canary) that point at a specific version and can be re-pointed.

prompt_id: customer_support_v3
labels:
production → v2 (servida al 100% del tráfico)
canary → v3 (servida al 5% del tráfico via gateway)
staging → v3

Promoting a version is moving a label, not editing the prompt. Rollback is moving the label back, not copying text. The operation reduces to an atomic mutation of a (label, version) tuple.

3. Read cache

The prompt is read on every request to the model. If every read calls the prompt registry service, you add latency and a dependency. The standard solution is a local cache in the client (TTL on the order of minutes) that invalidates when the label changes or when the TTL expires.

Langfuse implements a native client cache with configurable TTL and lazy invalidation; MLflow Prompts leaves the responsibility to the client or to a gateway layer. In both cases, in production the client serves the prompt from memory with negligible overhead (<1 ms), and only goes to the registry when it refreshes.

┌──────────────────┐
│ Client (app) │
│ - local cache TTL=60s
│ - lookup label "production"
│ - gets template
│ - renders variables
│ - sends to LLM
└─────────┬────────┘
│ (when the TTL expires or a change event arrives)
▼
┌──────────────────┐
│ Prompt registry │
│ - Langfuse / MLflow
│ - GET label="production"
│ - response: version_id + template
└──────────────────┘

With these three primitives, any reasonable tool is equivalent in the essentials. What distinguishes one from another is UI, integrations, RBAC, eval integration, and so on.

The two dominant tools in 2026

The field has converged on two main options. Any serious production deployment uses one of the two (sometimes both, for different teams).

Langfuse (OSS, prompt-management UI built in)

Langfuse is the prompt-first system: it was born for tracing and observability, and prompt management is one of its core layers. Key features for versioning:

Built-in UI to create, edit and version prompts. Versions are generated automatically on save; the history is visible and diffable.
Arbitrary labels beyond the usual ones (production, latest). You can define eu-prod, internal-only, customer-a for fine-grained routing.
Native client cache in the official SDKs (Python, JS), with configurable TTL, event-driven invalidation and fallback to the last-known-good if the registry is down.
Native tracing integration: when you record a call to the LLM, Langfuse automatically associates the prompt_id@version that served it. In the UI you see: this trace, this span, this prompt version X.
Eval integration: Langfuse lets you register eval suites that fire when a new prompt version is created. The results stay linked to the prompt_id@version and are the natural gating for promoting staging → production.
Self-hosted or cloud: the core is OSS (MIT), runs on Docker compose or Helm; the cloud version adds SLA, SSO and support.

When Langfuse suits:

Teams that want a rich UI so product/PM/analyst can manage prompts without touching code.
OSS-first deployments where control of the runtime and of persistence is a requirement (on-premise, ENS).
When LLM observability is already in Langfuse: prompt management is marginal in setup.

MLflow Prompts (included in MLflow 3.10, March 2026)

MLflow Prompts is the classic MLOps ecosystem’s answer for LLMs. Features:

Integrated into the MLflow Model Registry: prompts are first-class registry artefacts, with the same stage semantics (Staging, Production, Archived) MLOps teams already know.
API consistent with the rest of MLflow: mlflow.register_prompt(), mlflow.load_prompt(name, stage="Production"). The learning curve for teams already using MLflow for models is nil.
Automatic versioning with a numeric version_id (1, 2, 3, …) and optional comments on promotion.
No built-in UI dedicated to prompts (the MLflow UI does the job, but it is designed for models; the flow is less polished than in Langfuse).
No native GenAI-aware tracing (MLflow Tracing in the 3.10 GenAI dashboard provides it, but the trace↔prompt integration is more manual than in Langfuse).
Compatible with any model registry backend MLflow supports (filesystem, Postgres, MySQL, S3, GCS, Azure Blob).

When MLflow Prompts suits:

Teams already operating MLflow for classic ML who want to extend the same discipline to LLMs without adding vendors.
Deployments where the centre of gravity is the model registry and the prompt is one more artefact.
CI/CD pipelines that already speak MLflow (CLI, REST API).

Comparison

Feature	Langfuse	MLflow Prompts
Core licence	MIT (OSS)	Apache 2.0 (OSS)
Prompt-first UI	✅	⚠️ via Model Registry
Immutable versioning	✅	✅
Mutable labels	✅ (arbitrary)	✅ (Staging/Production/Archived)
Native client cache	✅	❌ (DIY)
Integrated tracing	✅ native	⚠️ via MLflow Tracing
Eval gating on promotion	✅	⚠️ DIY with MLflow Recipes
Easy self-hosting	✅ Docker/Helm	✅ standard MLflow
Curve if you come from MLOps	medium	nil
Curve if you come from DevOps	nil	medium

In May 2026, the most widespread hybrid pattern is to use MLflow for the models+adapters registry and Langfuse for prompts+tracing, connected by trace_id and prompt_id travelling in the OpenTelemetry span attributes. Covered in evals and MCP observability.

Minimum schema of a versioned prompt

Whatever the tool, what the registry stores in each version has a reasonable minimum schema:

# prompt_id: customer_support_v3, version: 3
template:
 system: |
 Eres un asistente de soporte de {{company_name}}.
 Responde en español neutral, máximo 3 frases.
 Formato de respuesta: JSON {"answer": "...", "needs_human": bool}
 user: |
 Pregunta del cliente: {{user_message}}
 Contexto del ticket: {{ticket_context}}

variables:
 required: [company_name, user_message, ticket_context]
 defaults: {}

recommended_params:
 model: "llama-3-70b-instruct"
 temperature: 0.3
 max_tokens: 300
 response_format: "json_object"

metadata:
 author: "mlops@empresa.com"
 created_at: "2026-05-21T14:23:00Z"
 commit_message: "Añade límite de 3 frases tras feedback ticket #1842"
 eval_suite: "customer_support_v3_evals"
 related_traces: ["trace_id_x", "trace_id_y"]

This is the minimum contract. What sets a serious deployment apart:

variables.required is validated in the client before sending to the model. A missing variable blows up at client time, not in a confusing model response.
recommended_params.model ties the prompt version to a model. Changing model opens a debate (does the new version work with Llama 3 70B and with GPT-4o?). If it is not tied, the model is one more variable that wrecks reproducibility.
metadata.eval_suite is what the eval suites hook into: on creating v3, MLflow/Langfuse fires customer_support_v3_evals automatically.

Integration with eval gates: governed promotion

The real value of prompt versioning appears when it is integrated with eval. The canonical pattern:

Developer edits the prompt in the UI (Langfuse) or via the API (MLflow). v4 is created.
Automatic trigger: the prompt_created event fires the associated eval suite (the eval_suite in the metadata).
The suite runs against the golden dataset (questions+answers labelled by a human). Covered at first level in the evals post.
Results are attached to the version: v4 now has eval_score: 0.84, regression_vs_v3: -0.03.
Promotion gate: if eval_score >= threshold and regression < tolerance, the staging label moves to v4 automatically. If not, it alerts the developer.
Manual promotion to production: with the eval passed, someone with permission moves production from v3 to v4. Atomic, auditable, reversible.

Developer edits prompt → v4 created
│
▼
[eval suite trigger]
│
▼
Golden dataset 200 examples
│
▼
score = 0.84 (vs 0.87 for v3)
│
├── If it passes the threshold → label staging → v4
│ └── Manual promotion to production after review
└── If it does not pass → block + alert the developer

This flow turns the prompt change from “someone touched the code and we prayed” into “a prompt change is a PR that passes CI”. It is the same discipline classic MLOps applied to models.

Per-request traceability: which version served each response

The last piece is operational traceability: given a model response in production, which prompt version generated it?

The pattern is to propagate the version as a span attribute in OpenTelemetry, following the gen_ai.* semantic conventions we covered in MCP observability:

# In the client (common pseudo-code)
prompt = registry.load("customer_support_v3", label="production") # v3 → v_id=14

with tracer.start_as_current_span("llm_call") as span:
 span.set_attribute("gen_ai.prompt.id", "customer_support_v3")
 span.set_attribute("gen_ai.prompt.version", "14")
 span.set_attribute("gen_ai.prompt.label", "production")
 span.set_attribute("gen_ai.request.model", prompt.params.model)

 response = llm.complete(prompt.render(user_message=msg), **prompt.params)

 span.set_attribute("gen_ai.usage.input_tokens", response.usage.input)
 span.set_attribute("gen_ai.usage.output_tokens", response.usage.output)

In any trace (Langfuse, Phoenix, Jaeger, Honeycomb) you can see which exact version served that response. In an incident (“customer X received this on 22 May”) you reproduce literally the version and the model that generated the output.

Without that traceability, the incident stays an anecdote; with it, it is debuggable.

Applied to typical on-premise hardware

Prompt versioning is a computationally light layer compared with the inference engine or the fine-tuning pipeline. Its requirements:

Storage: the prompt registry typically weighs megabytes (hundreds to thousands of prompts with their versions). Postgres with a prompts(id, version, template, params jsonb, metadata jsonb, created_at) schema is more than enough. Langfuse uses Postgres by default; MLflow uses it for metadata (the blobs go to object storage or the filesystem).
Registry compute: a small instance (1-2 vCPU, 2 GB RAM) serves tens of thousands of reads per minute if the client cache is enabled. Without a cache, it scales linearly with QPS but is still trivial.
Triggered eval compute: here there is a real cost. Every time a new version is created, the eval suite runs. If the suite does LLM-as-judge over 200 examples and each eval costs 4 K tokens, one promotion costs on the order of 1 M tokens, minutes on a decent cluster, seconds if the suite already has its hot prefix cache.

For an RTX 4090 serving Llama 3 8B with a self-hosted prompt registry (Langfuse or MLflow): the registry runs on the same node in a sidecar container, the local app caches in RAM, the eval triggers run against the same inference engine at low priority. Full setup in a morning.

For a 4×H100 SXM cluster serving a large model to several tenants: registry in a dedicated K8s pod with replicated Postgres, eval suites running in pods with the spot priority class (covered in the cluster as a platform), OTel tracing propagating prompt_id+version to a central Langfuse.

Traps and things that are not what they look like

Prompts hardcoded in the app code. The most common antipattern. The prompt lives in a prompts.py file or templates/customer.txt that gets deployed with the app. There is no real versioning (git history is not a substitute: it does not tie commit ↔ production trace operationally). Migrating to a registry is 1-2 sprints of work; it is worth every hour.

Badly calibrated cache. A TTL of hours with a mutable label means a rollback takes time to propagate. A TTL of seconds overloads the registry. The reasonable default is 60-300 seconds with event-driven invalidation (the registry emits a message to Kafka/Redis when a label changes, and clients invalidate immediately).

Unvalidated variables. The template uses {{user_name}} but the app passes {{username}}. The render produces a prompt with a literal {{user_name}}. The model answers something bizarre and nobody knows why. Validating required variables in the client before sending to the model is the minimum discipline.

Prompts inside chains evaluated at runtime. If your stack uses LangChain, LlamaIndex or similar with chains that compose prompts at runtime, the final prompt the model sees may not be in the registry because it was composed from several fragments. Solutions: either register the chains as artefacts, or log the effective composed prompt in every trace.

Eval suite not hooked to the prompt_id. Without that link, a prompt change gets promoted without passing evals. The integration has to be a field in the prompt metadata (eval_suite: ...) that the system reads and fires automatically. If it depends on the developer “remembering”, the pattern will fail.

Non-existent RBAC roles. Anyone with access to the UI can move production to any version. Without separating editor (creates versions) from releaser (moves production labels), a junior developer can break production with an accidental promotion. Langfuse Enterprise has granular RBAC; MLflow has it via the server backend with per-experiment/registry permissions.

Prompts with sensitive data inline. The prompt template includes few-shot examples with real names, addresses, customer IDs. The registry stores that indefinitely. Under GDPR, the right to erasure applies to the registry too. Good practice: variables for sensitive data, not inline; periodic audit of the registry content.

Recommended operational pattern: the cycle on one screen

A serious team with prompt versioning properly set up has the following cycle, repeatable and cheap:

Developer opens a PR in the repo: changes the app code if necessary, but does not touch the prompt there.
Edits the prompt in the Langfuse/MLflow UI: creates v_new. Adds a commit message (“adds a 3-sentence limit after feedback on ticket #1842”).
The eval suite fires automatically: it runs against the golden dataset, results appear in the UI within minutes.
If it passes eval: the staging label moves to v_new automatically. The developer can test staging with controlled traffic.
Human review (1-2 people, optional depending on severity): approval.
Promotion to production: move the label, atomic. The client caches for 60-300 s, then serves the new version.
Observe: in Langfuse/Phoenix, production metrics and eval scores are segmented by prompt version. If the score drops with v_new, alert.
If there is a serious regression: rollback is moving the label back. A 5-second operation.

Every step is audited, every decision leaves a trail, every rollback is an atomic operation. This is what separates a GenAI system from “demos that worked once” and makes it operable for years.

What we have not covered (upcoming posts)

Automatic prompt optimisation: techniques such as DSPy, TextGrad, PromptBreeder that generate prompt candidates and optimise them against a measurable objective. The extension of versioning where the “developer” can be an optimiser.
Prompt injection and red teaming: integrating versioning with the adversarial evaluation flow. Partially covered in guardrails.
Different versions per tenant: when the same prompt_id needs variants per customer (i18n, branding, domain). Fork + override pattern.

References

Langfuse documentation, Prompt Management: https://langfuse.com/docs/prompts/get-started.
MLflow 3.10 release notes, Prompts in Model Registry (marzo 2026): https://mlflow.org/releases/3.10.
OpenTelemetry, Semantic Conventions for Generative AI (estables desde 1.36 de OTel): https://opentelemetry.io/docs/specs/semconv/gen-ai/.
Google Cloud, Prompt Management: Best Practices for Production LLM Systems (2025).
Chip Huyen, Designing Machine Learning Systems — capítulo sobre model registry y prompt-as-artifact (2ª edición, marzo 2026).
Eugene Yan, Prompt Engineering as Software Engineering (blog, 2025).

Continuous fine-tuning in production: from real traffic to the deployed adapter

Thu, 21 May 2026 10:00:00 +0200

TL;DR

Continuous fine-tuning is not “training the model every so often”. It is a closed loop where real production traffic generates the datasets, a short pipeline trains a LoRA adapter, a battery of evaluations decides whether it is promoted, and vLLM loads it without restarting. The state of the art in May 2026 has fragmented the stack: it is no longer DPO against everything, but a choice between SFT, DPO, KTO, ORPO and SimPO according to the type of signal your product captures. What has consolidated the pattern is the combination of PostgreSQL 18 + pgvector 0.8 as the nervous system of the pipeline, traffic capture, dataset versioning, eval results, adapter registry, together with vLLM multi-LoRA hot-swap, which turns deployment into an HTTP call. This article takes the cycle apart with concrete schemas, real queries, and the numbers it costs on an RTX 4090 against a 4×H100 cluster.

You are here: Tune + Retrain

This post crosses two stages of the six-stage LLMOps pipeline: the decision to train a new adapter (the Tune stage) is triggered by the signals from Observe that travel through the Retrain stage until the loop closes. The post takes apart the complete circuit between the two boxes.

Picture a neighbourhood restaurant with a signature dish that works, but the chef knows it can be tuned. Every night things happen:

Some diners leave part of the dish: a weak signal that something did not quite land.
Others ask for another version (“could you put less salt in it?”): an explicit, directional signal.
Others finish the dish and come back the following week: the only signal that really matters, but it arrives late.
And a select group offers an opinion without being asked, usually a negative one.

The chef does not redo the menu every night. He does something more interesting: he writes down in a notebook the dishes served, the returns, the changes requested, the tips. Every so often, he reads the whole notebook, decides on minimal adjustments to a recipe, tries the new version at a private table with his staff, and only if they take to it does he add it to the next day’s menu. Sometimes he even serves two different versions of the dish to different tables for a week, measures what happens, and picks one.

That is continuous fine-tuning. The notebook is Postgres. The dish is the base model. The notes are feedback signals, explicit and implicit. The “minimal adjustment” is a 30 MB LoRA adapter. The private table is the battery of automated evaluations. The next day’s menu is vLLM with multi-LoRA hot-swap, which loads the new adapter without restarting the service. Serving two versions to different tables is A/B testing with real traffic.

The analogy is exact on one critical point: the chef does not throw away the original recipe. He keeps the base recipe and holds a separate notebook with the “modifications that work well for the neighbourhood regulars”. That notebook is the LoRA adapter: on top of the base model, not in its place.

The cycle, taken apart

Before going into components, it helps to fix the complete flow. These seven steps are what any serious team replicates with variations:

The cycle lasts between 1 and 4 weeks in real production. What changes between teams is the tempo (faster in assistant chat, slower in regulated banking) and the details of each step. The structure is the same.

Why continuous fine-tuning (and why it is not RAG)

Before going deeper, a distinction that people still confuse. Fine-tuning is for form, not for facts. If your problem is that the model does not know the customer’s rates or the updated catalogue, do not fine-tune: use RAG. If your problem is that the model responds in a tone that does not fit, does not respect your JSON format, refuses legitimate cases or invents structure, then fine-tuning is the answer.

In 2026 the boundary is well established by community practice:

Observed problem	Solution
The model does not know X (X changes weekly)	RAG
The model knows X but responds badly in tone or format	SFT fine-tuning
There are two ways of responding and I prefer one over the other	Fine-tuning with preferences (DPO/KTO/ORPO/SimPO)
The model reasons badly in a verifiable domain (code, maths)	RL with verifiable reward (GRPO/DAPO)
The model is competent, it just needs a memory of facts	RAG, not fine-tuning

Continuous fine-tuning is the disciplined version of the second and third cases. The key word is continuous: it is not a one-off “we aligned the model” event, it is a process that runs every time the traffic distribution drifts far enough, or new use cases appear.

The four techniques according to the signal you capture

The most important change of the last 12 months has been the end of DPO’s monopoly. In 2024 every team doing alignment used DPO with (chosen, rejected) pairs. In 2026 the choice is finer and depends on what the signal you collect in your product looks like:

Real signal in the product	Recommended technique	Why
Labelled correct examples (input → expected output)	SFT + LoRA	Still the baseline. 500-5,000 examples are enough for style.
Explicit `(chosen, rejected)` pairs	DPO or SimPO	SimPO removes the reference model → 50 % less VRAM in training.
Loose 👍 / 👎 on responses	KTO	The method that fits real telemetry most naturally.
SFT and preferences in a single pass	ORPO	A single model in memory, avoids drift between phases.
Verifiable reward (tests, solutions)	GRPO / DAPO	Reasoning, not chat. A different world.

The practical rule: design the feedback capture in the product thinking about which method you will be able to use afterwards. If your UI only has 👍/👎, you force the path to KTO. If you add a “regenerate response” button, you unlock DPO from regenerate-as-rejected (we will see it below). If you add an “edit response” button, the edited response becomes high-quality direct SFT.

There is a cost detail that gets little publicity. DPO needs to keep two models in memory: the one you are training and the reference one. SimPO removes that second model. ORPO does too. For a Llama 3 8B in BF16 this is the difference between needing ~32 GB of active VRAM during training (DPO) or ~16 GB (SimPO/ORPO). It is the difference between the training fitting on an RTX 4090 with aggressive QLoRA, or not fitting without offload.

Postgres as the nervous system of the pipeline

Here is the strong technical opinion of this article, and it is one worth defending with data: Postgres 18 + pgvector 0.8 + an S3/MinIO bucket for the weights is enough for the whole pipeline. You do not need MLflow, you do not need lakeFS, you do not need DVC.

This is not ideological minimalism. It is about three concrete advantages that no alternative stack matches in the on-premise scenario with compliance:

A single source of truth, a single authorisation model. The ACLs you already have for Postgres cover the training data, the eval results, the adapter registry and the audit log. You do not multiply control planes.
SQL as the pipeline’s universal language. The query that generates the dataset, the eval gate predicate, the A/B traffic assignment, the promotion decision: it is all SQL. Your team already knows SQL.
Audit and cryptographic reproducibility for free. The pg_audit and pgcrypto extensions, combined with set_hash over the dataset, give you cryptographic traceability with no additional code. It is a subject that would fill an article of its own.

A concrete schema

We start with the traffic table, partitioned by week so that DROP PARTITION is cheap:

CREATE TABLE obs.inference_log (
 id BIGSERIAL,
 request_id UUID NOT NULL,
 tenant_id INT NOT NULL,
 user_hash BYTEA, -- GDPR pseudonymisation
 adapter_id TEXT NOT NULL, -- e.g. "support-es-v4.1"
 experiment TEXT, -- e.g. "rerank-v2-canary"
 variant CHAR(1), -- 'A' | 'B' | NULL
 messages JSONB NOT NULL,
 completion TEXT,
 ttft_ms INT,
 tokens_in INT,
 tokens_out INT,
 -- feedback signals
 fb_explicit SMALLINT, -- -1/0/+1 (KTO-ready)
 fb_regen BOOLEAN DEFAULT false, -- user regenerated -> DPO-rejected
 fb_edited BOOLEAN DEFAULT false, -- user edited -> SFT golden
 parent_id BIGINT, -- self-reference for regenerate
 -- vector and meta
 embedding HALFVEC(1024), -- pgvector 0.8, half the RAM
 pii_flags SMALLINT DEFAULT 0, -- bitmask
 created_at TIMESTAMPTZ DEFAULT now()
) PARTITION BY RANGE (created_at);

CREATE TABLE obs.inference_log_2026w21 PARTITION OF obs.inference_log
 FOR VALUES FROM ('2026-05-18') TO ('2026-05-25');

CREATE INDEX ON obs.inference_log_2026w21
 USING hnsw (embedding halfvec_cosine_ops);
CREATE INDEX ON obs.inference_log_2026w21
 (tenant_id, adapter_id, created_at);

Three decisions deserve a note:

HALFVEC(1024). Native FP16 vectors in pgvector 0.8. Half the RAM and disk, with a loss of precision that is irrelevant for semantic deduplication. This alone, at a scale of millions of rows, saves between 4 and 8 GB.

Weekly partitioning by time range. At 90 days, DROP TABLE obs.inference_log_2026wXX frees space in milliseconds without a prolonged lock. Autovacuum never touches frozen partitions again.

Self-referencing parent_id. The user regenerates the response → a new row is inserted with parent_id pointing at the previous one. That will give us a DPO dataset without touching the UX.

The adapter registry

CREATE TABLE serve.adapter (
 id TEXT PRIMARY KEY,
 base_model TEXT NOT NULL,
 rank INT, alpha INT,
 target_modules JSONB,
 method TEXT, -- 'sft'|'dpo'|'kto'|'orpo'|'simpo'
 training_run_id UUID,
 dataset_snapshot_id UUID,
 weights_uri TEXT, -- s3://.../v4.2.safetensors
 eval_summary JSONB,
 status TEXT NOT NULL, -- 'training'|'canary'|'prod'|'retired'
 traffic_pct NUMERIC(5,2) DEFAULT 0,
 promoted_at TIMESTAMPTZ
);

The vLLM router reads this table with a TTL of a few seconds. An UPDATE serve.adapter SET status='prod', traffic_pct=100 WHERE id='v4.2' is a promotion. An UPDATE ... SET status='retired' is a rollback. The audit of who did what and when comes from pg_audit without writing a single additional line of code.

Generating DPO and KTO datasets from real traffic

This is where the elegance of the schema pays off. The dataset is not a static file: it is a materialised view built with SQL over obs.inference_log.

KTO dataset from 👍/👎

KTO is the method that best fits the signal any decent chat product captures. The query:

CREATE MATERIALIZED VIEW train.kto_v3_candidate AS
SELECT
 messages AS prompt,
 completion AS response,
 CASE WHEN fb_explicit > 0 THEN true ELSE false END AS label,
 adapter_id, created_at
FROM obs.inference_log
WHERE fb_explicit != 0
 AND created_at > now() - interval '60 days'
 AND pii_flags = 0
 AND tenant_id IN (SELECT id FROM tenant WHERE consent_training);

Simple. Every row with explicit feedback becomes an example (prompt, response, desirable_yes_no). KTO trains directly on this signal, with no need to build pairs.

DPO dataset from “regenerate”

The trick that is worth this article on its own. When the user presses “regenerate response”, they are giving an extraordinarily strong signal: the first response was no good to them. If the second is neither regenerated nor rated negatively, we assume it was. That is a DPO pair without a single extra click in the UI:

CREATE MATERIALIZED VIEW train.dpo_v3_candidate AS
SELECT
 rej.messages AS prompt,
 cho.completion AS chosen,
 rej.completion AS rejected,
 rej.adapter_id
FROM obs.inference_log rej
JOIN obs.inference_log cho ON cho.parent_id = rej.id
WHERE rej.fb_regen = true
 AND cho.fb_explicit >= 0
 -- length bias mitigation in classic DPO
 AND cho.tokens_out BETWEEN rej.tokens_out * 0.5
 AND rej.tokens_out * 2.5;

The clause on lengths is the cheap cure for the length bias documented in DPO. Without it, the model learns that “longer = better” because the responses the user accepts tend to be slightly longer. With SimPO or ORPO this filter is optional; with classic DPO it is necessary.

Semantic deduplication with pgvector

Before training, dedup. Two almost identical prompts in the dataset are noise that biases the model:

WITH ranked AS (
 SELECT id, embedding,
 row_number() OVER (
 PARTITION BY hashtext(messages::text)
 ORDER BY fb_explicit DESC, created_at DESC
 ) AS rn
 FROM obs.inference_log
 WHERE created_at > now() - interval '60 days'
)
DELETE FROM train.kto_v3_candidate kto
USING ranked r
WHERE r.rn > 1 AND kto.id = r.id;

And for the semantic duplicates (paraphrases) we use pgvector 0.8 directly with an iterative index scan:

-- Find near-duplicates of any given example
SELECT id, messages, embedding <=> $1 AS dist
FROM obs.inference_log
WHERE created_at > now() - interval '60 days'
 AND embedding <=> $1 < 0.05
ORDER BY embedding <=> $1
LIMIT 50;

The iterative scan is a key improvement in pgvector 0.8: before, the HNSW index could return fewer results than requested when there were additional filters (WHERE); now it iterates until it meets the limit. Without that improvement, curation queries over datasets of millions of rows were unworkable without a brutal pre-filter.

Eval gates: three stages, all SQL

The most common mistake when implementing continuous fine-tuning is skipping or watering down the eval gates. That turns the cycle into a roulette wheel. The pattern that works in 2026 is three stages, each with a different latency/coverage trade-off:

And this is where Postgres shines again: the promotion gate is expressed as an SQL predicate. Nothing more:

CREATE TABLE train.eval_result (
 adapter_id TEXT REFERENCES serve.adapter(id),
 suite_id TEXT, -- 'safety-es', 'support-helpfulness'
 metric TEXT,
 score NUMERIC,
 judge_model TEXT,
 judged_at TIMESTAMPTZ DEFAULT now(),
 PRIMARY KEY (adapter_id, suite_id, metric)
);

CREATE OR REPLACE FUNCTION serve.can_promote(candidate TEXT, current TEXT)
RETURNS BOOLEAN AS $$
 SELECT NOT EXISTS (
 SELECT 1
 FROM train.eval_result c
 JOIN train.eval_result p USING (suite_id, metric)
 WHERE c.adapter_id = candidate
 AND p.adapter_id = current
 AND suite_id IN ('safety-es','support-helpfulness','refusal-rate')
 AND c.score < p.score * 0.98 -- 2 % tolerance
 );
$$ LANGUAGE sql STABLE;

An SQL function as the gate. Usable from CI with psql -c "SELECT serve.can_promote('v4.2','v4.1')" and an exit code of 0/1. No orchestrator needed, no specific UI needed. The audit stays in the Postgres log.

vLLM multi-LoRA: the deploy is an HTTP POST

Two years ago, deploying a new fine-tune meant rotating inference pods. Today it is an HTTP call. vLLM 0.7+ supports loading and unloading LoRA adapters hot, keeping several resident in VRAM and choosing the right one per request.

Server configuration:

vllm serve meta-llama/Llama-3.1-8B-Instruct \
 --enable-lora \
 --max-loras 4 \
 --max-lora-rank 64 \
 --env VLLM_ALLOW_RUNTIME_LORA_UPDATING=True

Deploying a new adapter:

curl -X POST http://localhost:8000/v1/load_lora_adapter \
 -H "Content-Type: application/json" \
 -d '{
 "lora_name": "support-es-v4.2",
 "lora_path": "/mnt/adapters/support-es-v4.2"
 }'

From that moment on, requests including "model": "support-es-v4.2" are served with that adapter applied on top of the base model. Switching between adapters has negligible latency (the most recent research on Activated LoRA takes this to levels where the switching cost is invisible).

This changes operations substantially. Deploying a new fine-tune stops being an infrastructure event and becomes a state change in Postgres. The router queries the serve.adapter table, sees that v4.2 is in canary with traffic_pct=5, and directs 5 % of requests to the new adapter. The exact route of that 5 % is decided with deterministic hashing of the user_id so that the same user always sees the same variant (sticky):

-- No assignment table, no additional state
-- the variant is computed in-place in SQL or in the router:
SELECT
 CASE WHEN hashtext($user_id || $experiment) % 100
 < (SELECT traffic_pct FROM serve.adapter WHERE id = $candidate)
 THEN $candidate ELSE $current END AS adapter_id;

A/B with real traffic: measure or live deceived

The eval gates measure against fixed benchmarks. That is necessary but insufficient. Reality is only measured with real traffic. Once the adapter is in canary, what matters are the online metrics measured over obs.inference_log for each variant:

SELECT
 adapter_id,
 COUNT(*) AS n,
 AVG(fb_explicit) AS mean_score,
 STDDEV(fb_explicit) / SQRT(COUNT(*)) AS sem,
 AVG(ttft_ms) AS ttft_avg,
 percentile_cont(0.5) WITHIN GROUP
 (ORDER BY ttft_ms) AS ttft_p50,
 percentile_cont(0.95) WITHIN GROUP
 (ORDER BY ttft_ms) AS ttft_p95,
 AVG(CASE WHEN fb_regen THEN 1 ELSE 0 END) AS regen_rate
FROM obs.inference_log
WHERE experiment = $1
 AND created_at > now() - interval '7 days'
GROUP BY adapter_id;

What to look at: explicit feedback, latency (TTFT, p50, p95), regeneration rate. An adapter that raises the mean feedback but also raises the regeneration rate is suspicious, probably it is responding in a flashier but less useful way. An adapter that lowers latency but lowers feedback may be worth studying: it may be being more concise than it should.

Promotion to prod happens when, after 24-72 hours in canary, the candidate adapter beats the current one on at least one key metric without degrading the others. Again: it is an UPDATE in Postgres.

Applied to typical on-premise hardware

Let us come down to two representative configurations, one for iteration and one for production.

Case 1 — RTX 4090 (24 GB) for development iteration

An RTX 4090 with 4-bit QLoRA can train adapters on an 8B model without trouble. The VRAM budget combines four components; the KV cache during the intermediate evaluations is not negligible and deserves an explicit margin:

8B base model in 4-bit: ~5 GB
Activations + gradients: ~8 GB (depends on batch and context)
Optimizer state (LoRA r=16): ~0.5 GB
KV cache during eval: ~2 GB
Safety margin: ~8 GB

Typical times (an estimate based on community benchmarks; measure in the lab before promising anything):

Dataset	Technique	Adapter rank	Approximate time
1,000 SFT examples	LoRA r=16	16	20-40 min
5,000 SFT examples	LoRA r=32	32	2-4 h
2,000 DPO pairs	LoRA r=16	16	1-2 h
5,000 KTO examples	LoRA r=32	32	3-5 h

This puts the iteration cycle, a change in the dataset, retrain, eval, look at the number, in the range of a working day. Enough to validate a hypothesis before moving anything to the production cluster.

Case 2 — 4×H100 SXM cluster (320 GB, NVLink) for production

With a cluster of this order the whole picture changes. You can:

Train LoRA on a 70B in BF16 with no quantisation with tensor parallel = 4.
Do full DPO with a resident reference model when the reference is quantised to FP8, or move to SimPO / ORPO, which remove that intermediate model and simplify VRAM planning (see the table of techniques above).
Support multi-tenant fine-tuning: several customer adapters training in parallel in separate logical pipelines, each isolated in a different Postgres partition with its own ACLs.
Serve multi-LoRA with --max-loras 8 on top of the base model without concurrency lowering throughput perceptibly.

The practical budgeting rule: over a 12-month horizon, a team with this cluster can run ~150-200 continuous fine-tuning cycles (training + eval + canary + promotion or discard) if the discipline around the dataset and the eval gates is strict. If it is not, it will run twice as many but with half the usefulness.

Position within the architecture: what this article covers and what it does not

To place the scope: the cycle drawn at the start has seven boxes, all of them covered here in their mechanics. Three cross-cutting layers are deliberately left out, and they are the ones that end up separating a pipeline that works technically from one that survives an audit:

Cryptographic provenance and traceability. We have mentioned dataset_snapshot and pg_audit, but the complete mechanics, the set_hash over the examples, the integration with the EU AI Act, the frozen query_sql as proof of what trained the model, is enough for a whole analysis.
Judge calibration. We have assumed LLM-as-judge works. It needs calibrating against a human rubric on at least 100 cases per critical suite before trusting it. Without that calibration, the eval gates are theatre.
The forgetting problem. What happens if a user exercises their GDPR right to be forgotten and their interactions were part of the dataset of an adapter already in production? There is no clean solution. There are options, incremental retrain, sample-level machine unlearning, negative LoRA, and it is worth knowing them before a customer asks.

What we have not covered (upcoming articles)

Cryptographic provenance on Postgres: how set_hash and a frozen query_sql compose an auditable chain of custody under the EU AI Act.
Honest judge calibration: why score > 0.85 means nothing without a human baseline, and how to build that baseline without it costing a month of work.
The forgetting problem in adapters: sample-level machine unlearning, incremental retrain and other techniques for answering GDPR without throwing the adapter away.
Online DPO and on-policy continual learning: the state of 2026 research (Fast-Slow Chasing, RLOO, iterative on-policy) and why it is not production yet.

References

Hu et al., LoRA: Low-Rank Adaptation of Large Language Models (ICLR 2022).
Dettmers et al., QLoRA: Efficient Finetuning of Quantized LLMs (NeurIPS 2023).
Rafailov et al., Direct Preference Optimization: Your Language Model is Secretly a Reward Model (NeurIPS 2023).
Meng, Xia, Chen, SimPO: Simple Preference Optimization with a Reference-Free Reward (NeurIPS 2024).
Hong et al., ORPO: Monolithic Preference Optimization without Reference Model (2024).
Ethayarajh et al., KTO: Model Alignment as Prospect Theoretic Optimization (2024).
Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023) — original vLLM.
Official vLLM Multi-LoRA documentation: https://docs.vllm.ai/en/stable/features/lora/.
Official pgvector 0.8 documentation: https://github.com/pgvector/pgvector.
TRL (HuggingFace) docs: https://huggingface.co/docs/trl.
EU AI Act, consolidated text and application timeline: https://artificialintelligenceact.eu/.

The GPU cluster as a platform: turning a shared cluster into a multi-tenant service your teams can consume

Thu, 21 May 2026 07:15:00 +0200

TL;DR

Having an expensive GPU cluster and many different workloads that want to use it is not an infrastructure problem: it is an internal product problem. What separates “we have a cluster” from “we have an inference platform” are four layers the market consolidated in 2026: a gateway layer that centralises authentication, routing and policies (LiteLLM, Portkey, Kong AI Gateway); a GPU isolation model appropriate to the tenants’ profile (MIG hardware isolation for untrusted multi-tenant, MPS for processes from the same team, time-slicing only for dev); a quota and rate limiting system with budgets per tenant/team/project (LiteLLM does this in its core at team/user/api-key level with descriptive 429s); and a multi-tenant observability plane that enables real cost attribution (showback as an intermediate step, chargeback as the destination), per-tenant tracing and differentiated dashboards. Applied to a typical mid-scale GPU cluster (one node with 4-8 H100 SXM and NVLink, a common place to start in production), this translates into concrete decisions: with ~640 GB of aggregate VRAM across 8 GPUs and two typical production models (a large 70B+ model with tensor parallel and a replicated mid-sized model), the cluster serves between tens and low hundreds of simultaneous sessions depending on the mix; GPU isolation is usually resolved with MIG on smaller workloads and per-model dedication on large ones; and the platform’s success metric is effective utilisation, which in typical production sits at 30-40% and where the reasonable optimisation target is to push it to 60-70% without degrading SLAs.

This is the fifth post in the MLOps for LLMs series. It is the most operationally oriented one and cuts across several pipeline stages (Deploy + Observe plus cross-cutting concerns). The “you are here” marker flags the two active stages because the notion of a multi-tenant platform does not live in just one.

You are here: Deploy + Observe (the cluster as a product)

The question that changes the framing

When a platform team buys expensive GPU hardware and starts building inference, the first version is almost always single-purpose: one model, one client, one target latency. It works. When the second team turns up asking for the same resource, that single purpose becomes internal politics: how many replicas do we give them? What do we do if the SLAs clash? Who pays for the tokens of team B’s experiment? And when the third one arrives, what was an SRE project becomes an internal product project.

The distinction is not technical, it is one of framing. A cluster is infrastructure. A platform is a service with customers, contracts and success metrics. The change of framing implies:

Identifiable customers (tenants), not anonymous users.
Contracts (latency SLA, guaranteed throughput, available models), not “whatever we manage”.
Success metrics that are not technical but product ones: adoption, satisfaction, cost per query per tenant, time to the first “hello world”.

This post walks through how that change of framing is carried out. It lands it on a mid-scale cluster (4-8 H100 SXM with NVLink in a single node), the usual configuration when starting with serious LLM inference; but the principles generalise to any topology, from a single node with two GPUs to multi-node clusters with InfiniBand.

The four layers of a multi-tenant inference platform

The canonical architecture that settled in 2026 has four layers that any serious multi-tenant platform implements, from the outside in:

Each layer solves a concrete problem. Let us take them one by one.

Layer 1 — AI Gateway: the single front door

The AI Gateway is the component your tenants see. It is an HTTP/gRPC API compatible with OpenAI (typically /v1/chat/completions, /v1/embeddings, /v1/models) that centralises everything that happens before touching the inference backends.

Why centralise

Without a gateway, tenants connect directly to vLLM or whatever the model is. Every change (rotating an endpoint, adding a model, changing credentials, applying a policy) means notifying every tenant. Every tenant has its own retry logic, its own logging, its own auth model. It is unworkable from the third customer onwards.

With a gateway, the change is made in one place. Tenants have a stable URL and a set of credentials; the rest is the gateway’s problem.

The three dominant options in 2026

LiteLLM is the most popular OSS option, Python-first, deployed as a proxy. It supports 100+ providers (OpenAI, Anthropic, Bedrock, self-hosted vLLM, Ollama, and so on) behind a unified OpenAI-compatible API. It has a native multi-tenant hierarchy with Organizations → Teams → Users → API Keys, each level with its own budget. The Apache 2.0 version covers the basics; RBAC, SSO, audit logs and team-level enforcement require the paid Enterprise version. Deployed on K8s with the official Helm chart.

Portkey is the most mature commercial / SaaS option. A single control plane that enforces budgets, quotas, permissions and compliance. Real-time spending tracking with alerting. RBAC, audit, workspaces and SSO included. The trade-off: dependency on an external service and a per-request pricing model.

Kong AI Gateway is the option for organisations that already have Kong as their API gateway. An AI plug-in on top of the existing Kong gateway, integrated with its plugin, consumer and rate-limit model. If your platform team already runs Kong, it is the lowest friction.

When to choose each one

Situation	Gateway
Pure OSS, self-hosted, Python-first team	LiteLLM
You need RBAC, SSO, audit log out of the box, and have budget	Portkey
You already run Kong as a corporate API gateway	Kong AI Gateway
Greenfield enterprise with strict compliance	Portkey (probably)
Mid-sized OSS-first company without regulated compliance	LiteLLM (typically)

The minimum the gateway has to do

Whichever option you pick, this is what any serious deployment must enforce:

Auth and identity: every request carries an API key resolvable to a tenant + user + team.
Routing by model: the tenant asks for model: "gpt-4o"; the gateway decides whether it goes to OpenAI, to Azure OpenAI, or to your vLLM with Qwen3 32B (a cheaper fallback), according to policy.
Rate limiting: RPS per tenant, TPM (tokens per minute), concurrency limits.
Caching of identical responses: 5-30% of RAG queries are repeats; caching saves latency and cost.
OTel emission: every call produces a span with gen_ai.* semantic conventions and tenant_id as an attribute. Covered in the Evals post and MCP observability.
Failover: if vLLM goes down, the gateway redirects to the OpenAI API. If OpenAI rate-limits, the gateway falls back to Anthropic. Configurable policy.

Example of a multi-tenant LiteLLM configuration

# litellm-config.yaml — simplified example
model_list:
 - model_name: llama-3-70b
 litellm_params:
 model: openai/llama-3-70b
 api_base: http://vllm-llama3-70b.inference/v1
 api_key: os.environ/VLLM_API_KEY

 - model_name: qwen3-32b
 litellm_params:
 model: openai/qwen3-32b
 api_base: http://vllm-qwen3-32b.inference/v1
 api_key: os.environ/VLLM_API_KEY

 - model_name: gpt-4o
 litellm_params:
 model: openai/gpt-4o
 api_key: os.environ/OPENAI_API_KEY

router_settings:
 routing_strategy: usage-based-routing-v2
 fallbacks:
 - llama-3-70b: [qwen3-32b, gpt-4o]  # if vLLM goes down, fall back to the external one

general_settings:
 master_key: os.environ/LITELLM_MASTER_KEY
 database_url: os.environ/DATABASE_URL  # Postgres for budgets/keys

# Hierarchy: Organizations → Teams → Users → API Keys
# Created via the API, not in static YAML

Creating a team with a budget:

curl -X POST http://litellm/team/new \
 -H "Authorization: Bearer ${LITELLM_MASTER_KEY}" \
 -d '{
 "team_alias": "soporte-chat",
 "max_budget": 500, # 500 USD/month
 "budget_duration": "30d",
 "tpm_limit": 100000, # 100K tokens/min
 "rpm_limit": 1000, # 1000 requests/min
 "models": ["llama-3-70b", "qwen3-32b"] # access to these
 }'

And the team’s API key:

curl -X POST http://litellm/key/generate \
 -H "Authorization: Bearer ${LITELLM_MASTER_KEY}" \
 -d '{
 "team_id": "<team-id>",
 "duration": "30d",
 "metadata": {"environment": "production", "app": "support-bot"}
 }'

That API key is what the tenant uses. Every request made with it consumes from the team’s budget. When it runs out, LiteLLM returns HTTP 429 with a description.

Layer 2 — Policy & Quota Plane: what each tenant can do

The gateway is where enforcement happens. Policy is what gets enforced. Five axes of multi-tenant policy:

Technical quotas

TPM (tokens per minute): the hard consumption limit. For a Llama 3 70B at TP=5, ~3000 sustained output tokens/s = 180K aggregate TPM. If you have 10 tenants, assign 18K each as the ceiling.
RPS / RPM: load control, not consumption control. A 4K-token session counts as one request; so does a batch of 100 mini-completions. Useful against abuse.
Concurrency: how many simultaneous active requests per tenant. Important for latency SLAs: 100 RPS with concurrency=50 works out at 2 seconds per request.

Economic budgets

Monthly per tenant: a hard cap in USD.
Daily and hourly: soft caps to avoid a runaway in a single day.
Per project / API key: fine granularity within a single tenant.

LiteLLM has a max_budget field at every level of the hierarchy (organization, team, user, api key). Budgets are inherited and constrained downwards.

Model whitelist and blacklist

Tenants with critical workloads → only stable models (llama-3-70b, gpt-4o). Research tenants → access to experimental models as well.

Priority classes

Not all requests are equal. Three typical classes:

Guaranteed: workloads with an SLA, latency respected even under pressure.
Best-effort: normal workloads without a strict SLA.
Spot: batches that can wait, evictable if a guaranteed one arrives.

The Token Management in Multi-Tenant AI Inference Platforms paper (2026) formalises this with a model of token pools per priority class that has started to be adopted in production. It maintains guaranteed P99 latency for guaranteed workloads even under overload, with selective throttling of spot.

Admission control

Before accepting a request: is there capacity? If not, return 429 immediately instead of queueing and degrading everyone. It is the most underrated operational discipline. A cluster with admission control done properly has predictable latency; without it, catastrophic degradation when the peak arrives.

The typical pattern in 2026

# Conceptual policy for a "soporte-chat" tenant
tenant: soporte-chat
quotas:
 tpm: 50000
 rpm: 500
 max_concurrency: 30
budget:
 monthly_usd: 800
 alert_thresholds: [0.5, 0.8, 0.95] # warn when you get there
models_allowed:
 - llama-3-70b
 - qwen3-32b
priority: guaranteed
fallback_on_overload:
 - qwen3-32b  # if guaranteed fills up, fall back
 - gpt-4o-mini  # last resort, external model

Layer 3 — Isolation Plane: isolating workloads physically

This is the technically densest layer. You have a node with several H100 SXM GPUs interconnected by NVLink. How do you partition them between tenants?

MIG (Multi-Instance GPU) is the strongest isolation. It partitions the GPU into up to 7 instances with physically separate HBM memory and dedicated compute units (SMs). Tenants in different MIG instances cannot touch each other: one workload does not consume memory another needs, one does not degrade another’s throughput. Hardware isolation. Available on A100, H100, B100 and B200.

MPS (Multi-Process Service) is soft. Several processes share the GPU concurrently and NVIDIA distributes SMs according to usage. Good performance if all the processes are yours and you trust them. Worse for multi-tenant setups between customers who do not know each other, because a noisy process can degrade the others.

Time-slicing is the simplest: the GPU is assigned alternately, slot by slot, to different processes. Much worse latency (waits between slots); not recommended for production workloads with an SLA.

The choice for multi-tenant in 2026

According to the enterprise adoption survey: 80% use MIG for untrusted multi-tenant (different customers who do not know each other) and MPS for trusted environments (processes from the same team) where you want to maximise throughput. Time-slicing is only used in dev/staging so that every developer can touch a GPU without the cost of exclusivity.

An important limitation of MIG: it isolates compute and HBM memory, but the PCIe path remains shared. For PCIe-bound workloads (a lot of host↔device traffic), tenants in different MIG instances can still affect each other. For LLM inference the main path is HBM, so this is rarely a problem. But it is worth knowing.

MIG partitions on the H100

An H100 (80GB HBM3) can be partitioned into fixed profiles:

Profile	SM	Memory	Max instances per GPU
1g.10gb	14	10 GB	7
1g.20gb	14	20 GB	4
2g.20gb	28	20 GB	3
3g.40gb	42	40 GB	2
7g.80gb	98	80 GB	1 (the whole GPU)

For a mid-scale cluster with NVLink, MIG has a fundamental problem: when you partition with MIG, NVLink between GPUs is disabled. An H100 in MIG mode does not take part in multi-GPU tensor parallel. If you are going to serve a large model with tensor parallel (Llama 3 70B with TP=4 or TP=8, for instance), those GPUs have to be whole, without MIG.

That defines the architectural decision. There are two main approaches:

Approach A — A shared large model with quotas at the gateway

All the node’s GPUs serve a single large model with tensor parallel spanning the whole node. All tenants share that instance. Isolation happens at the gateway layer (quotas, rate limiting) and the policy layer (priority classes). The cluster’s kernel is one enormous vLLM instance with --max-num-seqs=128 or similar; internally, vLLM shares GPU time between the active requests with continuous batching.

Advantages: you use all the GPUs to the full, NVLink stays active, better KV cache utilisation. Disadvantages: soft isolation. A tenant that saturates does not degrade the others directly (vLLM batches), but it does compete for batch slots. You need serious priority classes.

Approach B — Dedicating GPUs per model / tenant

You split the GPUs into pools dedicated to different models. Examples on an 8-GPU node:

4 GPUs: a large 70B model with TP=4.
2 GPUs: a mid-sized 32B model replicated (2 independent instances) for tenants with a strict SLA.
2 GPUs: miscellaneous workloads (smaller models, experimentation).

Advantages: physical isolation between critical models / tenants. Disadvantages: worse aggregate utilisation; some GPUs idle while others saturate.

Approach C (advanced) — MIG on some GPUs plus dedication for the rest

If you have small workloads (4B, 7B models), you can use MIG on 1-2 GPUs to serve them and dedicate the rest to tensor parallel for the large model. It combines strong isolation for the small workloads with full use of NVLink for the large model.

The operational choice: start with A, move up to C if you need to

In most deployments, Approach A (a shared large model plus quotas) is the right starting point. Utilisation is better, operation is simpler, and the gateway’s soft isolation works for reasonable workloads.

When there is a tenant with a strict SLA that cannot tolerate competing with others, you move to Approach B for that tenant in particular (dedicating GPUs to an instance of the model just for them), keeping the rest of the cluster shared.

Approach C is for when you have 10+ tenants with very heterogeneous profiles.

Isolation at the Kubernetes level

Independently of GPU isolation, pod-level isolation is applied in K8s:

Namespaces per tenant: tenant-soporte, tenant-legal, and so on.
ResourceQuotas and LimitRanges: CPU/memory limits per namespace.
NetworkPolicies: tenant A cannot talk to tenant B’s namespaces.
K8s PriorityClasses: classes with a numeric value defining preemption order if a more critical pod arrives.
PodDisruptionBudgets: how many pods of each deployment can go down simultaneously.

Layer 4 — Observability Plane: seeing what happens per tenant

The fourth layer: observability with a tenant dimension. Without it you cannot do cost attribution, you cannot debug incidents for a single tenant, you cannot show dashboards to stakeholders.

The four mandatory properties

1. tenant_id on every span. The AI gateway resolves the API key and attributes a tenant_id. That ID is propagated via params._meta or OTel headers to every downstream component (vLLM, retrieval, MCP servers, tools). Any span in any system carries that label. It is what lets you reconstruct tenant-specific traces.

2. Metrics labelled per tenant. gen_ai.usage.input_tokens{tenant="soporte-chat"} or equivalents. Prometheus, Grafana, groupable by tenant.

3. Real cost attribution. The sum of tokens × cost/token per tenant gives the cost. For self-hosted vLLM, the cost is per GPU hour plus the proportional share of tokens (you can compute an equivalent cost per 1k tokens).

4. An immutable audit log. Every API key used, every model invoked, every quota change, every budget exceeded. For compliance.

Showback vs chargeback

An important FinOps distinction that gained clarity in 2026:

Showback: visibility without consequence. “Support team, you have consumed 623 USD this month on LLMs”. Information, not an invoice. It lets you spot abuse without penalising anyone before the team understands what is going on.

Chargeback: the cost is charged to the team’s budget. When it runs out, it runs out. It changes behaviour.

The practice that works: 6-18 months on showback while tags are calibrated, misattributions are identified and teams are trained. Then chargeback, once the numbers are credible. Launching chargeback on day 1 when the costs are still dirty creates an immediate political fight; launching showback prepares the ground so that chargeback lands in an orderly way.

Only 14% of organisations have chargeback active according to a recent survey, which indicates this is still mostly showback in real production.

Tools

Kubecost: cost allocation per namespace, deployment and pod on Kubernetes. For the cost of the shared GPU, allocate proportionally to the tokens consumed per tenant.
Finout: a FinOps platform that combines cloud bills and LLM API costs in a unified view with virtual tagging.
Langfuse: already covered. Cost tracking per trace, groupable by user or session metadata.
LiteLLM native tracking: LiteLLM’s master DB keeps running spend per team, user and API key, accessible via the API or the UI.

The minimum multi-tenant dashboard

Any platform should have:

Summary per tenant: monthly spend, current RPS, TPM consumed, % of budget spent, active sessions.
Top users within each tenant (for detecting internal abuse).
p95 latency per tenant: SLA tracking.
429 / 503 errors: how many requests are being rate-limited or rejected because of overload.
Cost trend: the monthly trajectory with a projection.
Drift per tenant (from the post-tracing series): if a tenant starts getting worse results, raise an alert.

Sizing on mid-scale GPU clusters: concrete decisions

Let us get down to hardware. As a reference we take a node with N H100 SXM (between 4 and 8) with NVLink/NVSwitch, 80 GB HBM3 each. That gives between 320 GB and 640 GB of aggregate VRAM. Inter-GPU connectivity is 900 GB/s (NVLink 4) or 600 GB/s (NVLink 3) depending on generation. HBM bandwidth per GPU is 3.35 TB/s.

Default decisions

Start with Approach A: all the node’s GPUs serving a single large 70B model in BF16 with tensor parallel = N. Expected real capacity (computed for a standard 8-GPU HGX node as an example; it scales roughly linearly with N):

Model VRAM (70B BF16): ~140 GB (≈ 17.5 GB/GPU at TP=8).
vLLM overhead + activations VRAM: ~10 GB/GPU.
VRAM free for KV cache: ~52 GB/GPU. On an 8-GPU node that is ~416 GB aggregate; on a 4-GPU one, ~210 GB.
With --kv-cache-dtype=fp8 and a 70B GQA model: ~320 KB/token.
Aggregate cache capacity (8-GPU node): ~1.3M tokens to distribute between simultaneous sessions.

That translates into throughput and concurrency (indicative figures for an 8-GPU node):

Simultaneous sessions	Average context per session	Aggregate throughput (tokens/s)
32	16K	~5000
64	8K	~8000
128	4K	~12000

Typical latencies: TTFT ~150ms at low traffic, TPOT ~15-20 ms/tok. Under high concurrency, TTFT rises to ~500ms if the queue is saturated.

An example tenant scheme

A cluster with 4 tenants and a research pool:

Tenant	TPM cap	RPM cap	Concurrency	Budget	Priority	Models
Support chat	80K	800	50	1500 USD/month	Guaranteed	llama-3-70b, qwen3-32b
Legal RAG	30K	200	15	600 USD/month	Guaranteed	llama-3-70b
Code agent	50K	300	25	1200 USD/month	Best-effort	llama-3-70b, qwen-coder
Data extr. batch	40K	1000	40	400 USD/month	Spot	llama-3-70b, qwen3-32b
Research / notebooks	10K	100	5	200 USD/month	Spot	all

TPM total: 210K. Aggregate cluster capacity: ~180K sustained TPM. It is overcommitted by about 15%, assuming not every tenant hits the ceiling at the same time. That is normal and desirable; if they all do it at once, the priority classes degrade in an orderly way.

When to add hardware

Signals that the node has become too small:

Sustained p95 TTFT > 500 ms during peak hours → the queue is building up.
vllm:num_requests_waiting constantly > 20 → admission control starting to reject.
Sustained GPU utilisation > 80% at critical hours without dropping in quiet hours → there is no headroom.
429 rate on guaranteed tenants > 1% → the platform is breaking its SLA in production.

When several of these hold, the natural next step is to add another HGX node with internal NVLink and stand up a second vLLM instance of the same model. The gateway load-balances between the two instances. Aggregate throughput doubles; latency stays the same.

Common operational traps

A gateway without auth: a backdoor into the cluster

Your vLLM sits on a ClusterIP Service and the main app talks to it. Some tenant discovers the endpoint directly and hits it without going through the gateway. Quotas and costs are silently evaded. Use a strict NetworkPolicy: only the gateway can talk to the vLLM Services; the rest of the cluster cannot.

MIG and NVLink are incompatible

You enable MIG on a GPU thinking you will get isolation plus multi-GPU; you discover that MIG disables NVLink. Any large model with TP becomes unusable. Decide MIG vs NVLink globally per cluster, not per individual GPU.

Quotas pinned to the cluster’s ceiling

You add up the TPM of all the tenants and it comes to exactly the cluster’s capacity. When two tenants peak at the same time, both wait or one gets rejected. An overcommit of 10-20% is healthy (it assumes they do not all peak at once); more than that is dangerous.

No multi-tenant observability from day 1

You launch with quotas and isolation but without tenant_id on spans. Three months later, your CFO asks “how much does the support agent cost versus the legal one?” and you cannot answer. OTel with a mandatory tenant_id from the first version, even if there are no dashboards yet; having the data is worth more than having perfect dashboards with no data.

Showback that never reaches chargeback

You have been on showback for 18 months, the teams know the numbers, nobody changes behaviour. Without the pressure of real chargeback, the incentive dilutes. Set an explicit calendar for the transition to chargeback, with an owner and a deadline.

Non-whitelisted models eating the budget

A team discovers that LiteLLM has gpt-4o configured. They use it without permission. The budget burns on an external API when the idea was to use the cheap self-hosted one. Keep an explicit per-team whitelist of accessible models.

Badly calibrated priority classes

Everyone declares themselves “guaranteed”. At the first peak, there is nothing left to degrade and everything suffers. Priority classes only for critical cases, with justification. The majority should be best-effort.

No failover from the gateway

Your vLLM goes down. The gateway has no fallback configured and returns 503 to every tenant. Configure a fallback to another model, ideally an external one (OpenAI) for guaranteed workloads, even if it costs more per hour. Availability is worth more than cost per hour.

An operational roadmap for getting started

If you start from scratch with an empty GPU node, the minimum order is as follows. Each milestone is a day’s work with slack, not a tight one:

Day 1-2 — Base K8s infrastructure. NVIDIA GPU Operator + nvidia-device-plugin + dcgm-exporter + cluster-default NetworkPolicies. Validation: a basic pod with nvidia.com/gpu: 1 gets scheduled.

Day 3 — vLLM with a large model and tensor parallel across the whole node. The vLLM Production Stack Helm chart (or bare vLLM manifests). Model weights on a shared PVC (CephFS or NFS). Validation: a curl request against the internal Service responds.

Day 4 — AI Gateway: LiteLLM. Helm chart, Postgres for budgets, master key, a first model_list pointing at vLLM. Validation: an OpenAI-compatible request through LiteLLM responds with the same content as vLLM directly.

Day 5 — Basic multi-tenancy. Create teams, API keys, budgets, model whitelists. Test with two teams. Validation: the second team, using the model it does not have whitelisted, gets a 403.

Day 6 — Minimum observability. Prometheus + Grafana scraping vLLM and LiteLLM. A dashboard with TTFT, TPOT, throughput, num_requests_waiting and budget_consumed_per_team. Validation: visible in Grafana with real data.

Day 7-8 — Pilot customer. One real tenant (ideally a controlled internal one) starts using it. Measure real latencies, discover the first operational incidents.

Day 9-10 — Tuning. Adjust --max-num-seqs, --gpu-memory-utilization, priority classes and quotas based on what the pilot taught you.

Day 11-14 — Onboarding the second tenant plus iteration. Repeat. Every new tenant onboarded reveals new cases.

After two weeks you have an operational platform with two real tenants and data to decide whether it is ready for more. The line of progress from here on is horizontal (more tenants) until you saturate; from then on, vertical (more hardware).

What we have not covered (upcoming posts)

Continuous fine-tuning in production (post 6, already decided): LoRA/QLoRA/DPO, dataset curation, eval gates, A/B versioning with real traffic between model versions.
Constitutional AI and alignment at runtime: an option still on the table.
Edge LLMs: when an H100 cluster is too expensive for a specific workload, distilled models running on NPUs or consumer GPUs.
GPU networking deep dive: NCCL, InfiniBand, GPUDirect, RDMA. For multi-node clusters with cross-host tensor parallel.

References

Multi-tenancy and GPU isolation:

Multitenant GPU Infrastructure: 4 Powerful Design Rules — survey of enterprise patterns.
Run Multiple LLMs on One GPU: MIG, Time-Slicing, and MPS Guide (Spheron).
A Practical Guide to GPU Partitioning with MIG (Medium).
GPU Partitioning for AI Workloads: NVIDIA MIG with SUSE Virtualization (KubeCon EU 2026).
Predictable LLM Serving on GPU Clusters (arxiv 2508.20274).
Token Management in Multi-Tenant AI Inference Platforms (arxiv 2603.00356) — paper on priority + admission control.

AI Gateways:

Multi-tenant FinOps:

Cross-references:

Previous posts in series 4: MLOps for LLMs overview, RAG over Kafka, The six-stage pipeline, PostgreSQL + Qdrant.
Relevant posts from the inference series: vLLM on Kubernetes — the multi-GPU HGX node scenario we develop here. LLM operators on K8s — the vLLM Production Stack and OME that the gateway can direct.
Observability: Evals, MCP observability, eBPF + drift.

The six-stage LLMOps pipeline: global architecture and a deep dive into every component

Thu, 21 May 2026 06:30:00 +0200

TL;DR

The first two posts in the series established the LLMOps landscape and went down into the detail of the data pipeline with Kafka. This post does the intermediate zoom: it draws the complete map of the system, a global architecture of a modern LLMOps setup with all the pieces the field has settled on in 2026, and goes into depth on each of the six canonical stages of the pipeline: Data, Tune, Eval, Deploy, Observe, Retrain. For each stage we give the operational sub-tasks, the dominant tools, the design decisions that always show up, and the specific traps that are seen repeatedly in production. And, most important operationally: every stage carries a “you are here” mini-map over the cycle, which will be reused in any later post in the series to place the reader. The idea: that anyone reading a post about fine-tuning, about prompt versioning, about eval gates or about drift detection can look at the mini-map and know immediately which piece of the larger system they are thinking about that day.

This is the third post in the MLOps series specific to LLMs. Previous ones: Landscape 2026 and RAG over Kafka. Here we move from “the what” and “one piece” to the whole map, with detail per stage.

The global architecture: the master map

Before going down into each stage, let us fix the whole map. What follows is the reference drawing of a production LLMOps system in 2026, with every component the field has settled on in its place:

What you see: the six large boxes are the stages; the solid arrows are the flow of the pipeline; the dashed arrow that goes from Retrain to Data is the feedback cycle that turns LLMOps into a living process rather than a project that ends. The grey band at the foot holds the cross-cutting components (observability, prompt versioning, MCP, gateway, schema) that run through every stage and connect to each one.

Three quick readings of the map:

Horizontal, top: the happy path, data → tune → eval. What happens while you prepare the model.
Horizontal, bottom: the service path, deploy → observe → retrain. What happens once the model is alive.
Vertical: the connection between the two floors. Eval gateway feeds Deploy; Observe feeds Retrain; Retrain returns to Data.

From here on, every stage will include a navigation mini-map (“you are here”) to place you in the complete cycle. Let us go through each one.

Stage 1 — Data: ingestion, transport, versioning, indexing

Operational sub-tasks

The Data stage is the most underrated and the one that blocks the most projects. Its sub-tasks:

Ingestion from heterogeneous sources: OLTP databases (Postgres, MySQL), external APIs, file shares, scraping, SaaS systems, application logs, internal messaging.
Change capture (CDC) in streaming if the data is dynamic. Debezium over Kafka, Flink CDC, modern alternatives such as RisingWave that reads the WAL directly.
Transformation (cleansing, dedup, normalisation, PII sanitisation).
Schema management: schema registry, compatible evolution, backward/forward compatibility.
Versioning of training datasets and golden datasets: DVC + lakeFS (unified in November 2025). Covered in detail in the dedicated data versioning post.
Indexing for RAG: chunking, embeddings, writing to vector stores. Covered in depth in the Kafka post.
Materialisation into analytical tables: Tableflow → Iceberg/Delta, for BI consumption and low-latency queries.

Dominant tools

Sub-task	Tools 2026
CDC	Debezium, Flink CDC, RisingWave
Transport	Kafka (Confluent Cloud, Redpanda, pure Apache)
Schema Registry	Confluent Schema Registry, Apicurio
Stream processing	Apache Flink, RisingWave, Kafka Streams
Data versioning	DVC + lakeFS
Vector stores	Milvus, Qdrant, Weaviate, pgvector, LanceDB
Materialised tables	Tableflow → Iceberg/Delta
Batch ETL/ELT (where it applies)	dbt + Snowflake/Databricks

Design decisions

The three decisions that always show up:

Batch vs streaming: the more dynamic the data, the more streaming. For static corpora (manuals that never change) a nightly batch is enough; for transactional data the agent needs to see minute by minute, streaming from day 1.

Embedding model: changing the embedding model invalidates every indexed vector. An architectural decision: pin the model and have an explicit migration plan (the dual-index pattern seen in the Kafka post).

Vector store: pgvector if you already run Postgres and you are below 10M vectors; Qdrant if you want mid-scale simplicity; Milvus if you need billions; Weaviate if you value native hybrid search.

Traps

Hardcoding connections to the source (with no abstraction): when the database changes (version, host, schema), you break the whole pipeline. An adapter layer from day 1.
No schema registry: topics start breaking silently.
Full reindexing whenever something changes: it costs hours or days. Design the dual-index pattern from the start.
Unsanitised PII: the RAG is serving sensitive data without meaning to. Anonymisation in the pipeline, not at consumption time.

Stage 2 — Tune: preparing the model for your case

Operational sub-tasks

Base model selection: Llama, Qwen, Mistral, Gemma, DeepSeek according to licence, size, quality in your domain.
Dataset preparation: train/val/test split, format (chat templates, JSONL), augmentation where it applies.
Adapter configuration: LoRA rank, target modules, alpha; QLoRA if you want to train on a consumer GPU; full fine-tune only if you have the budget.
Training loop: HuggingFace Transformers + PEFT + TRL as the canonical stack; Axolotl or llama-factory as convenience wrappers; Unsloth if you want 2-4× more speed on consumer GPUs.
Hyperparameter sweep: W&B Sweeps, Optuna, Ray Tune.
Checkpointing and resumability: save every N steps, resume from failure.
Promotion: the adapter is promoted to the registry after passing the next stage (Eval).

The three modes of Tune

The detail of the table we saw in the landscape:

Supervised fine-tuning (SFT) with LoRA/QLoRA. You collect (prompt, ideal-response) pairs and apply SFT with cross-entropy loss. The simplest option. The rule of thumb: 300-3,000 well-curated examples are usually more useful than 50,000 noisy ones.

DPO (Direct Preference Optimization) and RLAIF. Instead of “ideal-response”, you collect (prompt, good_response, bad_response) triples and train the model to prefer the good one. More stable than classic RLHF, same objective. It is what most teams use when they go beyond SFT.

Agent training (RFT / Reinforcement Fine-Tuning, pure RLHF). For cases where the model needs to learn multistep trajectories: when to choose tool A over B, when to ask for confirmation, how to decompose a large task. Far more expensive and complex. OpenAI’s work with RFT set the pattern in 2024-2025; in 2026 it is coming out of the experimental phase.

RAG as an alternative to Tune: although conceptually it is another stage (it lives in Data + Deploy), functionally it competes with fine-tuning for many cases. The 2026 verdict: hybrid is the default (60% of deployments), fine-tune for behaviour plus RAG for volatile knowledge.

Tools

Aspect	Tools 2026
Base framework	HuggingFace Transformers, PEFT, TRL
Convenience wrappers	Axolotl, llama-factory
Consumer speed	Unsloth (2-4× speedup on RTX GPUs)
Distributed training	DeepSpeed, FSDP, NeMo Framework
Experiment tracking	MLflow, W&B, ClearML
Adapter registry	Private HuggingFace Hub, MLflow registry
Hyperparameter	W&B Sweeps, Optuna, Ray Tune

Traps

Catastrophic forgetting: very aggressive SFT destroys the model’s general capabilities. Keep a small % of the original dataset or use regularisation.
Overfitting to the golden dataset: the model learns to memorise the eval set. Keep a holdout test set that nobody on the team looks at until the final release.
Train/serve skew: prompts in training with a different format from production. The same chat template in both.
LoRA rank too high: it looks like it improves metrics but it inflates the adapter with no real benefit. Start with r=8 or r=16; raise it only if there is evidence.

Stage 3 — Eval: validating before promoting

Operational sub-tasks

Covered in depth in Evals: the layer after tracing. A structured summary for the pipeline:

Golden dataset curation: 100-500 examples as a minimum, actively maintained with incident cases.
Evaluators: heuristic (regex, length), semantic (embeddings), LLM-as-judge (G-Eval), human (golden labels).
Running in CI: block the merge if critical metrics drop by more than X%.
Running on the platform: over sampled production traffic, persist results, detect long-term regression.
Judge calibration: 85-90% agreement with humans before accepting the judge as productive.
Eval gates: explicit thresholds per metric (faithfulness > 0.85, relevancy > 0.80, and so on).

Tools

CI gates: DeepEval (Apache 2.0, pytest-style), Promptfoo (MIT, CLI), Ragas (RAG-specific), Inspect AI (safety/capability).
Platform: Langfuse (MIT, complete suite), LangSmith (LangChain), Phoenix (ELv2, OTel), Braintrust.
Judges: GPT-4 (expensive but the reference), Claude 3.5 Sonnet, Prometheus (OSS, 0.897 correlation), JudgeLM.

Traps

Aged golden dataset: if it is not updated, it stops reflecting production.
Contaminated judge: the judge knows the dataset (it appeared in its training).
Insufficient sample size: fewer than 50 examples makes differences look like noise.
Runaway costs: G-Eval with GPT-4 over many cases costs thousands of USD a month.
Forgetting the segment: an average of 0.85 can hide a 0.55 in German.

Stage 4 — Deploy: putting the model into production

Operational sub-tasks

Covered in depth in vLLM on Kubernetes and LLM operators on K8s. A summary for the pipeline:

Runtime selection: vLLM (default), SGLang (agents with high prefix caching), TensorRT-LLM (pure latency), llama.cpp (edge).
Operator selection: vLLM Production Stack, KServe, OME (LMSYS), NVIDIA Dynamo, llm-d (CNCF).
Serving configuration: --tensor-parallel-size, --kv-cache-dtype=fp8, --enable-prefix-caching, --enable-chunked-prefill, --gpu-memory-utilization=0.92.
Routing between models: LiteLLM as the multi-provider abstraction.
Release strategy: canary (1% → 10% → 100%), blue-green (all or nothing with fast rollback), shadow (eval in parallel without affecting users).
Autoscaling with LLM metrics: KEDA + Prometheus over vllm:num_requests_waiting or equivalent.
Gateway / Inference Extension: Gateway API Inference Extension once it is GA.

Dominant tools

Serving engines: vLLM, SGLang, TensorRT-LLM, llama.cpp, MLX.
Operators: OME, vLLM Production Stack, NVIDIA Dynamo, llm-d, KServe.
Routing: LiteLLM (100+ providers), OpenRouter (managed), LangChain Router.
GPU primitives: NVIDIA GPU Operator, LeaderWorkerSet (LWS) for multi-pod tensor parallel, KEDA for autoscaling.

Traps

Naïve rolling update that cuts sessions: maxUnavailable: 0, maxSurge: 1 and terminationGracePeriodSeconds: 120+.
A short readiness probe that kills pods while they load: startupProbe with failureThreshold: 60.
HPA on CPU% with no LLM metrics: vLLM batches internally, one replica serves dozens. KEDA on queue depth.
Unquantised KV cache: --kv-cache-dtype=fp8 is almost always worth it.
Tensor parallel on GPUs without NVLink: all-reduce saturates PCIe and throughput collapses.

Stage 5 — Observe: seeing what happens in production

Operational sub-tasks

This is the stage we have covered most deeply in previous series: the whole eBPF series (4 posts) and the whole post-tracing series (4 posts) deal with sub-tasks of Observe. A structured summary:

Tracing: OpenLLMetry/Traceloop, Langfuse, Phoenix, LangSmith. Spans with OTel GenAI semantic conventions (gen_ai.*, mcp.*).
Metrics: Prometheus + Grafana. TTFT, TPOT, throughput, queue depth, KV cache usage, cost per tool.
Active guardrails (not just eval): NeMo Guardrails with 5 types of rails, multimodal Llama Guard 4, Llama Prompt Guard 2 (86M/22M), LLM Guard.
eBPF observability (zero-instrumentation): Hubble (network), Tetragon (process/syscall), AgentSight (LLM agent with SSL uprobes + stdiocap MCP).
eBPF in the local engine (inference): ProfInfer-style with uprobes in llama.cpp / vLLM / libcudart.
Drift detection: Evidently AI, NannyML, WhyLabs. KS, PSI, MMD over embeddings.
MCP observability: OpenTelemetry GenAI MCP semantic conventions, trace propagation via params._meta, a centralised MCP Gateway.

The four mandatory metrics

Out of everything covered, the four any minimal dashboard must have:

TTFT p50/p95 (time to first token) — what the user perceives.
TPOT p50/p95 (time per output token) — streaming speed.
Throughput (aggregate tokens per second) — capacity planning.
Queue depth (vllm:num_requests_waiting) — the leading indicator.

To these you add, per domain:

For RAG: faithfulness rolling mean, retrieval hit rate.
For agents: tool call accuracy, multi-step task completion.
For multi-tenant: cost per tenant, p95 latency per tenant.

Traps

Cardinality in Prometheus: metrics carrying every K8s label blow up.
Tracing with no sampling: storage grows out of control.
Guardrails permanently in monitoring mode: they never reach enforce.
Drift with no alerts: you spot drift on the dashboard once a month; meanwhile the problem has been running for weeks.
OTel with no propagation: MCP, Tetragon and AgentSight spans left disconnected.

Stage 6 — Retrain: closing the loop

Operational sub-tasks

This is the stage most neglected in GenAI projects. Closing the loop turns LLMOps into a living practice; not closing it leaves it as a project that ages.

Explicit feedback: thumbs up/down in the UI, annotations by power users, forms for “what went wrong”.
Implicit feedback: anomalous latency, abandonment rate, user retries, aborted sessions.
Incident triaging: classify incidents by root cause (model issue, retrieval issue, prompt issue, infra issue).
Dataset enrichment: add to the golden dataset the cases where the system failed, with the correct answer labelled by a human.
Retrain cadence: quarterly by default, incident-driven when a problematic pattern goes over a threshold.
Promotion: the new model/adapter goes through the Tune → Eval → Deploy stages, with eval gates that compare against the model in production.

The two cadences

Scheduled retrain (quarterly or half-yearly): an established process. It lets you plan capacity, budget and risk. The default.

Incident-driven retrain: when a serious incident (detected drift, a failing segment, a prompt injection attack) crosses a threshold, a mini-cycle fires. More expensive but necessary for critical cases.

Dominant tools

Annotation and feedback collection: Langfuse (built-in UI), Argilla (OSS), Label Studio.
Dataset enrichment: pipelines in Airflow or Argo Workflows.
Triaging: Langfuse dashboards with filters on traces with a low eval.
Promoting candidates: MLflow model registry stages.

Traps

An open loop: production does not inform the dataset; the model never improves.
Human feedback gets lost: thumbs down with no structured capture channel.
Undefined cadence: “we retrain when it is needed” → nobody ever retrains.
No holdout test set: the golden dataset is enriched with the same cases used to evaluate; eval measures memorisation.
Promotion with no gates: the new model enters production without passing the checks the previous models passed.

The complete cycle: how the stages fit together

Now that we have seen each stage separately, the key insight is how they hook into each other. Five emergent properties of the cycle:

1. Data is the raw material of every stage. Tune reads from the golden dataset. Eval reads from the eval dataset. Deploy reads from the RAG (vector store). Observe produces new data. Retrain creates new datasets. The Kafka log is the gospel of the whole system (post 2 in the series).

2. Eval is the bidirectional gatekeeper. Before Deploy: it blocks the release if the model regresses. After Observe: it feeds Retrain by identifying the worst-scoring cases. The quality of the eval determines the quality of the whole cycle.

3. Observe feeds Retrain and Eval simultaneously. Traces produce metrics for Observe; problematic traces are annotated and go to the dataset; the new cases enrich the eval golden set. Observe is the operational source of truth.

4. The cross-cutting components (the grey band on the map) are not a stage, they are an infrastructure. OpenTelemetry, prompt versioning, MCP gateway, model gateway, schema registry. Badly configured, every stage suffers separately. Well configured, the stages integrate without friction.

5. The cycle is not strictly sequential, it is concurrent. At any given moment the system has: requests being served (Deploy + Observe), a new version in training (Tune), continuous eval in CI (Eval), data arriving from CDC (Data), incident analysis (Retrain). Every stage is alive at once.

Cross-stage traps: things that break the whole system

There are errors that do not belong to a stage but to the interfaces between stages. The most common ones:

Train/serve skew

The exact format of the prompt in training is different from the one in production. The result: the model trained to answer <|im_start|>user\n...\n<|im_end|> receives User: ...\nAssistant: in production and performs worse. Solution: extract the chat template into a shared library used by the Tune pipeline and the Deploy one.

An eval that does not reflect production

Your golden dataset is careful questions; production is real questions with typos, mixed languages, and so on. Eval passes at 95%, production performs at 70%. Solution: continuously enrich the golden set with real samples.

Drift with no response pipeline

You spot drift on the Observe dashboard; nobody has a defined workflow for what to do. Solution: every drift alert must have a clear runbook: investigate, classify, act (retrain, adjust the prompt, widen the retrieval).

Schema break cascade

You change the schema at the OLTP source; Debezium reflects it; the Flink job breaks; the embedded topic stops updating; the vector store ages; the RAG answers over old data. Three stages affected by one change in Data. Solution: mandatory backward-compatible schema evolution, contracts between producers and consumers.

No observability of the pipeline itself

The LLMOps pipeline is a complex system. If it has no observability of its own (how long training takes, how many jobs fail, how many re-embeddings run), debugging failures is an exercise in spelunking. Solution: OTel over the pipeline itself, not just over the LLM calls.

Invisible vendor lock-in

Pipelines written against LangChain, prompts stuck in LangSmith, embeddings in Pinecone, the model in OpenAI. Migrating is a months-long project. Solution: LiteLLM and OpenLLMetry abstractions, vendor-neutral from the start.

What comes in the following posts

Post 4 — PostgreSQL + Qdrant in the ingestion stage — the first post that applies the “you are here” pattern over the Data stage. Synchronisation patterns (outbox + CDC), microservices architecture, deployment manifests.
Upcoming posts — still to be decided: the cluster as a multi-tenant platform, Constitutional AI / runtime alignment, continuous fine-tuning in depth, edge LLMs.
In any later post in this or future series, the “you are here” mini-map will tell you which stage of the cycle the topic fits into. If you read a post about quantization, you will know you are in Deploy. If you read one about evaluator ensembles, you will know you are in Eval. If you read one about RAG over Iceberg, you will know you are in Data.
If you want to see the whole pipeline in action following a single real request, the synthesis post Anatomy of an LLM request in production does exactly that: it rewinds a request back to the data that trained it 90 days earlier and follows it forward to the feedback that will reappear in the next Retrain cycle, crossing the six stages and the two cross-cutting components in one coherent story.
If what interests you is comparing how each stage is built in open source against the hyperscalers, the post The parallel catalogue: the six LLMOps stages in OSS and in AWS / GCP / Azure makes the vertical cut: for every stage, which tools the blog’s reference OSS stack uses and what the cloud equivalents are, with summary tables, gap identification and the multi-tenant chatbot ported to an AWS stack as a concrete example.
If you want the OSS toolbox piece by piece, the post The OSS catalogue for LLMOps in six stages: entry by entry does the zoom in: ~150 words of description per core tool (what it does, how it differs from its alternatives, licence and governance, typical gotcha), a decision matrix per stage, a diagram of the connected OSS stack and a master table of licences and EE / SaaS offerings.
If what you are missing is the common vocabulary that runs through the six stages, the post Ontologies and knowledge graphs in LLMOps walks the six stages from the perspective of shared formal nomenclature: how TBox + ABox + SHACL + SKOS change the operation of Data, Train/Adapt, Eval, Deploy, Observe and Govern, the GraphRAG 2026 landscape (Microsoft GraphRAG v2, LightRAG, HippoRAG 2, KAG/OpenSPG), vertical ontologies actually deployed (FIBO, SNOMED CT, schema.org, ENS, EU AI Act) and an open source stack with its licensing caveats (Neo4j Community GPLv3, KuzuDB upstream archived in October 2025).

References

Foundations:

Per stage (entries from the blog series):

Data: RAG over Kafka — technical architecture and the dedicated post on data versioning with DVC and lakeFS — the four artefacts to version separately (training, RAG corpus, golden eval, enriched retrain), schema contracts, end-to-end lineage from dataset to trace, and why a golden set without a strict holdout measures memorisation.
Tune: partly covered in Landscape 2026; a deeper treatment in post 4 if continuous fine-tuning is chosen.
Eval: Evals: the layer after tracing.
Deploy: vLLM on Kubernetes and LLM operators on K8s.
Observe: the whole eBPF series and the whole post-tracing series.
Retrain: the dedicated post on how to close the loop — feedback capture (explicit + implicit), triage by root cause, dataset enrichment with human annotation (Argilla / Label Studio), scheduled vs incident-driven cadences, governed promotion with eval gates.

Cross-cutting components:

Prompt versioning: the dedicated post with Langfuse and MLflow Prompts — the three-primitive pattern (immutable version, mutable label, cache), eval gates on promotion, and per-request traceability.
MCP: deep MCP observability.
Drift detection: eBPF + drift detection.
Local inference: PagedAttention deep dive, KV cache.

Frameworks and tools referenced:

MLflow, W&B, Kubeflow, ZenML, BentoML, Metaflow.
HuggingFace PEFT, TRL, Axolotl, Unsloth.
DVC + lakeFS.
Langfuse, Evidently AI, Phoenix.
vLLM Production Stack, KServe, OME, NVIDIA Dynamo, llm-d.

RAG on Kafka: a technical reference architecture for streaming datalakes, with fresh embeddings and vector stores that are always up to date

Thu, 21 May 2026 06:00:00 +0200

TL;DR

The piece that blocks enterprise GenAI projects most in 2026 is not the model, nor even the guardrails: it is data ingestion for RAG. Companies hold valuable information in OLTP databases, in operational logs, in SaaS systems, and all of it is quietly changing every second. Batch RAG systems that reindex every night arrive late (the model’s answer is backed by a snapshot from 18 hours ago) and open the door to operational hallucinations even when the retriever is perfect. The dominant answer in production in 2026 is to build the RAG piece on top of Kafka as the source of truth: an immutable log, massive throughput, managed schema evolution, and a mature stream processing ecosystem (Flink, Kafka Streams, RisingWave) that makes it possible to transform and embed events as they happen, delivering them to vector stores (Milvus, Qdrant, Weaviate, pgvector) in milliseconds. The canonical pattern: source → CDC with Debezium → Kafka topics → Flink SQL with an embedding UDF → sink connector to a vector store → serving with vLLM or equivalent. The 2026 developments that change the game: Confluent Tableflow turns Kafka topics into Iceberg/Delta tables automatically (read from Snowflake/Databricks/Trino with no ETL, 30-50 % lower TCO); native Flink SQL brings openai_embedding() and vector search integrated with Cosmos DB and Amazon S3 Vectors; Confluent’s official MCP server lets AI agents query Kafka/Flink/Tableflow in natural language. This post develops the end-to-end architecture with manifests, Flink SQL code and concrete numbers.

This is the second post in the MLOps series specific to LLMs. The first (2026 landscape) set out the framework. Here we go down to the most operational piece of the stack: how a real enterprise system connects to an LLM agent while keeping RAG fresh without falling into explosive complexity.

The analogy: Kafka as the “single source of truth”

Anyone who has spent time in distributed systems has seen the pattern again and again: an immutable, append-only, replicated, time-ordered log has become the canonical primitive for rebuilding complex systems. DBAs know it as the write-ahead log (PostgreSQL WAL, MySQL binlog). Event system developers know it as event sourcing. Data architects know it as the Kappa architecture. Kafka is the massive, distributed, mature implementation of that primitive: a log that lives on disk, partitioned to scale, replicated for durability, retained by time or size, readable from any point in history.

When you think about RAG, this is exactly what you need. A well-designed RAG system has two critical questions: how do you keep the index fresh? And how do you rebuild the index when something breaks? Kafka answers both naturally: fresh because every change at the source is published as an event to the log and the pipeline processes it in milliseconds; rebuildable because the whole log is there: you drop the vector store, you have the Kafka topic from offset 0, and you build the index back exactly as it was.

There is a second layer to the analogy as well. For a modern GenAI architecture, Kafka plays the role of the WAL of the entire system. Just as the Postgres WAL is the gospel of the database’s state (if you lose the DB but keep the WAL, you can rebuild it), the Kafka log is the gospel of the state of the business as a whole: orders, users, transactions, documents. Connecting your AI agent to Kafka means connecting it to the real pulse of the system, not to stale snapshots.

The static RAG problem

Before presenting the architecture, it is worth pinning down what concrete problem we are solving. The antipattern that trips up most GenAI projects:

A team builds RAG on a static dataset: it dumps Confluence documents, product PDFs, database snapshots.
It embeds the lot with a nightly cron that regenerates the index every 24 hours.
It ships the product.
Day 2: a user asks about a change that happened two hours ago. RAG does not have it; the model answers about the old version.
The team adds fragile logic: “if the query mentions a recent date, escalate to a human agent”.
Day 30: the dataset has moved so much that half the RAG is out of date. The team decides to refactor and migrate to streaming.

It is the repeated story of so many projects that the ecosystem has learned the lesson: streaming from day 1, even when the volume is low. The operational complexity of a well-designed streaming pipeline is constant; the complexity of migrating from batch to streaming on a live project is enormous.

From Lambda to Kappa to Streaming RAG

Three architectures in historical order:

Lambda (the classic big-data design from 2014): two parallel pipelines, one batch for accuracy and one streaming for freshness. The query combines both. It works, but it means maintaining two pipelines.

Kappa (Jay Kreps, 2014, mainstream since 2020): a single streaming pipeline. Batch is a special case of streaming (reprocessing from the beginning). It simplifies a great deal.

Streaming RAG (emerging in 2025-2026): a specific variant of Kappa where the pipeline’s output is embeddings indexed in a vector store that the LLM queries at runtime. The Kafka log is the source of truth, the vector store is a queryable projection.

The mental conversion: think of the vector store as the materialised view of the Kafka log. If the view is corrupted, you rebuild it from the log. If you want a new view (a different embedding model, a different chunking strategy), you create another consumer of the log and build a second view in parallel.

The reference architecture

On to the diagram. I am going to present the canonical architecture that stabilised in 2026, showing where each component fits:

[OLTP DB (Postgres)] [Other sources]
│ │
│ WAL via logical decoding │
▼ ▼
┌──────────────────────────────────────────────────────────┐
│ Debezium / Kafka Connect (Sources) │
└──────────────────────────────────────────────────────────┘
│
▼ produces events
┌──────────────────────────────────────────────────────────┐
│ Kafka cluster │
│ ┌───────────────────────────────────────────────────┐ │
│ │ topic: orders.raw (3 partitions, RF=3) │ │
│ │ topic: users.raw (3 partitions, RF=3) │ │
│ │ topic: documents.raw (6 partitions, RF=3) │ │
│ └───────────────────────────────────────────────────┘ │
│ + Schema Registry (Avro/Protobuf) │
└──────────────────────────────────────────────────────────┘
│
▼ consumes and transforms
┌──────────────────────────────────────────────────────────┐
│ Flink SQL streaming jobs │
│ - chunking text │
│ - calls to the embedding model (UDF) │
│ - enrichment with metadata │
│ - sink to a curated topic: documents.embedded │
└──────────────────────────────────────────────────────────┘
│
┌───────────┼────────────────────┐
▼ ▼ ▼
[Vector store] [Tableflow] [Iceberg/Delta]
Milvus/Qdrant auto-convert for analytics
/pgvector/ topics →
Weaviate tables
│
▼ queried at runtime
┌──────────────────────────────────────────────────────────┐
│ LLM serving (vLLM / SGLang) + Retriever │
│ - receives the agent's query │
│ - searches top-K in the vector store │
│ - builds prompt + context │
│ - generates an answer with citations │
└──────────────────────────────────────────────────────────┘

The five layers you can see (source, ingestion (CDC), transport (Kafka), processing (Flink), storage (vectors + tables)) are the ones that structure any serious RAG on a datalake in 2026. Let us go through each one.

Layer 1 — Sources: your OLTP as the starting point

The typical source is an OLTP database (Postgres, MySQL, SQL Server). It is where the live state of the business lives. The technique for extracting changes in real time is Change Data Capture (CDC): reading the database’s transaction log (PostgreSQL WAL, MySQL binlog) and turning each commit into a Kafka event.

The OSS standard is Debezium. It supports Postgres, MySQL, SQL Server, MongoDB, Oracle, Cassandra and others. The typical deployment is a Kafka Connect cluster with Debezium connectors.

An example Debezium configuration for PostgreSQL:

{
 "name": "postgres-orders-connector",
 "config": {
 "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
 "tasks.max": "1",
 "database.hostname": "postgres.prod.internal",
 "database.port": "5432",
 "database.user": "debezium",
 "database.password": "${secret:postgres-creds}",
 "database.dbname": "ecommerce",
 "database.server.name": "ecommerce-prod",
 "table.include.list": "public.orders,public.users,public.products",
 "publication.autocreate.mode": "filtered",
 "slot.name": "debezium_slot",
 "plugin.name": "pgoutput",
 "topic.prefix": "ecommerce",
 "key.converter": "io.confluent.connect.avro.AvroConverter",
 "value.converter": "io.confluent.connect.avro.AvroConverter",
 "key.converter.schema.registry.url": "http://schema-registry:8081",
 "value.converter.schema.registry.url": "http://schema-registry:8081"
 }
}

For each commit in the database, this produces an Avro event on the corresponding topic (ecommerce.public.orders, ecommerce.public.users, and so on) carrying the change: type (INSERT/UPDATE/DELETE), values before and after, commit timestamp, position in the WAL.

A simpler alternative for 2026: RisingWave can read the Postgres WAL directly, with no Debezium or Kafka Connect in between. When the case is CDC only, with no other sources, it is operationally simpler. For architectures with multiple sources (CDC + APIs + scrapers + logs), Debezium remains the standard piece.

Layer 2 — Kafka as transport and persistence

The Kafka cluster is where all events land. Key operational decisions:

Topics: raw vs curated

The convention that settled in 2026:

*.raw: the raw event exactly as it arrived. Untransformed CDC, unparsed application logs.
*.cleaned: after dedup, schema validation, type normalisation.
*.enriched: after adding metadata (geolocation, cross identifiers, and so on).
*.embedded: the event with its vector embedding already computed.

Multi-stage topics allow debugging per layer and partial reprocessing: if you change the embedding model, discarding *.embedded and rebuilding from *.enriched costs hours; rebuilding from *.raw costs days.

Schema Registry

Without a schema registry, topics break silently when someone changes the schema at the source. Confluent Schema Registry or the OSS Apicurio are the dominant options.

Common formats:

Avro: versioned schema, strict evolution rules. The historical default.
Protobuf: compatible with gRPC stacks, good performance.
JSON Schema: textual, debuggable by eye, less efficient.

For RAG on Kafka we recommend Avro by default. Schema evolution matters because source tables change over time, and an unversioned schema breaks downstream consumers.

Partitions, replication and retention

Operational decisions for RAG topics:

Partitions: typically 3-12. More partitions means more parallelism in the Flink consumer, but more overhead. The rule of thumb: partitions = expected peak events/s ÷ 1000.
Replication factor: 3 as a minimum in production. Replication protects against broker failure; with RAG, the cost of losing a topic can be weeks of re-embedding.
Retention: for topics that feed RAG, use long retention or compaction by key. If document doc-42 changes 100 times, compaction keeps only the last state per key, leaving a smaller, rebuildable log. For data that is never updated (historical logs), retention by time (90 days, 1 year).

Cross-cluster replication

For multi-region or multi-cloud deployments, MirrorMaker 2 or Cluster Linking (Confluent) replicate topics between Kafka clusters. RAG can then query the local cluster without crossing regions.

Layer 3 — Flink as the streaming processor

Apache Flink is the dominant stream processing piece in 2026. Apache 2.0, a mature distribution, a broad ecosystem. The main alternative is Kafka Streams (simpler, Java-only); RisingWave is the emerging option for pure SQL cases.

What Flink adds to Kafka:

Stateful streaming: time-based aggregations, joins between streams, sessions.
Exactly-once semantics: with checkpoint coordination.
Watermarks: correct handling of out-of-order events.
UDFs in Python/Java: including calls to LLM models.

Flink SQL: the most operational piece

Flink SQL is the most usable part of Flink for data engineers who are not streaming experts. Here is a realistic RAG pipeline example:

-- 1. Define the source: Kafka topic with CDC events from documents
CREATE TABLE documents_raw (
 doc_id STRING,
 title STRING,
 body STRING,
 category STRING,
 updated_at TIMESTAMP_LTZ(3),
 PRIMARY KEY (doc_id) NOT ENFORCED
) WITH (
 'connector' = 'upsert-kafka',
 'topic' = 'ecommerce.public.documents',
 'properties.bootstrap.servers' = 'kafka:9092',
 'key.format' = 'avro-confluent',
 'value.format' = 'avro-confluent',
 'value.fields-include' = 'EXCEPT_KEY'
);

-- 2. Define the sink: vector store via an intermediate Kafka topic
CREATE TABLE documents_embedded (
 doc_id STRING,
 chunk_id INT,
 title STRING,
 chunk_text STRING,
 category STRING,
 embedding ARRAY<FLOAT>,
 embedded_at TIMESTAMP_LTZ(3),
 PRIMARY KEY (doc_id, chunk_id) NOT ENFORCED
) WITH (
 'connector' = 'upsert-kafka',
 'topic' = 'rag.documents.embedded',
 'properties.bootstrap.servers' = 'kafka:9092',
 'key.format' = 'json',
 'value.format' = 'json'
);

-- 3. UDF for chunking (defined in Python or Java)
-- CREATE TEMPORARY FUNCTION chunk_text AS 'com.example.ChunkingUDF';

-- 4. Pipeline: chunk, embed, write to the sink
INSERT INTO documents_embedded
SELECT
 doc_id,
 chunk_idx AS chunk_id,
 title,
 chunk AS chunk_text,
 category,
 OPENAI_EMBEDDING(chunk,
 'text-embedding-3-small') AS embedding,
 CURRENT_TIMESTAMP AS embedded_at
FROM documents_raw
CROSS JOIN UNNEST(chunk_text(body, 512, 64))
 WITH ORDINALITY AS t(chunk, chunk_idx);

What happens here, line by line:

The documents_raw table reads the CDC topic in upsert-kafka mode (each new event on the same key replaces the previous one). This correctly reflects the semantics of “this is the latest version of doc 42”.
The documents_embedded table will be the intermediate topic where Flink writes the embedded chunks.
The chunk_text UDF (defined in Python or Java) splits each doc into 512-token chunks with an overlap of 64.
The INSERT INTO query runs continuously: every new event in documents_raw is chunked, each chunk is embedded with OPENAI_EMBEDDING (a built-in Flink SQL function in Confluent Cloud 2026), and written to the embedded topic.

OPENAI_EMBEDDING can be swapped for a custom function that calls a self-hosted model (vLLM with an encoder), SentenceTransformers, or a managed service. The syntax is the same; you change the provider.

Watermarks and late events

For cases where an event can arrive late (for example, the Postgres WAL falls behind because of a network blip), Flink lets you define watermarks:

CREATE TABLE documents_raw (
 doc_id STRING,
 title STRING,
 body STRING,
 updated_at TIMESTAMP_LTZ(3),
 WATERMARK FOR updated_at AS updated_at - INTERVAL '5' MINUTE
) WITH (...)

This tells Flink “assume no event arrives more than 5 minutes late relative to the event timestamp”. For temporal joins and aggregations, Flink uses the watermark to decide when to “close” a window.

Layer 4 — Sinks to vector stores

The last step is indexing the embeddings in a vector store. Three patterns in 2026:

Pattern A — direct Kafka Connect sink

Each vector store has its official connector:

Milvus: official Zilliz sink connector. Supports named/unnamed dense/sparse vectors.
Qdrant: official sink connector. Supports dense, sparse, multi-vector.
pgvector: has no dedicated connector, but the JDBC Sink Connector is used with custom SQL.
Weaviate: community connector.
LanceDB: community connector.

An example Milvus sink configuration:

{
 "name": "milvus-rag-embeddings-sink",
 "config": {
 "connector.class": "com.milvus.io.kafka.MilvusSinkConnector",
 "tasks.max": "3",
 "topics": "rag.documents.embedded",
 "milvus.host": "milvus.prod.internal",
 "milvus.port": "19530",
 "milvus.collection.name": "documents",
 "milvus.collection.dim": "1536",
 "milvus.collection.partition": "default",
 "key.converter": "org.apache.kafka.connect.storage.StringConverter",
 "value.converter": "org.apache.kafka.connect.json.JsonConverter",
 "value.converter.schemas.enable": false
 }
}

Three parallel tasks (tasks.max: 3) consume the embedded topic and write to the Milvus collection. Latency from “event in Kafka” to “indexable vector in Milvus” is typically <5 seconds.

Pattern B — pgvector with a direct CDC pipe

For teams that already live in PostgreSQL, pgvector is the lowest-friction option. The pattern: the same source Postgres cluster holds a second DB for embeddings with the pgvector extension enabled; the Flink pipeline writes to it directly over JDBC.

-- On the Postgres cluster with pgvector enabled
CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE document_embeddings (
 doc_id TEXT,
 chunk_id INT,
 chunk_text TEXT,
 category TEXT,
 embedding vector(1536),
 embedded_at TIMESTAMP,
 PRIMARY KEY (doc_id, chunk_id)
);

CREATE INDEX ON document_embeddings
 USING hnsw (embedding vector_cosine_ops)
 WITH (m = 16, ef_construction = 64);

Advantages: the same DBA operates everything, transactionality across tables, trivial joins with relational metadata. Limitation: beyond 10M vectors, pgvector performance starts to give way compared with dedicated systems.

Pattern C — Confluent Tableflow → Iceberg + Flink SQL vector search

This is the 2026 development that changes the mechanics. Confluent Tableflow materialises Kafka topics automatically as Apache Iceberg or Delta Lake tables. Features:

No ETL pipeline: you do not write Flink/Spark jobs to move Kafka into a table. Tableflow does it.
Automatic schema evolution: changes to the topic schema are reflected in the table.
Unified catalogue: the table appears in Glue, Unity Catalog, Snowflake, Databricks. Any analytical engine queries it without copying data.
Native CDC: handles inserts, updates and deletes correctly.
30-50 % lower TCO according to the figures Confluent publishes against traditional pipelines.

And since 2026, Tableflow + Flink SQL offer native vector search integrated with Cosmos DB and Amazon S3 Vectors. The RAG query can be made directly in Flink SQL:

SELECT doc_id, chunk_text, category
FROM documents_embedded
WHERE VECTOR_SEARCH(embedding,
 OPENAI_EMBEDDING('user query', 'text-embedding-3-small'),
 top_k => 10) > 0.7
ORDER BY VECTOR_SEARCH_SCORE DESC;

This unifies layers that used to be separate (vector store + analytics). For many cases, it removes the need to maintain a dedicated vector store.

Confluent’s official MCP server

One piece added in 2026 deserves a mention: Confluent has published an official MCP server that exposes Kafka, Flink and Tableflow as tools accessible to AI agents over MCP. Any MCP client (Claude Desktop, Cursor, your own agents) can:

List topics, read recent messages, publish to topics.
Run Flink SQL queries in natural language (“give me the orders from the last 24 hours worth more than 1000 €”).
Query Tableflow Iceberg tables.
Manage Kafka Connect connectors.

This closes the circle: as well as reading data from the datalake through RAG (with vector search), your AI agent can write data to the log (through MCP) and trigger transformations (through Flink SQL in natural language). It is the deepest point of fusion between LLM ops and data ops this year.

A connection with the earlier series: this MCP server emits traces with the OpenTelemetry GenAI MCP semantic conventions we covered in the MCP observability post. The spans show up in Langfuse, Phoenix or your OTel backend with the right cardinality. Zero instrumentation code.

Vector stores: a 2026 comparison

The five dominant options:

Vector store	Licence	Operation	Where it fits
pgvector	Postgres ext, OSS	Your DBA	<10M vectors, Postgres-heavy team
Qdrant	Apache 2.0	Self-host or managed	Mid-scale, performance focus
Milvus	Apache 2.0	Self-host or Zilliz Cloud	Large-scale, scalability focus
Weaviate	BSD-3	Self-host or managed	Native hybrid search, semantically rich
LanceDB	Apache 2.0	Embedded or serverless	Small-medium, simplicity

The choice depends on:

Scale: pgvector falls short beyond 10M vectors. Milvus and Qdrant scale to billions.
Hybrid search: Weaviate brings lexical + vector natively. Others support it but less integrated.
Operation: pgvector if you already run Postgres. Qdrant if you want simplicity. Milvus if you need maximum scale.
Cloud managed: Zilliz Cloud for Milvus, Qdrant Cloud for Qdrant, Pinecone if you want pure SaaS (with no OSS behind it).

Freshness vs accuracy: the operational trade-off

A critical decision any RAG system on Kafka must answer: when is a new document considered “live” in the index?

Three options:

Synchronous streaming: the event reaches Kafka, Flink embeds it, the sink writes it to the vector store, and only then is it considered live. Typical latency: 1-5 seconds. The best freshness. But if the embedding model fails or the vector store is slow, events pile up in the topic.

Asynchronous streaming with a baseline: the event is considered live immediately; a background process embeds it when it can. In the meantime, queries asking for that document do not find it. Typical latency: 5-60 seconds. Acceptable for most applications.

Micro-batch: processing happens in mini-batches every 1-5 minutes. Less efficient than continuous streaming but more stable under variable load. Latency: 1-5 minutes.

The decision depends on the product’s SLA. For customer support chatbots, 5-60 seconds is acceptable. For systems that react to critical events (financial prices, alarms), synchronous streaming is necessary.

Schema evolution and re-embedding

When the embedding model changes (you move from text-embedding-3-small to text-embedding-3-large, or from OpenAI to Cohere), the vectors already in the index are incompatible: different dimensions, different semantic spaces. The distance between an old vector and a new one means nothing.

The standard pattern for handling this: dual-index during the migration.

T0: the active index is V1 (embedding model A).
T1: a parallel pipeline starts writing to a V2 index (embedding model B), consuming the topic from offset 0 (reprocessing the whole log).
T2: V2 has caught up with the present.
T3: you switch the retriever over to V2.
T4: a week later, you discard V1.

The Kafka log makes this pattern feasible because it is immutable and reproducible. Without the log, this pattern becomes a weeks-long data migration project.

Operational pitfalls

Topics without adequate retention

Configuring topics with 7-day retention on the grounds that “I already have the vector store” means losing the ability to rebuild if the vector store fails. Long retention (90+ days) or compaction by key for topics that feed RAG.

Heavy CDC during peak load

Debezium reading the WAL at peak hours can hit the source database’s performance. A dedicated read replica for Debezium, not the production primary. Or use logical replication restricted to just the tables you need.

Embedding cost run-away

OPENAI_EMBEDDING on every event of a topic with millions of messages a day comes to thousands of USD per month. Strategies: filter before embedding (only embed what adds value); deduplicate by content hash; use self-hosted open-source embedding models (BGE, E5, GTE) when the cloud cost is prohibitive.

Slow re-embedding due to limited throughput

Recomputing 10M embeddings with the OpenAI API at 3000 req/min takes 55 hours. If you wait for an incident to re-embed, that is two days without service. Embedding throughput is an explicit capacity planning item; reserve capacity or keep an offline job ready to start.

Downstream schema breaks

A change to the raw topic’s schema breaks Flink jobs downstream. Schema Registry with BACKWARD compatibility made mandatory; never ALLOW_ALL. And test schema evolution in CI.

A vector store with no backup

Your vector store holds 50M vectors. It is the only copy (the topics expired). A failure wipes it. Vector stores must be backed up like any primary persistence layer. For Milvus/Qdrant: periodic snapshots. For pgvector: plain pg_dump.

Multi-region without cross-cluster replication

Your RAG serves users in the US and the EU. The vector store is in US-east. Latency from the EU = 100 ms+ per query. MirrorMaker or Cluster Linking to replicate topics and vector stores in both regions.

What we have not covered

Hybrid search in production: combining BM25/lexical + vector + reranker. A topic for its own post.
Multimodal RAG: indexing images, audio and video alongside text. Multimodal embeddings (CLIP, Imagebind), a specific architecture.
GraphRAG: using structured knowledge (knowledge graphs) as well as vector retrieval. Microsoft GraphRAG, LlamaIndex KnowledgeGraphQueryEngine.
RAG with multi-tenant ACLs: filtering by permissions at runtime. The pattern uses metadata in the vector store + server-side filters.
Query rewriting with an LLM: using a first LLM to expand the query before retrieval (HyDE, multi-query, step-back prompting).

References

Kafka and stream processing:

Apache Kafka y Debezium.
Confluent Schema Registry y Apicurio Registry.
Apache Flink y Flink SQL docs.
RisingWave — alternativa SQL streaming con embedding built-in.

Vector store connectors:

Tableflow and 2026 architecture:

Streaming RAG:

Cross-references:

Previous post: MLOps specific to LLMs in 2026: the landscape.
Data series: PostgreSQL + Qdrant at the ingestion stage, RAG corpus curation: the active librarian, Reranker and hybrid retrieval — the four pieces that close the Data block: streaming, ingest, curation and retrieval.
Post-tracing series: Evals, Guardrails, MCP observability, eBPF + drift.
eBPF series: eBPF from zero to Cilium, Tetragon, Hubble, AgentSight.

MLOps specific to LLMs in 2026: the landscape of three modalities, six stages and ten tools that make them work

Thu, 21 May 2026 05:30:00 +0200

TL;DR

This is the blog’s fourth series and it is called MLOps specific to LLMs. It takes the operational craft of traditional MLOps (reproducible pipelines, model registries, dataset versioning, eval gates, controlled deployments) and redraws it for a world where the model is probabilistic, the outputs are subjective, the dependencies include external vendors that update weights without warning, and the “application” is not a model but an orchestration of models, embeddings, retrievers, guardrails and routers. Gartner predicts that more than 50% of enterprise GenAI deployments will fail before the end of 2026, and the main cause is not the model: it is that deterministic software assumptions were applied to probabilistic systems. This post opens the series with the frame: the seven structural differences between LLMOps and classic MLOps; the six-stage pipeline (data → tune → eval → deploy → observe → retrain); the three modalities of preparing a model (continuous fine-tuning, RAG over datalakes, agent training) with their decision matrix, where 60% of 2025-2026 deployments use hybrid because each modality solves a different problem: “fine-tune for behaviour, RAG for volatile knowledge”; and the 2026 tool landscape, which already forms reasonably stable layers: MLflow 3.10 (March 2026) as a GenAI-aware registry, W&B Weave and ZenML for tracing and pipelines, Kubeflow + KServe vLLM 0.8.1+ for serving, BentoML for flexibility, DVC + lakeFS (joined since November 2025) for data, Langfuse for prompts and observability. The three following posts will go down into the detail of the most critical pieces.

This is the opening of series 4: MLOps for LLMs. It continues the tradition of the previous series: LLM inference (the first), eBPF (the second) and post-tracing (the third). Here we enter the discipline that ties all the pieces together: how an LLM system is operated in production over months, not just deployed once.

The analogy: the SRE’s craft redrawn

Anyone who has worked for years as an SRE or a platform engineer recognises the classic pillars: reproducibility (same code + same data + same config = same result), observability (what happens can be measured), safe rollback (if something goes wrong, I go back within minutes), gradual rollout (the new thing goes to 1% before it goes to 100%). These pillars are not negotiable. The question is whether they hold up when the central component is an LLM.

The answer is: same pillars, radically different mechanics. Reproducibility: versioning code and data is no longer enough; you have to version prompts, retrieval configurations, snapshots of the vendor’s model (which change without warning). Observability: error and latency metrics are no longer enough; you have to measure subjective quality via LLM-as-judge and embedding drift. Rollback: dropping the binary’s version is no longer enough; you have to keep the old model cached because loading a new one takes minutes. Gradual rollout: a % of traffic is no longer enough; you have to decide what % of what kind of queries by segment.

It is the same craft, practised with partly new tools and reflexes. MLOps specific to LLMs, or “LLMOps”, as the field has christened itself, is the discipline that codifies those reflexes.

The seven structural differences between LLMOps and traditional MLOps

Before going down to the pipeline, let us fix the differences that make this territory new rather than a mere continuation. Each one has concrete practical consequences.

1. Non-deterministic outputs

Traditional MLOps: the model receives structured input and returns a bounded, reproducible prediction. Same input → same output. Unit tests work.

LLMOps: same input → different output every time (through sampling, through temperature, through the order of tools invoked, through the retrieval context that changed). The idea of a “unit test” breaks.

Operational consequence: tests over properties (was the tone maintained?, does it mention the source?, does it respect the JSON schema?), not over equality. Statistical evals over a distribution, not over samples.

2. Behaviour metrics, not statistical accuracy

Traditional MLOps: F1, accuracy, AUC, RMSE. Metrics with a clear number.

LLMOps: subjective rubric scores (G-Eval, faithfulness, helpfulness, toxicity), judge LLMs, human feedback. The “number” depends on who judges.

Operational consequence: platforms have to treat evals as versioned artifacts, not just “model v3 scored 0.87”, but “model v3 evaluated with the judge claude-3-5-sonnet-20251022 over the dataset gold-rag-v7 with the prompt judge-v2 scored 0.87”. Versioning the judge is as important as versioning the model being evaluated.

3. The model is an external dependency, not an internal asset

Traditional MLOps: you train the model, it lives in your registry, it does not change until you change it.

LLMOps: the base model belongs to Anthropic, OpenAI, Google, Meta. They change it on you without warning. The claude-3-5-sonnet version that answered well yesterday answers somewhat differently today.

Operational consequence: drift detection becomes far more critical (previous post). Pinning to specific snapshots (claude-3-5-sonnet-20251022) when the vendor allows it. For high-commitment apps, self-hosting the base model to guarantee reproducibility.

4. The application is an orchestration, not a model

Traditional MLOps: an app calls a model and consumes its output.

LLMOps 2026: an app connects foundation model + LoRA adapters + retrievers + vector stores + guardrails + routers + tool servers (MCP) + evaluators, all composing the final behaviour. Any component can degrade the result.

Operational consequence: cross-component debugging requires distributed tracing with OTel (covered in previous posts). The registry does not only store “the model” but the composition: which prompt version + which adapter + which vector store + which retriever config.

5. Cost per inference, not per training

Traditional MLOps: the high cost is training; serving is cheap. You optimise training.

LLMOps: the high cost is serving (every token costs, every call to the vendor is paid for, the GPUs doing the serving are switched on 24/7). You optimise inference.

Operational consequence: cost accountability per tenant, per agent, per tool. Metrics such as gen_ai.usage.input_tokens aggregated at customer and product level. Model decisions based on cost per query, not only on quality.

6. GPU-heavy infra with specific primitives

Traditional MLOps: CPU + some GPU for training. Standard Kubernetes.

LLMOps: Hopper/Blackwell SXM GPUs, NVLink/NVSwitch, tensor parallel, paged attention, KV cache. Infra that only fits into Kubernetes with primitives such as LeaderWorkerSet, GPU Operator, KEDA with LLM metrics (covered in vLLM on Kubernetes).

Operational consequence: the orchestration stack includes specialised operators (OME, vLLM Production Stack, NVIDIA Dynamo, llm-d, see LLM K8s Operators) that traditional MLOps does not contemplate.

7. RLHF and human feedback as a first-class citizen

Traditional MLOps: human feedback means labelling data before training.

LLMOps: human feedback lives inside the model in production, whether through RLHF of the foundation models (Anthropic, OpenAI), through RLAIF, through DPO, or through explicit user feedback that is fed back into fine-tuning.

Operational consequence: bidirectional production → training pipelines. Datasets grow with real incidents. Model decisions are taken with continuous feedback, not in a training project every N months.

Why Gartner predicts 50%+ failures

Gartner published that more than 50% of enterprise GenAI deployments will fail before 2026. The causes are not technical ones about the model but about the system:

Hallucinated outputs from poor grounding: badly designed RAG, poor retrieval, insufficient context.
Data architectures that are not ready: companies have data in silos, with no stable schemas, with no controlled freshness. Connecting an LLM to this data without a serious pipeline produces erratic answers.
Lack of structured workflows for prompt-driven systems: teams that treat prompts as code in hardcoded strings, with no versioning, no tests, no gates.

The conclusion the field draws: LLMOps is not optional. Companies that deploy GenAI without operational discipline fall into one of the three failure modes. Those that apply it (MLflow/W&B for tracking, DVC/lakeFS for data, Langfuse for prompts and evals, KServe or vLLM Production Stack for serving, drift detection in production) are the ones that keep the system working six months after the first release.

The six-stage LLMOps pipeline

On to the pipeline. The six stages that any serious LLM system goes through, in order:

[1. Data] → [2. Tune] → [3. Eval] → [4. Deploy] → [5. Observe] → [6. Retrain]
│
└─→ back to 1

Each stage is an operational domain of its own, with its own tools and traps:

Stage 1 — Data

What happens: ingestion, cleaning, curation, versioning, indexing of the corpus. It is where real projects suffer most, because companies have data in heterogeneous silos.

Typical sub-tasks: extraction from source (CDC over Kafka, batch from data lakes, scraping), cleaning (PII removal, dedup, format), curation (labelling for fine-tuning, golden datasets for eval), versioning (DVC + lakeFS), indexing (embeddings + vector store for RAG).

Traps: schema drift at the source, undetected PII, poor dedup that injects redundancy into training, a vector store that is not updated.

Stage 2 — Tune

What happens: preparing the model for your use case. Three modalities (we go deeper into them shortly): fine-tuning, RAG, agent training.

Typical sub-tasks: base model selection, adapter preparation (LoRA, QLoRA), training loop with continuous eval, hyperparameter sweep (Optuna, W&B Sweeps), saving the checkpoint.

Traps: catastrophic forgetting if the fine-tuning is very aggressive, overfitting to the golden dataset, no independent validation set.

Stage 3 — Eval

What happens: validating that the model + adapters + RAG configuration is acceptable before promoting. Covered in Evals.

Typical sub-tasks: running an eval framework (DeepEval, Promptfoo, Ragas) against the golden dataset, judge LLM evaluations, human review over a sample, gates with thresholds.

Traps: a golden dataset that ages, an uncalibrated judge, evals that pass in CI but fail in production because of distribution shift.

Stage 4 — Deploy

What happens: going from “the model evaluated well” to “the model serves real traffic”. Covered in LLM K8s Operators.

Typical sub-tasks: serving with vLLM/SGLang/TRT-LLM, runtime configuration, gradual rollout (canary, shadow, blue-green), routing between models (LiteLLM, OpenRouter, LangChain routers).

Traps: a naive rolling update that cuts sessions, autoscaling by CPU% that does not respond to LLM metrics (covered), a new model that performs worse in production than in eval.

Stage 5 — Observe

What happens: seeing what is happening in real time. Covered in the whole post-tracing series and the eBPF series.

Typical sub-tasks: tracing (Langfuse, LangSmith, Phoenix, OpenLLMetry), metrics (TTFT, TPOT, queue depth, cost per query), active guardrails (NeMo, Llama Guard), drift detection (Evidently, NannyML, WhyLabs).

Traps: cardinality explosion in metrics, batch evals with no tail-sampling over real traces, drift that is ignored until an incident makes it real.

Stage 6 — Retrain

What happens: closing the loop. Production feedback (incidents, worst-scoring cases, detected drift) generates new data to go back to stage 1.

Typical sub-tasks: extraction of problematic logs, human labelling of the sample, incorporation into the golden dataset, re-fine-tuning if applicable, decision on a new release.

Traps: an “open” loop where production never informs the dataset, human feedback that gets lost, lack of a clear retrain cadence.

The three modalities of “preparing the model”

Stage 2 (Tune) is where there is most confusion. In 2026 three modalities coexist, each solving a different problem:

Fine-tuning

What it does: modifying the model’s weights (or those of a LoRA/QLoRA adapter on top) so that it learns specific behaviour patterns: tone, output structure, idiomatic decisions of the domain.

When: when your main failure is behavioural inconsistency between calls. The model sometimes answers formally, sometimes not; sometimes it structures the JSON, sometimes not; sometimes it follows the company’s conventions, sometimes it invents them. Fine-tuning stabilises it.

When NOT: when what you need is up-to-date knowledge. Fine-tuning fixes knowledge in frozen weights; the day after the fine-tuning, the model knows nothing new.

RAG (Retrieval-Augmented Generation)

What it does: leaving the model intact and, on each call, retrieving fresh context from a knowledge base (vector store + lexical search, typically) and passing it to the model so that it answers on that basis.

When: when the knowledge you need is dynamic or very large. Documentation that changes, a product catalogue that is updated, an internal knowledge base that grows.

When NOT: when the problem is behavioural (RAG does not teach the model how to behave, it only gives it information). Or when retrieval is so noisy that the context arriving is worse than nothing.

Agent training

What it does: going beyond conventional fine-tuning with Reinforcement Learning techniques. RFT (OpenAI’s Reinforcement Fine-Tuning), classic RLHF, RLAIF (with AI feedback), DPO (Direct Preference Optimization) over datasets of (good, bad) pairs.

When: when the model needs to learn complex multistep trajectories: when to choose each tool, how to decompose a task, when to ask the user for confirmation. It is what is turning Claude, Gemini and GPT into agents capable of long tasks.

When NOT: when your case is simple chat or RAG. It is overkill, expensive and complicated for problems that the previous modalities solve.

Decision matrix

Observed problem	Modality
Inconsistent answers in tone/structure	Fine-tuning
The model makes things up (hallucinates)	RAG
Out-of-date knowledge (>1 year)	RAG
The model chooses tools badly	Agent training (RLAIF/RFT)
Mixed behaviour + knowledge	Hybrid (fine-tune + RAG)
Multi-step trajectory fails	Agent training
A specific language/regional style	Fine-tuning

The 2026 verdict: hybrid is the default

Multiple reports agree that in 2025-2026, around 60% of production projects use hybrid: fine-tuning for behaviour + RAG for knowledge. The key insight:

Fine-tune for behaviour (brand voice, decision protocol, output structure); use RAG for volatile knowledge you need the model to cite. Do not force a single tool to do both jobs.

A practical observation: the biggest quality improvements of 2025-2026 come from better reranking in RAG (cross-encoders), not from better embeddings. Rerankers add 15-35% of quality with little complexity.

On cost: combined fine-tuning + RAG tends to be 30-50% cheaper than pure RAG with frontier models at high volume, because the fine-tuned model can be smaller and cheaper while keeping equivalent quality.

The 2026 tool landscape

On to the concrete pieces, grouped by function. The field has matured enough for each piece to have 2-3 reasonable options and a couple of leaders.

Experiment tracking and model registry

MLflow remains the de facto standard, now with LLM-specific traction. MLflow 3 was published in June 2025; version 3.10.1 (March 2026) added:

A GenAI Overview dashboard with ready-made metrics for LLM apps.
Multi-workspace support for large teams.
Cost tracking in traces (gen_ai.usage.* aggregated per experiment).
MemAlign: a new specific eval algorithm.
Integrated native OpenTelemetry tracing.
First-class support for LangChain, LlamaIndex, AutoGen as frameworks.

MLflow treats prompts and agents as first-class citizens alongside classic models. It is the biggest change relative to MLflow 2.x.

Weights & Biases (W&B) with its LLM-specific product Weave offers tracing + eval + debug with a very polished UI. More commercial, less self-host friendly, but excellent UX.

ZenML is the piece that most cleanly integrates “classic MLOps + emerging LLMOps” into a single framework. Its automatic artifact versioning captures prompt templates, retrieval chunks and agent conversation histories with no extra work. Open-source. The most complete unification option that exists.

Dataset versioning

DVC remains the OSS standard. It extends Git to large files and pipelines. Important news from November 2025: lakeFS acquired DVC, consolidating the two OSS data versioning projects under one organisation. The combined roadmap is oriented towards LLM training and RAG datalakes specifically.

Typical pattern: Git for code + DVC for data/models + MLflow or W&B for experiment tracking + registry. Few teams use just one; the combination is what covers the cycle. Detailed in the dedicated post on data versioning for LLMOps — the four artifacts to version separately, schema contracts, end-to-end lineage, and a golden eval set with a strict holdout.

Prompt versioning and observability

Covered in depth in the AgentSight post where we go deeper into Langfuse as the OSS reference. Summary here:

Langfuse: MIT, self-host, prompt management with v1/v2/v3 versioning + labels + cache + linkage with traces.
LangSmith: if your stack is LangChain.
Arize Phoenix: ELv2, OTel-native.

Pipeline orchestration

For the LLMOps pipeline steps, the dominant options:

Kubeflow Pipelines: the K8s-native standard. KServe (the serving part of Kubeflow) has its vLLM runtime upgraded to v0.8.1+ with support for reasoning models, tool calling, embeddings, reranking, Llama 4 and Qwen 3.
ZenML: already mentioned; also a pipeline orchestrator.
Metaflow (Netflix-originated): Python-first pipelines, less LLM-specific but workable.
Argo Workflows: a pure K8s OSS alternative.
Flyte: Kubernetes-native, OSS.

Serving

Covered in depth in vLLM on Kubernetes and LLM K8s Operators. Summary:

vLLM Production Stack: a curated Helm chart.
KServe vLLM runtime: K8s-native, vLLM 0.8.1+ with full agentic support.
BentoML: flexible serving, popular in startups for its simplicity.
NVIDIA Dynamo: the successor to Triton.
llm-d: CNCF Sandbox.
OME: an LMSYS operator with native SGLang.

Evals and guardrails

Covered in Evals and Guardrails. Ultra-short summary:

Evals CI: DeepEval, Promptfoo, Ragas.
Evals platform: Langfuse, LangSmith, Phoenix, Braintrust.
Guardrails: NeMo Guardrails, Llama Guard 4, Llama Prompt Guard 2, LLM Guard, Lakera.

Drift detection and observability

Covered in the closing eBPF post. Summary:

Drift: Evidently AI, NannyML, WhyLabs.
Tracing: Langfuse, OpenLLMetry, Phoenix.
eBPF: AgentSight, Hubble, Tetragon, ProfInfer.

The typical 2026 stack table

Stage	Dominant piece	Alternatives
Data ingestion + versioning	DVC + lakeFS (unified Nov 2025)	Pachyderm, Quilt
Vector store / RAG index	Milvus, Qdrant, pgvector, Weaviate	LanceDB, Pinecone, Chroma
Experiment tracking	MLflow 3.10	W&B Weave, Neptune
Pipeline orchestration	Kubeflow + Argo	ZenML, Metaflow, Flyte
Model registry	MLflow registry	W&B Models, KServe ModelMesh
Prompt versioning	Langfuse	LangSmith, MLflow Prompts
Serving	vLLM Production Stack	KServe, BentoML, Dynamo, llm-d, OME
Evals CI	DeepEval, Ragas	Promptfoo, OpenAI Evals
Evals platform	Langfuse, Phoenix	LangSmith, Braintrust
Guardrails	NeMo + Llama Guard	LLM Guard, Lakera
Tracing	OpenLLMetry + Langfuse	Phoenix, LangSmith
Drift detection	Evidently AI	NannyML, WhyLabs
eBPF observability	AgentSight + Tetragon + Hubble	(new territory, few alternatives)

13 pieces. No org uses them all; any serious org uses at least six. This is the current LLMOps stack.

The operational reality: nobody uses a single tool

Multiple comparisons agree on one thing: the teams that win combine. Recurring patterns:

ZenML to orchestrate + MLflow for tracking + KServe for serving: the most popular OSS stack in companies coming from classic MLOps.
Kubeflow + W&B + BentoML: for teams with a research focus.
Langfuse + DeepEval + Phoenix + LiteLLM: for pure-LLM teams with no classic MLOps background.
MLflow + DVC + Argo + KServe: an idiomatic cloud-native stack without additional LLM-specifics (with its limitations).

The choice depends on the team’s background, on the licensing model they can afford, on the level of self-hosting they need, and on which frictions blocked them most in previous projects. There is no “one right answer”; there is a stable meta-pattern of layers worth covering.

Common operational traps

Treating the prompt as text in code

Hardcoding prompts in strings in the repo. Changing them requires a PR + redeploy. Result: teams that do not iterate on prompts because every change costs hours of pipeline. Solution: externalised prompt management (Langfuse, MLflow Prompts) with versioning, labels, hot-reload.

Skipping dataset versioning

“DVC is complicated, we will add it later”. Result: two months later, nobody knows which dataset trained which model. Impossible to reproduce incidents. Solution: DVC + lakeFS from day 1, even if only with a small subset.

Mixing layers in the same pipeline

Teams that put ingestion, fine-tuning, eval and deploy into a single giant pipeline. When something fails, the whole pipeline fails. Solution: independent pipelines per stage, with versioned artifacts as the interfaces between them.

Tracking with no structure

Logging everything to stdout and “we will look at it in CloudWatch later”. Result: impossible to correlate, compare, debug. Solution: OTel from day 1 with gen_ai.* semantic conventions.

Evals that block nothing

You have evals, you run them, you look at them, but they do not prevent the deploy if they drop. Eventually it drops gradually and nobody notices. Solution: eval gates in CI/CD that block the merge if critical metrics regress by more than X%.

No retrain cadence

You launch v1 and never go back to the model. Six months later, drift has degraded it but the team is on other projects. Solution: a formal retrain cadence (monthly, quarterly) tied to the production incident queue.

Invisible vendor lock-in

You start with the OpenAI API + LangSmith + Pinecone. When you want to self-host, you discover that migrating is a 3-month project. Solution: abstraction layers (LiteLLM, OpenLLMetry) and vendor-neutrality from the start.

What is coming in the following posts of the series

Post 2 — RAG over datalakes with Kafka: end-to-end technical architecture — the most hands-on one. Kafka as source-of-truth, Flink CDC, embedding pipelines, continuous indexing in Milvus/Qdrant, a complete example with real numbers and manifests.
Post 3 — The six-stage LLMOps pipeline: global architecture — the master map of the complete system with a reusable “you are here” SVG for the following posts. A deep dive into each of the six stages (Data, Tune, Eval, Deploy, Observe, Retrain).
Post 4 — PostgreSQL + Qdrant in the ingestion stage — synchronisation patterns (dual-write, outbox + CDC, event-driven), a complete microservices architecture, a Qdrant cluster manifest.
Upcoming posts — still to be decided: the cluster as a multi-tenant platform, Constitutional AI / alignment runtime, continuous fine-tuning in depth, edge LLMs.
Synthesis post — Anatomy of an LLM request in production, May 2026 — the complete journey of a real request through the six stages and the two cross-cutting components, with cross-links to each dedicated post. It works as a mental map of the blog and as an integrator’s guide.
Parallel OSS vs hyperscalers catalogue — The parallel catalogue: the six LLMOps stages in OSS and in AWS / GCP / Azure — for each stage of the pipeline, which tools the blog’s reference OSS stack uses and what the cloud equivalents are, with summary tables, identification of real gaps and the multi-tenant chatbot ported to an AWS stack component by component. A neutral editorial stance.
OSS catalogue entry by entry — The OSS catalogue for LLMOps in six stages: entry by entry — the zoom in on the open source side: ~150 words of description per core tool (vLLM, Langfuse, DVC, Qdrant, Airflow, NeMo Guardrails, Presidio…), licence and governance, a decision matrix per stage, a diagram of the connected OSS stack and a master table of licences and EE / SaaS offerings.

References

LLMOps vs MLOps:

Tools:

MLflow — registry + tracking + serving.
Weights & Biases Weave — LLM tracing.
ZenML — pipeline orchestration MLOps + LLMOps.
Kubeflow — K8s-native MLOps.
KServe — model serving K8s.
BentoML — flexible serving.
Metaflow — Netflix’s pipelines.
DVC — dataset versioning.
lakeFS — enterprise data versioning, acquired DVC in Nov 2025.

2026 comparisons:

RAG vs Fine-Tuning:

Cross-references (the three previous series):

LLM inference series: KV cache, vLLM on K8s, PagedAttention, LLM K8s Operators.
eBPF series: eBPF from zero to Cilium, Tetragon, Hubble, AgentSight.
Post-tracing series: Evals, Guardrails, MCP observability, eBPF + drift.

Mlops on lo0 — Tech Blog

Anatomy of an LLM request in production, May 2026: a tour of the six stages following a single request

TL;DR

You are here: every stage at once

The analogy: forensic analysis of a request

The scenario: a multi-tenant support chatbot for regulated customers

Step 0 — The frame: the answer the user sees

Rewinding backwards: what was already in place before the request

t = −90 days — The previous Retrain stage closes the earlier cycle

t = −60 days — The Data stage: the enriched dataset is versioned and enters circulation

t = −45 days — The Tune stage: the customer_support_v7 adapter is trained

t = −38 days — The Eval stage: the v7 adapter goes through eval gates

t = −31 days — The Deploy stage: the v7 adapter enters serving

Moving forward: the user’s request crosses the system

t = 0 ms — Entry through the gateway

t = 8 ms — Pulling the versioned prompt

t = 12 ms — RAG retrieval

t = 40 ms — Building the final payload

t = 45 ms — Prefill

t = 580 ms — Decode (streaming)

t = 4,200 ms — Complete answer, root span closed

In parallel: Observe is watching

The feedback: the loop closes

Insertion into feedback_signals

Triage by root cause

The next cycle picks it up

What goes in each trace: identity and traceability

Synthesis diagram: how the pieces fit

The on-premise stack applied

Ten bridges between stages where the system breaks

How to walk through the blog

What we have not covered (yet)

See also

References

Kubeflow in depth: which pieces are worth it in an on-premise LLM platform and which ones you already have

TL;DR

The analogy: the hypermarket and the specialist shop

What Kubeflow is in 2026

Component by component

Kubeflow Pipelines: the piece you may actually want

Kubeflow Trainer: it leans on your scheduler, it does not replace it

Katib: only half relevant

KServe: the piece that is no longer in Kubeflow

Model Registry, Notebooks and the rest

Multi-tenancy: powerful and tied to Istio

The cost of installing the whole thing

Kubeflow against the alternatives

The real fit with LLM workloads

Operational traps and honest scepticism

For an inference factory

See also

Sources

Reranker and hybrid retrieval: the committee that decides the 5 chunks the LLM will actually read

TL;DR

The analogy: the committee for a professorial chair

The problem: why dense on its own is not enough

Layer 1 — Hybrid retrieval: dense + sparse

Why BM25 is still alive in 2026

Dense bi-encoder: the second lane

The fusion: Reciprocal Rank Fusion (RRF)

Layer 2 — The reranker: bi-encoder vs cross-encoder

2026 OSS cross-encoder reranker stack

Late interaction: ColBERT-v2 as a compromise

Query rewriting and HyDE: correcting the query before retrieval

Minimum maths: latency budget and metrics

Latency budget of a canonical pipeline

Metrics: nDCG, MRR, Recall@k

Costs and throughput

The canonical pattern — which piece goes where

On-premise hardware: what fits in an RTX 4090 vs a generic 4×H100 SXM configuration

The seven traps that kill retrieval

What we have not covered (upcoming posts)

See also

References

RAG corpus curation: the active librarian who decides what goes in, what goes out and what gets signed

TL;DR

The analogy: the active librarian

The four data artefacts and where the RAG corpus fits

Layer 1 — Schema-validated ingest

Layer 2 — Three-level deduplication