Llmops on lo0 — Tech Blog

Anatomy of an LLM request in production, May 2026: a tour of the six stages following a single request

Fri, 22 May 2026 16:00:00 +0200

TL;DR

Across several series the blog has laid out the pieces that hold up an LLM system in production: the Data stage (dataset versioning, ingestion and vector stores, RAG over Kafka), the Tune stage (continuous fine-tuning), the Eval stage (evals as the layer after tracing, guardrails and safety), the Deploy stage (KV cache, PagedAttention, disaggregated serving, multi-tenant GPU cluster, vLLM on Kubernetes, LLM operators on K8s), the Observe stage (tracing with AgentSight, MCP observability, eBPF + drift), the Retrain stage (closing the feedback → dataset → adapter loop), and the cross-cutting components (prompt versioning and data versioning). What is missing is joining them up: seeing a single request cross every piece in order, in one coherent story. That is what this post does. We take a specific request from a multi-tenant support chatbot, rewind it back to the data that trained the adapter serving it today, follow it forward through serving, watch it reach the feedback store when the user marks a thumbs-down, and leave it as the seed of the next quarterly retrain cycle. The route works as a mental map and as an integrator’s guide: the system does not hold up if a single one of the seven pieces (six stages + two cross-cutting) is broken or absent. The practical lesson of the tour is not a new one. It is that everything is connected, that local measurements lie when taken in isolation, and that the real cost of not operating one stage well is paid by another stage further down the line.

You are here: every stage at once

Unlike previous posts, where the mini-map marked a single active box, this one covers the whole pipeline. It is the only post on the blog that activates the six stages and the two cross-cutting components simultaneously, because we follow a real request that crosses all of them.

The analogy: forensic analysis of a request

When an air accident happens, the forensic analysis does not limit itself to the last seconds of the flight. The investigation team rewinds to the maintenance of the previous six months, the manufacturer’s protocols, the pilot’s record, the weather briefing, the controller’s decisions, the incident history on the same model. The conclusion is rarely “the wing broke”; it is “the wing broke because an inspection protocol written a certain way did not detect microcracks that the 2014 calculation model did not consider critical and that were critical beyond a certain fatigue cycle”.

When an LLM request in production fails or succeeds, there is also a long causal chain behind it. The answer the user sees is the last frame; what determined it starts months earlier and branches across six operational stages. If you only look at the last frame, you attribute the result to the model. If you look at the whole chain, you see that the model is one of twelve factors and rarely the most important.

This post does that forensic analysis, but the other way round: instead of starting from a failure and rewinding, we start from a specific request that works and break down what had to happen for it to work, and what will happen to it afterwards. It is a guided tour, not an incident diagnosis. But the mental discipline is the same: no stage is autonomous, and understanding the system means understanding the bridges between stages, not just the boxes.

The scenario: a multi-tenant support chatbot for regulated customers

For the tour we use a concrete realistic scenario, representative enough that the observations carry over to most serious deployments in May 2026. It is an LLM customer support SaaS product, sold to several corporate customers (multi-tenant) in regulated sectors (banking, insurance, health). The product:

Accepts natural-language questions through a chat embedded in the customer’s website.
Retrieves relevant fragments from the customer’s internal knowledge base (product documents, contractual terms, FAQs).
Answers citing the fragment used, in the user’s language, with a tone adapted to the brand.
If the question requires an action (cancel, change plan, open a case), it invokes a tool from the catalogue exposed via MCP, with user confirmation.
It is contractually obliged not to invent customer data, to respect the tenant’s style guide, and to meet ENS / NIS2 compliance on data in the EU.

The infrastructure is on-premise over Kubernetes (RKE2 with the Cilium CNI, in the pattern covered in Cilium BGP and CNI networking). The GPU plane is a generic 4×H100 SXM cluster (80 GB each, NVLink) for serving and production training, plus an RTX 4090 (24 GB) for development, offline evals and ad-hoc re-embeddings.

The request we are going to follow:

User (end customer, Spanish, premium plan with an insurer): “How do I cancel my premium subscription? I have been paying for three months and I have not used half of the cover.”

At first sight, a simple question. Let us see what is behind it.

Step 0 — The frame: the answer the user sees

At 11:47:32 on a Tuesday, the user sends the message. 4.2 seconds later, this appears in the UI:

To cancel your premium subscription, follow these steps: (1) Sign in to the customer portal with your ID number and password. (2) Go to My policies → Premium Life 2024. (3) Press Request cancellation. (4) Select the reason (in your case, “not using the product”) and confirm. You will receive a confirmation email within 24-48 h.

Before you cancel, let me remind you that your plan includes a free annual medical check-up and 24/7 travel assistance, cover that is lost on cancellation. If you like, I can help you review your use of the cover over the past three months to see whether switching to a more tailored plan suits you better than cancelling.

Source: [Cancellation policy, section 4.2 of the 2024 General Terms]

After reading it, the user marks a thumbs-down and writes in the form: “the answer is correct but the tone is too salesy; I only wanted to know how to cancel”.

That sequence, question, answer, thumbs-down with structured feedback, is the last visible frame. Let us go backwards to understand what had to happen for the answer to come out that way.

Rewinding backwards: what was already in place before the request

Before the user typed, the system already had a model loaded in serving, an active prompt labelled production, an up-to-date vector index, a versioned dataset from the last fine-tuning, and a golden eval set that validated the promotion. Each of those artefacts got there through a process. We go through four jumps backwards.

t = −90 days — The previous Retrain stage closes the earlier cycle

Three months ago, during a quarterly Retrain cycle, two things happened. The first: the support team reviewed the feedback accumulated over the previous six months and saw a pattern, the model answering premium users in an excessively formal tone, with users reporting that it “feels robotic”. The second: a one-off incident (a customer cancelling over an answer perceived as curt) triggered an incident-driven mini-cycle.

The process, covered in detail in the Retrain post, followed five sub-processes:

Feedback capture — explicit thumbs-down plus implicit feedback (abandonments, retries) accumulated in a feedback_signals table in Postgres, all of them with a trace_id that allows rewinding to the exact context.
Triage by root cause — the “curt tone” incident cluster was categorised as a prompt issue (it was not the model answering badly, it was the system prompt asking for an overly formal register). One sub-cluster was a model issue (in some cases the model dug its heels in even with a warmer prompt).
Dataset enrichment — the team manually annotated 280 cases where the model was too curt, labelled with the reference answer (“how it should have answered”). Double annotation on the critical 20%; cases with a quality score below 4 were left out.
Cadence decision — the incident was treated as incident-driven; the rest of the quarterly Retrain followed the calendar.
Promotion — the new customer_support_v7 adapter went through eval gates against customer_support_v6, a 5% canary for a week, and was promoted when the golden set metrics showed a stable improvement in the “tone / clarity” segment with no regressions elsewhere.

The result: the adapter active in production when the user sent the Step 0 request is customer_support_v7, trained on the enriched dataset enriched_retrain_2026_q1 version 3, with double lineage back to the original incident.

t = −60 days — The Data stage: the enriched dataset is versioned and enters circulation

Immediately after Retrain, the Data stage of the LLMOps pipeline does its work. Three critical operations, covered in detail in the data versioning post:

Immutable versioning of the enriched dataset with DVC, sha256 hash propagated to the registry. The identifier (enriched_retrain_2026_q1, v3, sha256:9af...) becomes the luggage tag that will travel through the coming stages.
Schema contract validated by CI: every row meets the JSON Schema of the entry the trainer expects (example_id, input.user_query, input.retrieved_context, expected_output, rubric, segment, difficulty). A validation fails in CI if any row breaks the contract.
Holdout segregation check: the normalised sha256 hash of each input is compared against every hash in the active golden eval set (customer_support_golden_v12). Zero overlaps = the dataset does not contaminate the eval. Had there been a single one, CI would have blocked the merge.

In parallel, the RAG corpus (product manuals, FAQs, general terms of the insurer tenant) is kept alive. The ingestion pipeline keeps capturing changes from the customer’s CMS: a new section of the cancellation policy was modified in February and reindexed in Qdrant. As the post about RAG over Kafka explains, the corpus is not retrained with every change: only the delta is re-embedded, and lakeFS keeps a branch of the embeddings bucket with the new version. The branch is merged into main when recall@10 over a representative set of queries stays above the threshold (0.78 in this system).

t = −45 days — The Tune stage: the customer_support_v7 adapter is trained

Three weeks after closing the dataset, training of the new LoRA adapter starts. As the continuous fine-tuning post details, the production pattern in 2026 avoids retraining the base model, which is expensive, slow and irreversible, and favours a LoRA adapter over a stable base model (in this system, Llama 3 70B-instruct quantised to INT8 for serving). The training:

Runs over 4 of the H100s (NVLink, tensor parallel) for ~6 hours.
Uses transformers + PEFT + bitsandbytes, monitored by MLflow.
Every step records the dataset_id, dataset_version and dataset_hash as an input artifact in MLflow.
The output, a customer_support_v7.safetensors file of ~280 MB with the LoRA weights, is uploaded to MinIO with its own hash, and MLflow records model_id, model_version, parent_dataset.

At this point the lineage chain is closed for this leg:

enriched_retrain_2026_q1, v3, sha256:9af...
│
▼
mlflow run train, run_id: 0xa721...
│
▼
customer_support_v7, sha256:5c1...

t = −38 days — The Eval stage: the v7 adapter goes through eval gates

The freshly trained adapter is not promoted. It goes through an eval suite covered in detail in the post about evals. The golden eval set, customer_support_golden_v12, 850 human-curated examples with an inter-annotator kappa of 0.81, is run against two models: the candidate v7 adapter and the v6 currently in production. The metrics:

Metric	v6 (prod)	v7 (cand.)	Threshold
Faithfulness to the RAG fragment	0.87	0.89	≥ 0.82
Toxicity (low is good)	0.012	0.011	≤ 0.02
“Warm but professional” tone (judge LLM)	0.71	0.84	≥ 0.78
Format compliance (structured markdown)	0.94	0.93	≥ 0.90
Helpful-but-not-pushy (judge LLM)	0.66	0.79	≥ 0.75
Latency p95 (ms)	2,840	2,910	≤ 3,500

To this is added the guardrails and safety suite covered in the guardrails post: jailbreak resistance, PII leakage detection, prompt injection over MCP tools. The v7 improves on safety in two metrics and ties on the rest.

The v7 enters a 5% traffic canary for 7 days, under close monitoring. At the end of the canary, the online metrics confirm what the offline ones anticipated: an improvement in tone and helpfulness, equivalent latency, no new failure modes. Promotion approved. The v7 moves to the production label.

t = −31 days — The Deploy stage: the v7 adapter enters serving

The customer_support_v7 adapter is promoted to the serving cluster. Three pieces covered in separate posts come into play.

vLLM as the inference engine. The engine lives on Kubernetes, deployed via a dedicated Operator, as the post about LLM operators and the post about vLLM on K8s explain. The operator is responsible for detecting the new adapter in the registry, hot-loading it without restarting the engine (a native vLLM capability with --enable-lora), and directing traffic based on the label.

Disaggregated serving. As the post about disaggregated serving details, the system separates prefill (compute-intensive, throughput-bound) and decode (memory-intensive, latency-bound) into different GPU pools. The user’s request, when it arrives, prefills in a specialised pod and decodes in another, communicating over NVLink plus a shared KV cache fabric.

Multi-tenant GPU cluster. The H100 cluster serves several tenants, not just the insurer from Step 0. As the post about the multi-tenant cluster explains, isolation is realised on four planes: the Kubernetes namespace, ACLs over adapters (only the tenant’s namespace loads its LoRAs), partitioning of the KV cache per tenant (one tenant cannot read another’s cached prefixes), and a tokens-per-minute quota enforced at the gateway.

Synchronised prompt registry. The product’s system_prompt lives in Langfuse with the production label. The active version is customer_support_system_prompt, version 12. The gateway reads the prompt from Langfuse on the request path (with a few seconds of cache so it does not hammer the registry). Detailed in the prompt versioning post.

The result at t = −31 days: the combination (adapter v7, prompt v12, golden v12) is active and served. The system is ready for the request that will arrive 31 days later.

Moving forward: the user’s request crosses the system

Back to Step 0: 11:47:32 on a Tuesday. The user presses Enter. We go in real time, in milliseconds.

t = 0 ms — Entry through the gateway

The user’s browser POSTs to chat.aseguradora-ejemplo.com/api/chat. The traffic crosses the edge load balancer and enters the SaaS product’s API gateway. The gateway:

Authenticates the user’s JWT (an end customer of the insurer tenant).
Extracts the tenant_id and checks that its tokens-per-minute quota is not exhausted.
Resolves which model_id, adapter_id and prompt_id correspond to this tenant and product. In this case: llama-3-70b-int8 + customer_support_v7 + prompt label production.
Builds a unique trace_id (W3C TraceContext, propagable to OTel) and starts a root span.

At 8 ms, the gateway passes the request to the prefill pool.

t = 8 ms — Pulling the versioned prompt

Before serving, the OpenAI-compatible client the engine uses internally pulls the active system prompt. As the post about prompt versioning details, the pattern is:

prompt = prompt_registry.pull(
 name="customer_support_system_prompt",
 label="production", # now pointing at v12
)
# A local 30 s cache reduces the round-trip to 0.1 % of requests

The OTel span of the prompt pull carries the attributes gen_ai.prompt.id = customer_support_system_prompt, gen_ai.prompt.version = 12, gen_ai.prompt.label = production. They are propagated to all children.

t = 12 ms — RAG retrieval

The system needs context from the tenant’s knowledge base. It runs:

query_embedding = encoder.encode(user_query)
chunks = qdrant.search(
 collection=f"tenant_{tenant_id}_kb_v3",
 vector=query_embedding,
 limit=4,
 score_threshold=0.72,
)
reranked = reranker.rerank(user_query, chunks, top_k=2)

At 38 ms, the reranker returns two fragments: one from the Cancellation policy, section 4.2 and another from Premium plan benefits, section 2.1. As the post about PostgreSQL + Qdrant details, the tenant’s corpus is kept isolated by collection and ACL: no tenant can read another’s chunks.

t = 40 ms — Building the final payload

The engine composes:

[system_prompt v12]
+ [retrieved context: 2 chunks]
+ [short session history: 1 previous turn]
+ [user query]

Total: ~1,850 tokens of context. The OTel span records gen_ai.request.input_tokens = 1850, gen_ai.request.model = llama-3-70b-int8, gen_ai.request.adapter = customer_support_v7.

t = 45 ms — Prefill

The payload enters the prefill pool. The GPU processes the 1,850 tokens in a single parallel pass, computing for each token its K and V vectors (attention key and value). Those vectors are materialised as the KV cache, covered in detail in the KV cache fundamentals post. The resulting cache takes up ~120 MB of VRAM in INT8.

Here a key optimisation appears: the v12 system prompt is cached in the prefill pool (prefix caching, covered in the post about PagedAttention). Since the system prompt is the same for this tenant, the first ~500 tokens of the context are not recomputed: they are read from the prefix cache. That reduces the effective prefill from 1,850 tokens to ~1,350 tokens, saving ~270 ms of compute.

At 580 ms (effective prefill), the TTFT (time to first token) is ready. The first token leaves for the decode pool.

t = 580 ms — Decode (streaming)

The decode pool receives the prefilled KV cache and starts generation token by token. As the post about disaggregated serving details, the prefill/decode separation is what lets a multi-tenant system keep TPS stable: the decode pool is sized to sustain thousands of sessions decoding in parallel at a low cost per token, while the prefill pool is sized for bursts of short TTFT.

Generation at ~80 tokens per second. The answer will be ~290 tokens. Total decode time: ~3.6 s. Streaming: the user starts seeing words from t = 580 ms.

While decode advances, the engine emits child spans on every iteration with gen_ai.response.tokens_generated, gen_ai.response.cache_hit_ratio, gen_ai.response.cumulative_latency. The post about AgentSight and the post about MCP observability with OTel cover the detailed instrumentation of this layer.

t = 4,200 ms — Complete answer, root span closed

Generation finishes. The engine closes the root span with gen_ai.response.completion_tokens = 290, gen_ai.response.finish_reason = stop, gen_ai.response.total_latency_ms = 4200. The user sees the final answer. The session is ready for a next turn or for the user to click thumbs-up/thumbs-down.

By this point, every active stage has taken part:

Data (pre-existing): the indexed RAG corpus, the dataset that trained the adapter, the golden set that validated it.
Tune (pre-existing): the v7 adapter trained 45 days ago.
Eval (pre-existing): the gates that approved the promotion.
Deploy (at this very moment): vLLM + disaggregated + KV cache + multi-tenant.
Observe (at this very moment): the OTel spans emitted to Langfuse + Tempo, the metrics to Prometheus.
Retrain (about to activate): the feedback the user will leave in 15 seconds.

In parallel: Observe is watching

While the request happens, several pieces of Observe run in parallel and leave a structured trail.

OTel tracing. Every span (gateway, prompt pull, retrieval, prefill, decode) travels to Langfuse and to an OTel collector that forwards them to a backend (Tempo / Jaeger). The unique trace_id links all the spans. As the post about tracing with AgentSight details, end-to-end propagation is the main enabler of post-incident debugging: without it, you cannot reconstruct what happened three weeks later.

Runtime metrics. The engine emits Prometheus metrics per interval: gpu_utilization, kv_cache_usage, tokens_per_second, queue_depth, prefill_latency_p95, decode_latency_p95. The metrics are not tied to a trace; they are aggregated by tenant and service.

Online LLM-as-judge. A configurable percentage of answers (2% in this system) is also run through a judge LLM in the background, which scores the answer against a simple rubric (correct / partial / incorrect + a tone score). The judge does not block the answer to the user; it feeds the dashboard.

Statistical drift. In parallel, a slower pipeline computes drift over the distribution of inputs and outputs. As the post about eBPF + drift explains, low-level monitoring (latency, error rate per endpoint) is complemented with statistical drift detection (KS test, embedding distance) that spots when “something is wrong” before a thumbs-down confirms it.

Safety and guardrails monitor. The guardrails post describes the layer that watches for jailbreak attempts, PII leakage and prompt injection via MCP tools. In this case, none of them fires.

All these pieces operate continuously, not per request. But this particular request left its trail in every one of them.

The feedback: the loop closes

Fifteen seconds after reading the answer, the user marks a thumbs-down and writes in the form: “the answer is correct but the tone is too salesy; I only wanted to know how to cancel”. That apparently trivial gesture triggers an important sequence.

Insertion into feedback_signals

As the post about Retrain details, the thumbs-down is persisted as a structured row in a Postgres table:

INSERT INTO feedback_signals (
 signal_id, trace_id, request_id, signal_type, signal_value,
 prompt_id, prompt_version, model, user_segment, occurred_at
) VALUES (
 gen_random_uuid(),
 '4f5...', -- the trace_id from Step 0
 'r-22a...', -- request_id
 'thumbs',
 '{"vote":"down","reason":"too pushy","text":"I only wanted to know how to cancel"}',
 'customer_support_system_prompt',
 12,
 'llama-3-70b-int8+customer_support_v7',
 'premium-es',
 '2026-05-19T11:47:51+02:00'
);

With this, the row is linked by trace_id to everything that happened: prompt v12, retrieved context, complete output, latency metrics, judge score (0.82 in this case, considered good by the judge, though the human disagrees).

Triage by root cause

The MLE team runs triage the next morning. Combining heuristic rules, LLM-as-classifier and human review:

The signal is not a model issue: the model answered correctly to the prompt it received.
It is not a retrieval issue: the retrieved chunks were the right ones.
It is not an infra issue: latency was normal.
It is a prompt issue: the v12 system prompt instructs the model to “offer alternatives before processing destructive actions”. That instruction generates the “salesy tone” in some contexts.

The incident accumulates with others from the month in the “salesy tone” cluster. When the cluster crosses a threshold (typically 30-50 incidents of the same type or a percentage of the total), it will enter an incident-driven mini-cycle or wait for the quarterly Retrain, depending on the size.

The next cycle picks it up

Three months later, in the next quarterly Retrain, this feedback is one of many that will motivate two changes:

A new prompt version v13 with an adjusted instruction: “offer alternatives only if the user does not express a clear intention to cancel”.
A possible reinforcement of the adapter with cases of a more direct tone for premium-es. If the cluster justifies it.

The v13 will go into its own eval gate. The golden set will grow with cases where the correct tone is “direct, not salesy”. The v8 of the adapter (if it arrives) will retrain over the enriched dataset enriched_retrain_2026_q2 that already contains this annotated case.

The loop closes. The Step 0 request has contributed to the version of the system that will serve another user three months later.

What goes in each trace: identity and traceability

If the reader looks at the seven identifiers that are everywhere in this route, they see the network of identities that makes all of the above possible. It is the identity infrastructure of an LLM system in production:

trace_id 4f5... (unique per request)
request_id r-22a... (idem)
prompt_id customer_support_system_prompt
prompt_version 12
prompt_label production
dataset_id enriched_retrain_2026_q1
dataset_version v3 (sha256:9af...)
model_id llama-3-70b-int8
adapter_id customer_support_v7 (sha256:5c1...)
deployment_id d-prod-7b
schema_version 3.2
tenant_id aseguradora-ejemplo
user_segment premium-es
golden_set_id customer_support_golden_v12

If a single piece of that set is missing or does not propagate, the chain breaks. The next incident investigated will land in “we cannot rewind to the origin because the system did not record it”. That is why the cross-cutting components, prompt versioning and data versioning, are not luxuries: they are the connection without which the other six stages operate blind.

Synthesis diagram: how the pieces fit

 ┌─────────────────────────────────────────┐
│ User (end customer, B2C) │
└─────────────────┬───────────────────────┘
│ chat msg + JWT
▼
┌─────────────────────────────────────────┐
│ Edge LB + WAF + Cilium CNI │
└─────────────────┬───────────────────────┘
│ HTTPS, internal mTLS
▼
┌─────────────────────────────────────────────────┐
│ API Gateway (auth, quota, model routing) │
│ - Resolves tenant → model + adapter + prompt │
│ - Starts trace_id (W3C) │
└──────┬─────────────────────┬────────────────────┘
│ │
(pull prompt) │ │ (pull config)
▼ ▼
┌────────────────────┐ ┌──────────────────────┐
│ Langfuse Prompt │ │ Model registry │
│ Registry (v12) │ │ (adapter v7) │
└─────────┬──────────┘ └──────────┬───────────┘
│ │
└──────────┬───────────────┘
│ payload ready
▼
┌──────────────────────────────────────────┐
│ vLLM engine (K8s Operator) │
│ ┌──────────────┐ ┌──────────────┐ │
│ │ Pool prefill │ → │ Pool decode │ │
│ │ (H100×N) │ │ (H100×M) │ │
│ └──────┬───────┘ └──────┬───────┘ │
│ │ KV cache fabric │ │
│ └──────────────────┘ │
│ - prefix caching of the system prompt │
│ - PagedAttention │
└──────┬───────────────────────────────────┘
│ tokens stream
▼
┌─────────────────────────────────────────┐
│ User sees the answer + thumbs/UX UI │
└─────────────────┬───────────────────────┘
│ feedback (15 s later)
▼
┌─────────────────────────────────────────┐
│ feedback_signals (Postgres) │
│ + Langfuse scores │
└─────────────────┬───────────────────────┘
│
┌────────────────────────┼────────────────────────┐
│ │ │
▼ ▼ ▼
triage quarterly Retrain cycle dataset_id
root cause or incident-driven enriched (DVC)
│
▼
Tune of v8
(next cycle)
In parallel throughout the request, OTel instrumentation:
spans → Tempo / Jaeger ; events → Langfuse ; metrics → Prometheus

The on-premise stack applied

Taking the above to a generic on-premise consultant-profile infrastructure (RTX 4090 + 4×H100 SXM cluster):

Layer	Typical resources
Network plane	Edge LB (HAProxy / nginx ingress) + Cilium CNI with BGP, covered in Cilium BGP
K8s compute plane	RKE2 with two manager nodes + a GPU node pool
Production GPU plane	4× H100 SXM (NVLink, 80 GB each), partitioned via MIG into prefill/decode pools
Development GPU plane	1× RTX 4090 (24 GB) for offline evals, drift-check embeddings, smoke tests
Storage plane	MinIO or Ceph object store; DVC remote + lakeFS backend
OLTP data plane	Postgres 18 with replication; pgvector 0.8 for small cases
Vector plane	Qdrant or Milvus for large RAG corpora
Stream plane	Kafka (Redpanda / pure Apache) + Schema Registry; CDC with Debezium or Flink CDC
Observability plane	OTel Collector + Tempo (traces) + Prometheus (metrics) + Loki (logs); Langfuse for the LLM-specific part
Runtime security plane	Tetragon, covered in the runtime security post

The real density is not the sum of the boxes: it is the operations that tie the boxes together. A cluster with every piece but no versioning discipline, no end-to-end trace_id propagation, no schema contracts and no cadenced retraining is a cluster that serves LLM once and then ages. The difference between a project and a platform is exactly that.

Ten bridges between stages where the system breaks

The route reveals something important: failures are rarely inside a stage; they are on the bridges between stages. Ten common bridges:

Data → Tune: the dataset does not propagate its (dataset_id, dataset_version) to the trainer. The same dataset trained twice produces two model_id values that cannot be told apart.
Tune → Eval: the trained model does not propagate its lineage to the eval run. The eval passes, but there is no record of which dataset it was trained on. Three months later, irreproducible.
Eval → Deploy: promotion happens without the serving system recording which version of the adapter it is serving at each moment. The day the model gives a dangerous answer, nobody knows which adapter answered.
Deploy → Observe: the engine does not emit gen_ai.request.adapter, gen_ai.prompt.version, gen_ai.dataset.version as span attributes. The traces exist but cannot be crossed with the lineage.
Observe → Retrain: the feedback is captured in a tool (Langfuse, Phoenix) but nobody reads it. The Retrain stage “exists”, but feedback piles up untriaged.
Retrain → Data: the enriched dataset goes into the next Tune without passing through versioning discipline, schema contract and holdout check. Silent contamination of the golden set.
Prompt versioning ↔ everything: the prompt_id, prompt_version does not propagate to the spans. The day the team discovers that a prompt change regressed the system, it cannot isolate which one or when.
Data versioning ↔ everything: the dataset_id, dataset_version does not show up in experiment tracking. “v8 is retrained” but nobody can prove it was over the enriched dataset and not the old one.
MCP ↔ tools: the system invokes tools (cancellation, policy changes) but does not record a gen_ai.tool.invocation_id linked to the trace. The actions end up dissociated from the answer that generated them.
Schema Registry ↔ data: datasets version content but not schema. A breaking change in expected_output breaks the eval silently; nobody notices anything until a human reviews the results.

The bridges are covered throughout the blog. Operations enforce them. The team’s culture keeps them.

How to walk through the blog

If you arrive at this post from outside and want a reading route:

The map: The six-stage LLMOps pipeline — the master map of everything else.
The context: MLOps specific to LLMs in 2026 — the landscape and why LLMOps is not classic MLOps.
Inference from the inside out: KV cache → PagedAttention deep dive → Disaggregated serving → Multi-tenant GPU cluster → vLLM on K8s → LLM operators on K8s.
Data: Data versioning with DVC and lakeFS → PostgreSQL + Qdrant ingestion → RAG over Kafka.
Tune: Continuous fine-tuning in production.
Eval: Evals: the layer after tracing → Guardrails and safety.
Observe: AgentSight LLM tracing → MCP observability with OTel → On-device eBPF + drift.
Retrain: Closing the feedback → dataset → adapter loop.
Cross-cutting: Prompt versioning with Langfuse and MLflow.
Supporting infrastructure (the base everything is built on): RKE2 with Cilium BGP, Hubble + eBPF observability, Tetragon runtime security.

What we have not covered (yet)

At the first level the main things are there. The following posts on the blog, when the topics justify it, could go deeper into:

Schema Registry for LLM data and prompts: the other half of the data contract.
A dedicated AI Gateway: LiteLLM, Portkey, Kong AI Gateway as a control plane.
OTel gen_ai semantic conventions: the emerging standard that ties the seven identifiers of the “identity” block into well-formed spans.
Federated learning over regulated customer data: how to train without centralising the corpus.
Capacity planning for shared multi-tenant clusters.
Disaster recovery of an LLM service: how to reproduce the state of the system 30 days back.
Cost accounting per tenant: tokens × weights × adapter × infrastructure → invoice.

References

W3C Trace Context — end-to-end propagation of traceparent and tracestate.
OpenTelemetry GenAI Semantic Conventions — gen_ai.* attributes for LLM spans.
Langfuse documentation — observability and prompt registry.
vLLM documentation — production inference engine with PagedAttention and LoRA hot-swap.
Kubernetes Operators — the declarative management pattern.
MLflow Tracking and Model Registry — lineage of runs and input artifacts.
DVC and lakeFS — dataset versioning, unified in Nov 2025.
OpenLineage — an open standard for lineage events between systems.
ENS / NIS2: compliance frameworks that apply to operators in the EU; recommended reading for the context in which the scenario operates.

Real ClickHouse capacity in Langfuse v4: 2,782 bytes per observation, and why almost half of that is not data

Tue, 15 Sep 2026 09:30:00 +0200

Sixth article in the series on operating self-hosted Langfuse v4. The previous five cover what goes into a trace, putting LangGraph in front, two well-known agents instrumented, migrating with no window and the worker queues. This one measures what all of that takes up. Benchmark run on 15 September 2026 against the event schema of version 4.36.0, identical to that of 4.35.0.

TL;DR

Langfuse publishes no capacity figure at all. Not bytes per row, not events per second, not expected size. I searched their code with every reasonable combination and there is nothing. The only quantified thing is the commercial quotas of their managed service, which say nothing about disk.

One observation takes 2,782 bytes of disk. Split across the two event tables: 2,347 in the full table and 435 in the listings one. Measured over two hundred thousand observations shaped the way a real agent produces them.

41.8 % of the full table’s disk is not data, it is indexes. Four full-text indexes add up to 187 MiB against 260 MiB of data. The one covering the input costs 523 bytes per observation, almost three quarters of what the column it indexes costs.

The materialized view trims the text eight times over and saves eighteen per cent of disk. Truncating input and output to two hundred characters leaves the text at 64 MiB of the original 540, and even so the second table adds 18.5 % on top of the first. What survives the trimming is the seventy-eight-column skeleton and a metadata field that is truncated element by element, not as a whole.

In the listings table, metadata costs more than input and output together. 132.6 bytes per observation against 101.8. And its full-text index adds another 111.9, almost as much as the column.

A compression ratio quoted without saying what was compressed is worth nothing. My first pass, with repeated text, gave 38 times. With text of an entropy close to that of natural language it gives 3.68. Same schema, same indexes, same version.

Generations are 11 % of the observations and 65 % of the bytes. Their mean input is 15,655 bytes and the 95th percentile is 55,205, because each one reserialises the full history. It is the quadratic law from the second article in the series, landed on disk.

On deletion, the disk does not go down. A delete stops showing the rows and leaves the space occupied. A second step is needed, applying the deleted mask, and the cron that does it ships disabled, just like the retention one. On top of that the cleaner works table by table, so the second table keeps the rows the first one no longer shows.

And automatic retention is a paid feature when self-hosted. The data-retention entitlement is not in the open edition. Without it, the event tables grow without limit: they carry no expiry clause.

You are here: the sizing you cannot do by reading

The previous five articles describe how version 4 works and how to put it to work. The moment arrives to ask for disk, and there the series runs out of sources.

The official documentation gives a container sizing: two CPUs and eight gibibytes for ClickHouse, in Kubernetes eight of request and sixteen of limit, a hundred gibibytes of disk and three minimum replicas. It is a reasonable starting point and it does not answer the only question that matters, which is how long that disk lasts.

I went looking for it in the code, which is what this series does. I searched for events per second, rows per observation, bytes per row, capacity, benchmark, sizing, across the whole repository and its documentation. There is no figure at all. The closest thing is the ingestion quotas of their managed service, which are commercial API limits, and some working notes on a synthetic load generator that refer to the tables of the previous version and bring no results, only parameters.

So the only honest way out was to measure it. This article is that measurement, with the whole method laid out so that anyone can repeat it and contrast it with their own load.

The analogy: the archive and the index card

A municipal archive keeps case files. Each file has its folder with everything inside, and on top of that the archivist fills in an index card with the header data and the first lines of the subject, so that the file can be searched without pulling out the folder.

Intuition says the cards take up nothing compared with the folders. And that is true if you look only at the paper. It stops being true when the archivist, in order to be able to search by any word, also builds a term file: a box with one card per distinct word appearing in any case file, pointing to where it occurs. That box grows with the vocabulary, not with the number of case files, and ends up taking almost as much room as the case files themselves.

In this setup the folders are the full table, the cards are the listings table, and the term file is the full-text indexes. The article measures the three things separately, because whoever sizes by looking only at the folders falls short by almost half.

The benchmark

I want this to be reproducible, so the method goes in whole.

The schema is the real one. I extracted the ClickHouse migration DDL from the repository, in its single-instance variant. There are not three migrations but six: the three that create the two tables and the materialized view, plus one that adds three ingestion attribution columns, another that adds an n-gram index over the metadata, and another that adds three evaluation columns and two indexes. The result is 78 columns per table, eleven skip indexes in the full table and fourteen in the listings one.

I verified that this DDL is identical between the version I have pinned across the series and the current head of the repository: the diff of the migrations directory is empty.

The engine is ClickHouse 26.9.1. The minimum version 4 demands is 25.12, because of the text indexes the schema needs. I am using a newer version, and that is a difference from an installation that sticks to the minimum.

The data is synthetic and correctly shaped. The benchmark does not invent the rows: the columns and their typical values come from the worker function that builds the record. I write only into the full table, which is what the worker does, and let the materialized view populate the second one. That reproduces the real write amplification.

The topology of each trace reproduces the law we measured in the second article of the series: four fixed observations plus five for each tool call. The number of calls per turn comes from a distribution with a low median and a long tail, capped at fifteen. Generations carry the full message history, which grows with the turn, because that is exactly what the instrumentation does and it is the origin of the quadratic growth in bytes.

Result: 200,012 observations across 22,563 traces, that is 8.86 observations per trace.

And the entropy of the text is calibrated, which is the part that almost ruined the benchmark. My first pass used a repeated paragraph as filler. With that text, the full table compressed 38.1 times and the figures came out beautiful and false. A repeated lorem ipsum is the best possible input for a compressor. I redid the generator with a wide vocabulary sampled from a Zipf distribution and fifteen per cent high-entropy identifiers, calibrated to give the ratio that natural language mixed with identifiers gives.

With that text, the same table compresses 3.68 times. Ten times less. Any ClickHouse capacity figure that does not say what was compressed is noise, and mine would be too without this paragraph.

What one observation takes up

The two tables, after forcing the final merge:

	events_full	events_core
Rows	200,012	200,012
Uncompressed	957.87 MiB	328.73 MiB
Compressed	260.29 MiB	60.34 MiB
Ratio	3.68	5.45
On disk	447.68 MiB	82.92 MiB
Raw per observation	5,022 B	1,723 B
Compressed per observation	1,364.6 B	316.3 B
On disk per observation	2,347 B	435 B

The figure to take away is the last row added up: 2,782 bytes of disk per observation, counting both tables, with one replica and no retention.

And it is worth looking at the distance between the last two rows. The compressed data of the full table is 1,364.6 bytes per observation, but on disk it takes 2,347. The difference, 982 bytes per observation, is the next section.

Where the bytes go

Broken down by column, with what each one costs per observation:

Column	events_full	events_core
`input`	717.5 B	61.5 B
`metadata_values`	389.0 B	132.6 B
`output`	174.7 B	40.3 B
`tool_definitions`	28.0 B	28.1 B
`blob_storage_file_path`	12.6 B	12.5 B
`span_id`	8.1 B	8.0 B
The other 67 columns together	17.3 B	13.3 B

Three things jump out.

The first is that the skeleton is cheap. Sixty-seven columns, most of them empty in a normal observation, cost seventeen bytes between them all. ClickHouse compresses a column of empty strings down to almost nothing. Anyone who feared that the thirteen experiment columns or the telemetry ones would weigh can stop fearing it.

The second is that in the full table the input is half of everything. 717.5 of 1,364.6 compressed bytes. It is the direct consequence of each generation reserialising the history.

The third is the one I was not expecting. In the listings table, metadata costs more than input and output together: 132.6 against 101.8. The reason lies in how the materialized view truncates. It trims input and output to two hundred characters each, but it trims the metadata element by element: an array with forty values can survive with eight thousand characters. The table that exists in order to be light fills up with the one thing its truncation does not control.

The split by observation type

Type	Observations	Mean input	Input p95	Mean output
GENERATION	21,952	15,655 B	55,205 B	1,077 B
AGENT	22,563	2,585 B	6,013 B	1,300 B
TOOL	21,952	319 B	319 B	2,008 B
CHAIN	67,689	413 B	413 B	20 B
SPAN	65,856	272 B	272 B	206 B

Generations are 11 % of the observations and contribute 65 % of the input and output bytes. The 95th percentile of their input is 55 kilobytes, in a benchmark where no conversation goes beyond fifteen tool calls.

This has a sizing consequence worth saying out loud: the size of the database is not determined by the number of requests, it is determined by the depth of the turns. Doubling the users doubles the disk. Doubling the tool calls per turn multiplies it by rather more than two.

The indexes, which are almost half

Here is the finding that changes a sizing exercise the most.

	Data	Indexes	% of disk in indexes
events_full	260.29 MiB	187.31 MiB	41.8 %
events_core	60.34 MiB	22.49 MiB	27.1 %

The marks and the primary key are irrelevant: thirty-two kilobytes and three hundred bytes respectively, with the granule size of 64 mebibytes the full table has declared.

Broken down, and ordered by what they cost:

Index	Table	Bytes per observation
`idx_fts_input_low`	events_full	523.1
`idx_fts_metadata_values`	events_full	319.4
`idx_fts_output_low`	events_full	133.5
`idx_fts_metadata_values`	events_core	111.9
`idx_fts_metadata_names`	both	4.3
`idx_span_id`	both	1.2
The remaining Bloom filters	both	0.1 each

The full-text index over the input costs 523 bytes per observation when the column it indexes costs 717. That is, indexing the input costs almost three quarters of what storing it costs. And the metadata one in the listings table costs 111.9 when its column costs 132.6: eighty-five per cent.

The Bloom filters, on the other hand, are free as far as sizing goes. The four identifier ones and those for model name, experiment and evaluator add up to less than two bytes per observation between them all.

This is not a criticism of the design. Those indexes are what make text search in the interface respond, and without them the experience would be another thing. It is sizing information that is published nowhere: when asking for disk for ClickHouse you have to multiply what the columns say by 1.7.

The materialized view as an amplifier

The second table exists so that listings do not have to touch the large fields. It truncates input and output to two hundred characters and does without the text indexes over them.

The trimming works: the benchmark’s 540.55 MiB of input and output come down to 64.68 MiB, eight and a half times less.

And even so the second table adds 18.5 % of disk on top of the first. For the reasons already seen: the skeleton survives, the metadata truncated element by element survives, and its own text index over that metadata costs another 111.9 bytes.

It is worth keeping in mind because it is counter-intuitive. You would think a table storing two hundred characters of text is an appendix. It is four hundred and thirty-five bytes per observation, almost a fifth of the total, and it is paid always, because the materialized view cannot be switched off without leaving the interface with no listings.

What happens on deletion

The retention cron runs a lightweight delete. I reproduced it: I deleted the observations older than a date, which removed 67,395 rows of the 200,012.

Moment	Visible rows	Rows in the parts	Disk
Before	200,012	200,012	447.68 MiB
After the delete	132,617	200,012	447.68 MiB
After applying the mask	132,617	132,617	299.11 MiB

The delete frees not one byte. The rows stop being visible and the space stays occupied. The disk only goes down at the second step, applying the deleted mask, which frees 33.2 %.

And that second step is run by a different cron that ships disabled, just like the retention one. With the default values, whoever switches retention on in the interface will see the traces disappear from the view and will never see the disk go down, until an ordinary merge drags in the affected parts, which can take however long it takes.

There is a third detail. The cleaner works table by table, with an independent deleter for each one. In my test I deleted in the full table and the listings one kept its 200,012 rows, because the materialized view does not propagate deletes. In a real installation both deleters run, but they are two separate operations that can diverge if one fails.

On top of this you have to add what the fourth article already covered: neither of the two tables carries an expiry clause. There is no automatic expiry in the engine. The whole lifecycle depends on those crons. And per-project retention requires an enterprise edition entitlement, so in the open edition the only retention possible is the one you write yourself.

Projection

With the measured 2,782 bytes per observation, one replica and no retention:

Observations per day	Agent turns per day	Per day	Per month	Per year
100,000	~11,300	0.26 GiB	7.8 GiB	94.6 GiB
1,000,000	~112,800	2.59 GiB	77.7 GiB	945.6 GiB
10,000,000	~1,128,000	25.91 GiB	777.2 GiB	9,456 GiB

The conversion to turns uses the benchmark’s 8.86 observations per trace, which depend on the depth of the agents. With deeper agents, the same turn figure produces more observations.

And there is still a multiplication left. Langfuse recommends three minimum replicas of ClickHouse in Kubernetes. With one million observations a day and three replicas, a year is 2.8 tebibytes of raw disk. The official sizing of a hundred gibibytes per replica covers about thirty-eight days at that rate.

That is the number that is published nowhere and that has to go into the spreadsheet before committing to a service.

What this benchmark does not measure

For the sake of honesty, and because anyone repeating this is going to get different numbers:

The data is synthetic. The shape of the row and the topology of the traces are taken from the code and from the measurements in the previous articles, but the text is generated by a sampler, not by a model. I have calibrated its entropy so that compression resembles that of natural language, and that calibration is the largest source of error in the whole article. A load with a lot of structured JSON will compress better; one with a lot of code or a lot of base64, worse.

It is a single instance, with no replication. In a replicated installation there is a different engine and there are copies.

It is ClickHouse 26.9.1, above the 25.12 minimum. The text index implementations have changed between versions and I have not compared.

It does not measure performance: not ingestion per second, not query latency, not the CPU cost of merges. Disk only. The merge cost of full-text indexes is a topic with a life of its own and is left for the saturation runbook.

And it does not measure object storage, which in version 4 is mandatory and keeps a copy of the original event. That is additional disk this benchmark does not touch.

Checklist

Do not size using the official container sizing: it covers CPU and memory, not disk growth.
Count 2,782 bytes per observation as a starting point, and adjust by measuring your own load.
Multiply by the number of replicas before asking for the volume.
Multiply what the columns suggest by 1.7, because the indexes are 41.8 % of the full table’s disk.
Measure your own installation with the system tables instead of trusting this article: active parts for the total, columns for the breakdown, skip indexes for what is not data.
Watch the metadata, which is the only thing the materialized view’s truncation does not control, because it trims each element and not the whole.
Size by turn depth, not by number of requests.
Switch on both crons, the retention one and the deleted mask one, and check that the disk really goes down after the second.
Reckon on per-project retention requiring a licence, and budget the work of writing it if you do not have one.
Remember that the tables have no expiry in the engine: with no cron, they grow for ever.

Traps and things that are not what they look like

Langfuse publishes no capacity figure at all, neither in its code nor in its documentation.
A compression ratio without saying what was compressed means nothing: the same schema gives 38 times with repeated text and 3.68 with realistic text.
41.8 % of the full table’s disk is indexes, not data.
Indexing the input costs almost as much as storing it: 523 bytes per observation against 717.
The 78-column skeleton is irrelevant: seventeen bytes per observation across the sixty-seven small columns.
In the listings table metadata costs more than input and output together.
The materialized view’s truncation trims each element of the metadata, not the whole array.
The listings table is not an appendix: it adds 18.5 % of the first table’s disk despite storing two hundred characters.
A delete frees no disk. The mask has to be applied, and that cron ships disabled.
The retention one also ships disabled out of the box.
The cleaner works table by table; the materialized view does not propagate deletes.
The event tables carry no expiry clause.
Per-project retention is a paid feature when self-hosted.
Generations are 11 % of the observations and 65 % of the bytes.
Bloom filters are free as far as disk goes; the full-text ones are not.

Closing

What I take from this benchmark is not the figure, although the figure was needed. It is that the split is not where you look for it.

When somebody sizes a trace database, they look at the size of the prompts. It is the natural thing to do: they are the large fields and they are the ones you see in the interface. And it turns out that in this architecture the prompts are half the compressed data, and the compressed data is a little over half the disk. The rest is search structures nobody counts because they do not show up when you look at a row.

There is a pattern there that goes beyond this tool. An observability database is, by definition, a database queried in unforeseen ways: by free text, by user, by session, by whatever is needed when something goes wrong at three in the morning. That ability to search by anything is paid for in disk, always, and it is paid at write time and not at search time. Langfuse’s design pays it fairly explicitly, with four full-text indexes declared in the migration, in plain sight for anyone who wants to read them.

What was missing is for somebody to put the number alongside. Now it is there, with its margin of error and its method on the table, so that the next person to measure it can tell me where I got it wrong.

The series: the ten articles

What goes into a trace: version 4’s data model, limits, precedences, scores, masking and indexes.
Putting LangGraph in front: instrumenting an agentic platform and the measured cost of one turn.
Two well-known agents instrumented: Open Deep Research and GPT Researcher, with the measured figures of a real request.
Migrating from version 3 to 4 with no window: the three write-mode steps, the resumable background migrations and where the rollback point of no return sits.
The worker queues: the map of all thirty-nine, which pool to dedicate to each group, the per-queue switches, sharding and concurrency.
Real ClickHouse capacity and cost (this article): how to measure bytes per observation with the system tables, the difference between the full table and the listings table, and the merge cost of full-text indexes.
Retention, deletion and data protection: why a deletion does not free disk, the mask cleaner that ships disabled, the pending deletion queue and the S3 lifecycle that has to be implemented by hand.
Backup and cross recovery: restore ordering across Postgres, ClickHouse and object storage, what each mismatch breaks, and how far event replay goes.
Saturation runbook: what to alert on from the queue metrics, the stuck probes, draining through the readiness endpoint and the dead letter queue.
Getting the data out: the blob storage integration to Parquet, batch exports and the metrics API, to build the data lake.

Sources

Langfuse, migraciones de ClickHouse (packages/shared/clickhouse/migrations/canonical/, ficheros 0039 a 0047): https://github.com/langfuse/langfuse.
Langfuse, escritura del worker en ClickHouse (worker/src/services/IngestionService/, worker/src/services/ClickhouseWriter/): https://github.com/langfuse/langfuse.
Langfuse, limpiador de retención por lotes y limpiador de máscaras de borrado (worker/src/features/batch-data-retention-cleaner/, worker/src/features/deleted-mask-cleaner/): https://github.com/langfuse/langfuse.
Langfuse, guía de autoalojamiento y requisitos de infraestructura: https://langfuse.com/self-hosting.
ClickHouse, tablas del sistema parts, columns y data_skipping_indices: https://clickhouse.com/docs/en/operations/system-tables.
ClickHouse, índices de salto y de texto completo: https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/mergetree.
ClickHouse, borrados ligeros y máscara de borrado: https://clickhouse.com/docs/en/guides/developer/lightweight-delete.

deepagents on a cluster of your own: the SDK is free, the server is not, and what you have to build in between

Tue, 15 Sep 2026 07:00:00 +0200

Opening of the agentic platform vertical. The previous articles dealt with the pieces separately: the inference gateway, its second door, what a fleet of agents costs and isolation. This one deals with what goes on top. Verified against deepagents 0.7.14 and the LangGraph repository, commits of 14 September 2026.

TL;DR

deepagents is not a runtime. It is a factory that assembles middleware and delegates to LangChain’s agent factory. It returns a compiled LangGraph graph and nothing else. The repository’s own documentation says so without decoration: it introduces no new runtime. That is good news for whoever deploys it, because it runs wherever Python runs.

The planning tool no longer exists. It disappeared in 0.7.0 as a breaking change, along with the state channel that backed it. Almost everything written about this harness still describes it as one of its three legs. Today you have to ask for it explicitly, and you have to ask for it in each subagent too.

With the default configuration, the agent’s files live inside the checkpoint. The state backend stores the full content, binaries in base64, in the same place the conversation is persisted. Changing that is one line and it is the first architecture decision of the deployment.

The database grows per loop step, not per turn. Four tables, one row per superstep, one per channel and version, another per task write. And of the four cleanup operations the base class declares, the Postgres implementation has only one: delete an entire thread. The other three raise not-implemented.

Pruning by hand can empty the history silently. The code itself warns that a partial delete breaks the delta chain and leaves the channels rebuilding empty without raising any error. The only safe pruning is by whole thread.

Tenant isolation is the thread identifier, and there is no owner check. Whoever can send a thread identifier reads that thread. The on-disk filesystem backend fixes its root when the agent is built, so it separates nobody. And the documentation’s example for computing the per-user namespace does not work outside the managed platform.

There is a configuration trap that leaves a short-window model uncompacted. If the model does not expose its window, the compaction threshold falls back to 170,000 tokens. Served on a vLLM with a 32,768 window, the agent blows up on context length long before it compacts. The failure is silent and is fixed by passing the profile by hand.

Observability without the platform does exist, and it goes through a package that is not obvious. LangChain’s core does not emit OpenTelemetry: zero references. The exporter lives inside the LangSmith SDK, and there is a mode that emits over OTLP only, without talking to the service. It is the road to a Langfuse of your own, and it is worth knowing where it passes.

And here is where the free part ends: the server. The SDK, the graph engine and the checkpointers are MIT. The server that publishes them over HTTP carries an Elastic 2.0 licence, asks for a commercial licence key for production, adds Redis to the stack and reports metadata to an external endpoint unless there is an isolation agreement. The development tool is explicitly for development.

What is left is a list of things you have to write. Per-thread authorisation, retention, a run queue, cancellation, clean draining and migrations. None of them is hard. All of them are work, and none of them appears in the twenty-line front-page example.

You are here: the harness arrives after the model

The sequence repeats on every project. First you set up inference, which is the visible part. Then the gateway, because you need keys and budgets. Then observability, because nobody knows what is going on. And when all of that works, somebody asks why the assistant cannot do multi-step tasks, and the word agent enters the scene.

At that point the temptation is to write the loop by hand, and for two weeks it looks like a good idea. The loop is easy. What is not easy is everything you discover afterwards: what happens when the context fills up, where the intermediate files are stored, how you resume a task that was left half done, how you ask a human for confirmation without losing state.

An agent harness is exactly that: the packaged answer to those questions. And choosing one looks like a library decision, about the size of choosing an HTTP client. It is not. It decides where your users’ state lives, which database grows, what can be isolated and what cannot, and in the case at hand, whether there is a commercial licence waiting at the end of the road.

This article looks at deepagents with those eyes. It is not a usage guide, and I do not assess whether its agents are any good. I look at what it forces you to build.

The analogy: the scaffolding and the crane

A building site needs scaffolding and it needs a crane, and they are not the same thing even though both are metal structures surrounding the building.

You put up the scaffolding yourself with parts you bought. It adapts to the façade, it can be extended, and the day the work finishes you take it down and put it away. Nobody charges you for using it once it is bought.

The crane is another matter. You hire it, it comes with an operator, it has a serial number and you have to notify somebody when it goes up. Without a crane the work still moves forward, more slowly and with more people, but it moves forward.

Here the scaffolding is the SDK and the graph engine, and it belongs to whoever buys it. The crane is the server that exposes the agent over HTTP, and that one comes with a contract. This article is about where exactly the line between the two sits, because you cannot see it in the brochure, and about how much work can be done without a crane.

What deepagents actually is

The first thing to get out of your head is that it is a system. It is a function.

The main factory assembles a list of middleware and calls LangChain’s agent factory, which is what builds the graph. What it returns is a compiled graph, with an added configuration on top that raises the recursion limit to a very high number and adds an integration tag in the metadata. The repository’s own architecture document states it: no new runtime is introduced.

Compiled with the default options, the graph has the nodes you would expect from a reasoning-and-action loop: start, a prior node that repairs incomplete tool calls, the model node, the tools node and the end. The conditional edges run from the model to tools, to the model itself or to the end. There is nothing more. Everything that looks like a system is middleware that adds nodes only when it declares hooks of its own.

The tools that ship by default are the filesystem ones, plus the subagent delegation one: list, read, write, edit, delete, search by pattern, search by content, execute, and the task tool.

The good consequence of this is portability. If something is a LangGraph graph and nothing else, it runs in any Python container, with the checkpointer you choose, against the model you choose. There is no hidden service behind it.

What is no longer there

It deserves its own section because it contradicts almost everything published. The planning tool, the one that wrote a task list and kept it in a state channel, was removed in 0.7.0 as a breaking change declared in the changelog. The tool is gone, the channel is gone, and the prompt fragment that went with it is gone.

It survives as LangChain middleware, and you have to ask for it by hand. With one detail that bites: you have to ask for it in each subagent too, because each subagent assembles its own stack. All that remains from the previous stage are archaeological leftovers, such as the state key that is still on the list of keys excluded when passing context to a subagent.

The plan, today, rests on files. Which is consistent with the rest of the design, and is another reason to look closely at where those files live.

Where state lives

This is the first architecture decision and the one that is most expensive to get wrong.

The agent’s state has two large channels. Messages, with a reducer of its own that deduplicates by identifier and treats deletions as tombstones. And files, with its own reducer where a null value means deleted. Both use a channel type that writes deltas and stores a full snapshot every fifty updates, and the code comment explains why: to take checkpoint growth from quadratic to linear with respect to the number of messages.

That channel type is marked beta in its own code, with a warning that the on-disk representation may change. Worth knowing before you rest a platform on top of it.

The backends

There is a common interface and several implementations. The ones that matter for deciding:

Backend	Where the files live	Persistence
State	Inside the graph state, that is, inside the checkpoint	Per thread, in the checkpointer’s database
Filesystem	In a real directory anchored to a root	Wherever you mount the volume
Store	In LangGraph’s store, with a namespace	Crosses threads, with a time to live
Composite	Routed by path prefix among the above	Mixed

The default backend is the state one. That is: by default, everything the agent writes ends up inside the checkpoint, and binaries go base64-encoded inside the same JSON. For a demo it is convenient, because you have nothing to set up. For a platform with real users it is a sizing decision taken by omission.

There is a detail that rounds the matter off. The filesystem middleware automatically evicts large tool results to a file when they exceed a token threshold, so that they do not occupy the context window. It is a good mechanism. But with the state backend, evicting to a file means moving the content from one place in the checkpoint to another place in the same checkpoint. It leaves the model’s window and it does not leave the database.

What grows in the database

The Postgres checkpointer creates four tables: one for migrations, one for checkpoints, one for blobs by channel and version, and one for writes by task. The primary keys all start with the thread identifier, and there is an index per table on that column.

What you have to internalise is the rhythm. One checkpoint row is written per engine superstep, not per conversation turn. A turn with four tool calls is several supersteps. None of that is recycled on its own.

And here comes the uncomfortable part. The checkpointer’s base class declares four maintenance operations: delete a thread, delete by run, copy a thread and prune with a strategy. In the Postgres implementation only the first is implemented. The other three inherit the not-implemented exception from the base class. Deleting a thread is three delete statements by identifier, and that is it.

In other words: retention is yours to build. And with a care that the code itself documents and that is worth quoting in full, because it is the kind of warning you read after the incident. A naive pruning that removes intermediate checkpoints and their writes can cut the delta chain; the surviving checkpoint is rarely a snapshot point, so its channels would rebuild empty without any error being raised.

Translated into operations: the periodic sweep deletes whole threads and only whole threads. Never loose rows. And since the checkpoints table has no date column, to know which thread is old you have to keep count outside or rely on the checkpoint identifier being a time-sortable UUID.

There is a variant of the checkpointer that stores only the last state and retains no history, which caps growth at the root. With one important reservation to verify before using it: it is not clear that it is compatible with the delta channels this harness uses by default for messages and files. Combining the two without checking is asking for a silent problem.

The store, which does have cleanup

Curiously, the sibling piece is better resolved. The store has a configurable time to live, with refresh on read, skipping of expired entries and a sweep interval, and the sweep is a delete by expiry date.

Two warnings. The sweeper is a thread inside the process, so with several replicas there are several sweepers competing for the same delete; it is idempotent, but it is free contention and you avoid it by switching it off and putting the delete in a scheduled cluster job. And the store’s semantic search requires the vector extension in the database, which constrains the managed Postgres image, which normally does not carry it.

The isolation that does not exist

If the platform is going to have more than one tenant, this is the section that decides the deployment topology.

The only thing separating one thread from another is the thread identifier. It is the first column of the primary key in all three tables. There is no owner check anywhere in the checkpointer. Whoever manages to send a thread identifier in the call configuration reads that thread. Authorisation is the responsibility of the layer you write on top, and since that layer also has to be written, it is worth noting down now.

By backend, the situation differs:

The state one inherits thread isolation, which is enough if the HTTP layer validates that the user owns the thread.
The store one computes the namespace per call, through a function that receives the runtime context. It is the correct option for multi-tenant.
The filesystem one fixes its root in the constructor, when the agent is built. There is no path in the code that recomputes it per request, per thread or per tenant. Sharing one deployment between tenants with this backend means everybody sees the same directory.

On that last point, do not confuse it with the protection that does exist. Virtual mode, which now comes enabled by default, blocks paths that escape the root. That protects against directory traversal. It does not protect against the neighbour. The project’s own threat document admits it: that mode exists to support the composite backend’s prefix routing, not as a security boundary.

There is also a documentation trap worth flagging because it costs an afternoon. The example the documentation proposes for computing the per-user namespace reads the identity from a server-information field of the runtime context. That field is annotated in the LangGraph code as metadata injected by the managed server, and null when running open-source LangGraph without managed deployments. On a cluster of your own with an HTTP layer of your own, that example fails. What you have to use is the context you fill in yourself, with the identifier that comes validated from the identity provider.

And one more boundary that gets crossed without warning: the memory and skills middleware interpolate file content into the system prompt as is, without sanitising. If two tenants share a directory that serves as the source for that, the first writes instructions that the second’s agent executes. It is prompt injection through shared storage, and it is avoided with the same measure as everything above: do not share the filesystem backend between tenants.

The topology conclusion is short. Real multi-tenant with files means one deployment per tenant, with its namespace, its volume and its quota, or else the store backend with a namespace computed per request. There is no third way.

The model, and the trap you need to know about

The default model is a specific commercial provider, and it is marked deprecated: passing an empty model warns and will stop working. You can inject any LangChain chat model, which is what anyone serving their own weights will do.

With two integration details that are not obvious.

The first. To point at an OpenAI-compatible endpoint, which is how a vLLM behind a gateway looks, you have to pass the already-built instance, not the string with the provider prefix. With the string, a provider profile is applied that forces the use of the responses API, which a vLLM normally does not implement. Passing the built object applies no profile and the agent stays clean.

The second is the trap, and in my view it is the most likely configuration failure in the whole deployment.

The compaction middleware computes its thresholds from the model profile. If the model declares its input window, it compacts on reaching 85 % of the window and keeps 10 %. If it does not declare it, it falls back to a fixed value: 170,000 tokens.

The chain that leads to not declaring it is the usual one in a deployment of your own. The OpenAI client resolves the profile by looking the model name up in a static table of the provider’s models. A model served as qwen3-30b-a3b is not in that table, so the profile comes out empty, and empty becomes null. The method that assigns it swallows any exception, so there is no warning.

Result: a vLLM with a 32,768-token window will not compact until 170,000. The server will return a context-length error long before that, and it will do so intermittently, only when conversations get long. There is a safety net that catches the error and trims, but that is reactive recovery after a failed call, not planning.

The fix is one line, because the profile is a public field and only fills itself in when it is empty: you pass it when building the model, with a value below the real window to leave room for generation. And it deserves an assert at start-up, because it is the kind of thing nobody looks at until it fails.

A related sizing note. With the default thresholds and a 32,768 window, compaction would trigger at 27,852 tokens, while eviction of large tool results is fixed at 20,000. That is, a single tool result can occupy almost the whole budget before anything evicts it. With 128,000 windows the defaults are proportionate; below 64,000 you have to lower those two thresholds by hand.

The tools, and the MCP that is not where it seems

This harness has its filesystem tools and its delegation tool, and the rest are passed as an ordinary list of LangChain tools.

The fact that matters to anyone with an MCP gateway in place: the SDK does not integrate MCP. There is no reference to the protocol in the package, nor any dependency on adapters. The integration lives in the command-line tool that accompanies the project, which does depend on the MCP adapters for LangChain, resolves configurations in the style of the desktop clients, and turns each remote tool into a LangChain tool that then goes through the ordinary list.

So the path exists and is proven, but you have to walk it yourself: load the gateway’s tools with the adapters and pass them in the list. Which, incidentally, fits well with what we already know about the catalogue: the credential-based filtering the gateway does is applied in the listing, so each agent receives the subset that belongs to it without the harness having to know anything.

The subagents

They are declared in three ways and invoked with a task tool that takes a description and a type. What you need to know to design with them:

The subagent receives the parent’s state except the messages and a few private keys, and it starts with a single human message which is the task description. Files are inherited, and it shares the same backend object, so parent and child work on the same logical filesystem. The result comes back as a tool message with the structured response or the child’s last text.

Depth is effectively one level. Verified by compiling the graph: the stack assembled for a declarative subagent does not include the subagent middleware, so the child does not have the task tool and cannot delegate in turn.

And there is no concurrency limit of its own. The parallelism is that of the engine’s parallel tool calls, and the task tool’s prompt explicitly invites launching them in parallel. Anyone with a per-tenant GPU quota will want to put the brake where it belongs, because it is not here.

Code execution

The sandbox protocol is well defined and the packaged implementations are nearly all paid services: four commercial providers, plus an embedded JavaScript interpreter and a sandbox from the platform vendor itself.

That leaves a local shell backend that is worth looking at closely before considering it. It runs the command with the system shell, and its own docstring lists what it does not do: no isolation, no process separation, no resource limits, and it lists production and multi-tenant environments as inappropriate use cases. It also has an option to inherit the process environment which, enabled, hands the model every variable, including the gateway credentials and the database connection string. It comes disabled, and it is an easy foot to shoot.

The reasonable way out on a cluster of your own is to implement the protocol yourself, and it is smaller than it looks. The sandbox base class requires four things: an identifier, execute, upload files and download files. Everything else, list, read, write, edit, delete and search, is built by the base class on top of execute. Looking at one of the commercial implementations, it is on the order of a hundred and fifty lines.

One image requirement that constrains the Dockerfile and is worth knowing beforehand: the base class helpers inject small encoded Python scripts to resolve searches and checks, so the sandbox image needs Python and a POSIX shell or half the calls fail.

On how to implement it, the shape that fits a Kubernetes of your own is an ephemeral container per session with a bounded time to live, resource limits, no service account token mounted and a network policy that only allows egress towards the inference gateway. Execute is resolved against the cluster’s exec API, and upload and download through the same channel. The alternative with less code is an isolated runtime class for the agent pod, one pod per tenant, and the shell backend inside; you lose multi-tenancy in a single pod and you have nothing to write.

Observability, and the package it goes through

Here there is a fact that surprises and that is worth being clear about before designing the trace pipeline.

LangChain’s core does not emit OpenTelemetry. Zero references in the whole package. Its native tracer talks to the vendor’s observability service and that is that.

The OTLP exporter does exist, but it lives inside that service’s SDK. And there is the good news: that SDK has a tracing mode that emits only over OTLP, without talking to the service, reading the destination and headers from OpenTelemetry’s standard environment variables. The transport is HTTP with protobuf, which is exactly what a self-hosted Langfuse accepts on its ingestion route.

That is: to take the agent’s traces to your own observability you have to install the vendor’s package and ask it not to talk to the vendor. It works, it is supported and it is documented, but it is a dependency you have to declare in the supply chain analysis, not a configuration detail. And if the OpenTelemetry packages are missing, it emits a warning and silently stops tracing, so start-up should check for it.

There is the alternative of the callback handler that Langfuse itself publishes, with fewer pieces. The practical difference is correlation: over the OTLP route the agent’s spans share a trace identifier with the gateway’s and the inference engine’s, and you see a whole request end to end. Over the callback route, you do not. For anyone who already has a collector deployed, the OTLP route is the one that pays.

The server: here is where the free part ends

Everything above is an architecture decision. This is a licence decision, and it is the one that gives the article its title.

The harness SDK is MIT. The graph engine is MIT. The memory, SQLite and Postgres checkpointers are MIT. All of that runs anywhere, with no key, no registration and without calling anybody.

The server that exposes a graph over HTTP is another matter. The package that implements it carries an Elastic 2.0 licence, whose relevant clauses forbid offering the software to third parties as a managed service and forbid modifying or circumventing the licence key functionality. The command-line tool’s own command prints the conditions: for local development it asks for a service API key, and for production use it asks for a licence key in an environment variable.

There is also usage reporting. The server code contains a beacon endpoint to which it sends metadata, with a header carrying the licence key. The self-hosting documentation confirms it from the other side, listing among the requirements network egress to that domain for licence verification and usage reporting if not running in isolated mode. Isolated mode is a contractual variant, not a box you tick.

And it drags infrastructure along: besides Postgres, it asks for Redis.

The development tool, which is the one that appears in every tutorial, describes itself as development mode with hot reload and an in-memory server. It is not a production server and it does not claim to be.

None of this is a reproach. It is a legitimate and fairly common business model: permissive core, commercial control plane. What is not legitimate is finding out late. Anyone building a platform with a sovereignty requirement, with no dependencies on external services and with the argument that the whole stack is open, has to know that the piece that turns the graph into a service does not meet that requirement.

What you have to build

The way out is to write the HTTP layer yourself, and the good news is that it is boring. A service that compiles the graph once with the checkpointer pointing at the database, and exposes an endpoint that validates the identity provider’s token, derives the thread identifier and the tenant context, and invokes the graph. Event streaming if needed.

What you additionally have to write, and which the commercial server gave you ready-made:

Per-thread authorisation. The engine does not have it. It is the most important and the easiest to forget, because nothing fails if it is missing.
Retention and deletion. A scheduled job that deletes whole threads, never loose rows, for the reasons in the state section.
Migrations. Schema setup has to be called explicitly and needs data-definition permissions, so it goes in a separate job with a different role from the runtime one. And since it creates indexes concurrently, it cannot go through a connection pooler in transaction mode; point it at the direct write service.
Health probes and draining. The engine has a cooperative draining mechanism, but wiring it to the container’s termination signal is your own work.
Run queue and cancellation, if you need long background runs. This is where the absence is most felt, and the reasonable answer is a queue on the same database rather than another system.
Durability decided deliberately. There are three modes: persist before the next step, persist in parallel, or persist only on exit. The last one is no good on Kubernetes: if the pod dies, the whole turn is lost.

On that last point it is worth being explicit, because it affects tool design. The engine stores each task’s writes as soon as it finishes, and on resume it reapplies those that were already there and does not repeat them. A tool that finished is not run again. A tool that was in flight when the pod died is, because its write never landed. The semantics are at-least-once, so every tool with an external effect has to be idempotent or carry an idempotency key. There is no exactly-once.

And the same applies to interrupts for human confirmation: on resume, the node is re-executed in full, so any side effect that sat before the interrupt inside the same node is repeated. The rule is that the interrupt goes first.

How it fits with activity logging

Two things about this setup touch compliance head on, and both come out of earlier sections.

The first is that the checkpoint is, de facto, a repository of personal data. It contains the whole conversation and, with the default backend, the files the agent has written. If the platform serves identifiable people, that database needs encryption, retention of its own and separate access control, exactly like the gateway’s spend table. The difference is that here there is no redaction switch that helps: the state is the state.

The second is traceability. An agent that delegates to subagents and executes tools produces a chain of actions that you have to be able to reconstruct, and the only place where that chain is complete is the trace. Traces going out over OTLP to your own observability stops being a technical preference and becomes the logging mechanism. It is worth designing it that way from the start, and not adding it afterwards.

As always in this series, the annex’s specific codes have to be checked against the current text before taking them into a compliance document; the approach is developed in the technical controls article.

Checklist

Decide the filesystem backend before the first demo, because the default one puts the content in the database.
Pass the model profile by hand with the real window, and check it at start-up.
Lower the eviction thresholds if the model’s window is below 64,000 tokens.
Build the model as an instance, never as a string with a provider prefix, if there is an OpenAI-compatible endpoint behind it.
Write per-thread authorisation before opening the service to more than one user.
One deployment per tenant if you use the filesystem backend; a namespace computed per request if you use the store one.
Compute that namespace from your own context, not from the server-information field, which is null outside the managed platform.
A scheduled retention job that deletes whole threads and only whole threads.
Schema migrations in a separate job, with a role of their own and against the direct write service.
Synchronous durability, and tools with external effects made idempotent.
Traces over OTLP with the mode that does not talk to the vendor’s service, and a start-up check that the packages are there.
Put a concurrency limit on subagents if there is a GPU quota involved.
If you need code execution, implement the sandbox protocol against an ephemeral container; the local shell backend is advised against by its own documentation.
Note in the supply chain analysis the hard dependencies on commercial providers that the package drags in even when unused.

Traps and things that are not what they look like

The planning tool was removed in 0.7.0. You have to add it by hand, and in each subagent too.
The default backend stores files inside the checkpoint, binaries included in base64.
Evicting a large result to a file does not get it out of the database with that backend, only out of the context window.
One checkpoint is written per superstep, not per turn.
Of the four declared cleanup operations, only delete-a-thread exists in Postgres.
A partial pruning can empty the history without raising any error.
The delta channel is in beta and its on-disk representation may change.
There is no owner check in the checkpointer. The thread identifier is not a credential.
The filesystem backend’s root is fixed when the agent is built and it separates no tenants.
The documentation’s example for the per-user namespace does not work outside the managed platform.
The filesystem middleware and the subagent middleware cannot be removed.
If the model does not declare its window, compaction does not trigger until 170,000 tokens, silently.
Passing the model as a string with a provider prefix forces an API that a vLLM does not implement.
The SDK does not integrate MCP; the integration lives in the command-line tool.
Subagents do not delegate: depth is one level.
LangChain’s core does not emit OpenTelemetry; the exporter lives in the vendor’s service SDK.
The HTTP server carries an Elastic 2.0 licence, asks for a licence key for production and reports usage unless there is an isolation agreement.
The development tool is for development, with an in-memory server.
Resumption is at-least-once for in-flight tools.

Closing

The question this article started with was whether deepagents fits into a platform of your own. The answer is that it does, and that it fits better than I expected, precisely because it is less than it appears. A middleware stack on top of a graph is easy to host, easy to understand and easy to replace. What does not fit is the piece next to it.

That asymmetry is the pattern I have been seeing all year across the ecosystem, and it deserves naming. The core is published under a permissive licence because the core is where you compete for adoption. The control plane is published under a restrictive licence because the control plane is where you charge. It happened with the gateways, it is happening with observability, and it happens here. Anyone building a sovereign platform cannot assess a whole project by the licence of its main repository: you have to look package by package, and look at which of them is the one you are going to need the day the thing has users.

The good part is that the bill, in this case, is paid in work and not in money. What you have to build to do without the commercial server is an HTTP service with no mystery to it, a retention job, a migration job and an authorisation layer. None of that is research. All of it is a week of an engineer who knows what they are doing, and in exchange you are left with a system that deploys the same way in a school as in a classified datacenter, with no internet egress and with nobody counting runs on the other side.

What I do recommend is making that list before the first demo and not after. Because the front-page example works in twenty lines, and the twenty lines have the default backend, the model without a profile and no authorisation at all.

Sources

langchain-ai/deepagents, código y documento de arquitectura: https://github.com/langchain-ai/deepagents.
langchain-ai/deepagents, registro de cambios de la rama del SDK (eliminación de la herramienta de planificación en la 0.7.0): https://github.com/langchain-ai/deepagents/blob/main/libs/deepagents/CHANGELOG.md.
langchain-ai/langgraph, motor, canales y checkpointers: https://github.com/langchain-ai/langgraph.
langgraph-checkpoint-postgres, esquema de tablas y operaciones de mantenimiento: https://github.com/langchain-ai/langgraph/tree/main/libs/checkpoint-postgres.
langgraph-checkpoint, clase base del checkpointer y aviso sobre poda con canales de deltas: https://github.com/langchain-ai/langgraph/tree/main/libs/checkpoint.
LangChain, despliegue de servidor autónomo (requisitos de licencia, Postgres, Redis y salida de red): https://docs.langchain.com/langsmith/deploy-standalone-server.
LangChain, trazado con OpenTelemetry: https://docs.langchain.com/langsmith/trace-with-opentelemetry.
Elastic License 2.0: https://www.elastic.co/licensing/elastic-license.
Langfuse, integración nativa OpenTelemetry (ruta de ingesta y autenticación): https://langfuse.com/integrations/native/opentelemetry.
Langfuse, integración con LangChain por retrollamadas: https://langfuse.com/integrations/frameworks/langchain.
langchain-ai/helm, chart del servidor: https://github.com/langchain-ai/helm.
Boletín Oficial del Estado, Real Decreto 311/2022, Esquema Nacional de Seguridad: https://www.boe.es/buscar/act.php?id=BOE-A-2022-7191.

Making the model prefer our tools: the priority that exists in no layer, the two primitives that do, and the `_meta` that crosses the gateway

Tue, 15 Sep 2026 03:30:00 +0200

A follow-up to LiteLLM’s MCP gateway, which treated the tool catalogue as cost and as surface. This one treats the next question, the one that arrives in the meeting afterwards: so then, how do I get the model to use mine. Verified against LiteLLM 1.102.0 and MCP specification 2026-07-28, with a test of our own run on 14 September 2026.

TL;DR

Tool priority does not exist as a concept in any layer of the stack. Not in the specification, not in the clients, not in the providers’ APIs. What exist are two distinct primitives: removing the alternative from the catalogue, and leaving it behind a search step. Everything else is persuasion, and the model ignores it when it suits.

The specification has no ranking field. The Tool object is name, title, icons, description, schemas, five annotations that are unreliable hints by their own declaration, and _meta. There is a trap I see repeated: MCP does define a numeric priority, but in another structure, the one that applies to content blocks, resources and prompts. It does not govern tool selection.

The only filtering the specification blesses is by credential. The current revision says the set of tools must not vary by connection, and that it may vary according to the authorisation presented. That is exactly what a gateway with per-key lists does.

The client is the one that decides, and only one of them has anything resembling priority. Claude Code lets you exempt a server from deferral with alwaysLoad, so that its tools load in full at start-up while the rest require a prior search. It is not called priority and it works as such.

We proved that the per-tool mark crosses the gateway. We set up an MCP server with one marked tool, served it through LiteLLM 1.102.0 and captured the wire: the _meta arrives whole and with the correct key. The chain is clean on purpose, with a comment in the code that says so.

And it is lost on exactly three routes, which are the gateway’s three catalogue-trimming modes. Proxy mode, tool search driven by key permissions, and the REST listing. In other words: the per-tool mark and LiteLLM’s catalogue trimming are incompatible. You have to choose one.

There is a library trap that makes the field disappear silently. The SDK’s Tool model declares the alias without allowing population by field name, so building it with the field name puts the dictionary in the extras and serialises it under the wrong key. There is no error. The field simply does not arrive.

The cheap lever is the description, not the name. Swapping descriptions shifts the selection distribution substantially; changing only the name has minimal and inconsistent effects. And the gateway’s overrides really do replace what is sent to the client.

Reordering the catalogue is a weak lever. The model is already attending to the correct tool eighty per cent of the time when it fails. Interventions on prompt order repair at most 23 % of the failures.

And the asymmetry you build is attack surface. Whoever controls the order controls the selection: with injection rates of 1.2 % you can take over the head of a tool ranking between 91 % and 97 % of the time.

You are here: the question that arrives after the third server

The previous article closed with a decision ladder for the catalogue: trim, rewrite, split by route, and use progressive disclosure when the catalogue grows. That ladder answers how much catalogue to show.

The question that arrives afterwards is a different one, and it is more uncomfortable. A real agent has native tools from the client hosting it, it has the in-house gateway with the house servers, and it has two or three third-party servers that somebody connected. All of them compete. And what you want is not to show less catalogue: you want that, faced with a query three tools could serve, yours wins. The audited one, the one with access control in place, the one that leaves a trace.

The answer everybody gives is to write a rule in the agent’s instruction file politely asking it to prefer the in-house server. There is even a rule published in a template marketplace that does exactly that. It is text in the prompt and the model follows it when it feels like it.

This article is what lies underneath.

The analogy: the counter with two trays

Let us go back to the switchboard operator from the previous article, the one who also hands out keys. Now the building has three key providers: the house cabinet, the maintenance contractor’s, and the cleaning service’s. All three have a key to the store room and all three open it.

The head of security wants the house one used, because it is the one with a log. And he discovers he can do three things, not one more.

He can take the other two off the counter. It always works and it annoys whoever needed them.

He can leave his own in the front tray and the other two in a drawer, so that to take one you have to ask. It works almost always and annoys nobody.

And he can change the label on his key so that it describes better when it is the right one. It works sometimes, it is the cheapest, and it is the only one that also protects against the contractor changing his label without warning.

What he cannot do is put a number from one to ten on each key and trust the operator to respect it. That number does not exist. The rest of the article is why it does not exist and what to do instead.

What the specification says: nothing

The current revision is 2026-07-28. Read against the source schema, the Tool object has name, title, icons, description, input schema, output schema, annotations and _meta. There is no field for priority, weight, rank, cost, group or tag.

The annotations are exactly five: title, and the read-only, destructive, idempotent and open-world hints. The schema itself warns that all of them are hints and that a client should not make tool-use decisions based on them when they come from untrusted servers.

The `priority` that exists and is no use

Here is the mistake I see repeated most. MCP does define a priority field, numeric between zero and one. It is in the annotations interface that applies to content blocks, to resources and to prompts, alongside the audience and the last-modified date.

It is not in Tool. It has no relation whatsoever to which tool the model picks. Anyone who finds it by searching the word in the schema and concludes that tool priority is standardised has the wrong structure.

The only precedence the specification defines for a tool is one of presentation: to display the name, first the title, then the annotations title, then the name.

The only blessed filtering is by credential

There is a new sentence in the current revision that is genuinely useful and that goes unnoticed. The set of tools must not vary by connection nor as a side effect of other requests, and it may vary according to the authorisation presented in the request.

Translated: segmenting the catalogue by key, by team or by token is the legitimate way to do it. Segmenting it by session state stopped being legitimate when the July revision removed protocol sessions.

Discovery does not discover tools

server/discover, which the current revision adds and which servers have to implement, does not return tools. It returns supported versions, capabilities, server instructions, identity and the cache freshness fields. It accepts neither filters nor a cursor.

Listing is still asking for everything and paginating. The listing request accepts only a cursor: there is no query, no filter, no limit, no groups, no tags.

And on collisions between servers, the specification explicitly and reasonedly washes its hands: clients or proxies that aggregate tools from several servers will encounter collisions and should implement a disambiguation strategy, for example prefixing. And it adds that the server name is not guaranteed to be unique and should not be used to disambiguate. There is no precedence rule between servers. There is not going to be one soon.

What there is in proposals

Nothing accepted. The official proposals index only lists those in a final state, and none of them deals with priority, groups or search. In draft there is one asking for a free-text query in the listing and a filtering capability; another on groups and tags that ended up superseded; and one carrying the word priority but which is per-tool model routing, which is another thing.

The interest group exploring the grouping of primitives declares in its own minutes that it is not going to pick a canonical pattern soon, and the documents in its repository are empty. The year’s roadmap does not mention the problem.

The layer that decides: the client

If the specification gives you nothing, what is left is whatever each client has built on its own. And here there is a clear winner.

Claude Code

Tool search is on by default, and what it does is defer: MCP tools are not loaded in full at start-up, the model sees just enough and discovers them when it searches. It requires a model that supports tool reference blocks, that is Claude Sonnet 4.5, Haiku 4.5, Opus 4.5 and later.

On top of that, the lever:

{
 "mcpServers": {
 "gateway-casa": {
 "type": "http",
 "url": "https://litellm.interna.svc/mcp",
 "alwaysLoad": true
 }
 }
}

The documentation describes it bluntly: if a server’s tools must always be visible without a search step, you set alwaysLoad to true and all of that server’s tools are loaded into context at the start of the session, regardless of the global search setting. And it recommends using it for a small number of tools that the model needs on every turn, because each tool loaded up front consumes context.

That is visibility asymmetry, and it works as priority even if it is not called that. Ours in front, the rest behind one step.

There is an operational consequence that the documentation mentions and that matters more in a gateway than in an ordinary server: setting alwaysLoad makes start-up wait for that server to return its tools, capped by the standard five-second connection timeout. A gateway that aggregates eight upstream servers and does not cache the listing, as is the case, lists live against all eight every time. If one of them is slow, the session’s start-up feels it.

There is also the fine-grained variant. An MCP server can mark individual tools as always loaded by including "anthropic/alwaysLoad": true in the tool’s _meta object, with the same effect for that tool alone. That would let you load three tools from our gateway up front and leave the other forty from the same gateway deferred. It is the lever that matters, and it is the one that motivated the test in the next section.

The rest of this client’s arsenal is pure exclusion, and it is worth knowing one detail that decides whether it works or not. A deny rule with the bare tool name removes it from the model’s context; with parentheses, it does not. The documentation puts it in a table: plain WebFetch removes the tool and the model cannot search at all, whereas WebFetch(domain:*) keeps the tool and rejects every request. The difference is not cosmetic: in the second case the model still sees the tool, still tries it and still spends the context its definition occupies.

The --tools flag is the scalpel: it accepts the empty string to disable all the built-ins, and it does not affect MCP tools. That is, --tools "" leaves the model with only what the gateway serves. It is a command-line flag and has no equivalent as a settings-file key; the functional equivalent there is deny rules.

The other clients

Client	Disable the native ones	Filter the MCP ones	Priority
Claude Code	`--tools ""`, deny by bare name	deny by pattern, subagents	`alwaysLoad`
Codex CLI	`features.shell_tool` to false	`enabled_tools` and `disabled_tools` per server	No
Gemini CLI	`tools.core` as an allowlist	`includeTools` and `excludeTools` per server	No
Copilot in VS Code	Not documented	picker and tool sets	No
Cursor	No	per-server switches	No

Two notes. Copilot cuts off at 128 tools per request and its virtualisation threshold does not go above that; its embedding-based routing is internal and has no configuration surface. And Cursor’s rules are purely indicative: they are text prepended to the context.

The API layer

If the client is in-house, the margin is wider, though narrower than it looks.

Forcing a set is not possible. In Anthropic’s case, tool choice accepts auto, any, a specific one by name, or none, and the documentation says explicitly that pointing at a set of MCP tools or at a member is not supported. In vLLM it is the same: a specific one, all of them or none. There is no way to say “any of mine from the gateway”. The proposal that would introduce grammars restricting tool names to the request’s set has been open for months and is not merged.

Filtering the catalogue is possible, in all three of the big ones. In Anthropic, with an MCP tool-set block where the default configuration disables and specific ones are enabled. In OpenAI, with the allowed tools list inside the MCP block itself, which the documentation justifies on latency and cost grounds, avoiding the model seeing unnecessary definitions. And in Gemini, the remote MCP server configuration accepts restricting which of the server’s tools the agent may call.

It is worth not confusing two things that are named almost identically in OpenAI: the allowed tools list inside the MCP block filters the catalogue, and that is what matters here; the identically named tool-choice type is another thing, it is a selection restriction, and there is evidence that it does not combine well with hosted tools, without the current documentation recording it. If anyone depends on that, let them test it against their model: it fails as a request error, so it shows up on a cold start.

And there is a 2026 lever that almost nobody is using. Anthropic has in beta a mechanism for mid-conversation tool changes, with add and remove blocks that accept referencing an entire MCP set. The reason it exists is in its documentation: the tools array sits even earlier in the chunked prefix than the system field, so editing it invalidates the cache for the whole conversation; by declaring the full set at the start and using the blocks, the array never changes and the cached prefix stays intact.

Applied to our case: you can remove a rival server’s set hot, mid-session, without paying for the whole prefix. It is not available on every model.

The test: the `_meta` crosses the gateway

The fine-grained variant of Claude Code’s lever depends on a question no documentation answers: whether a gateway that aggregates upstream servers propagates each tool’s _meta all the way to the client, or loses it along the way.

We checked.

The setup

A minimal MCP server with the Python SDK, serving two tools over HTTP, one of them with the mark in place:

types.Tool(
 **{
 "name": "always_loaded_tool",
 "description": "tool with _meta",
 "inputSchema": {"type": "object", "properties": {}},
 "_meta": {"anthropic/alwaysLoad": True, "probe": "UPSTREAM_META_MARKER"},
 }
)

In front of it, a LiteLLM from the main branch with that server declared in config.yaml, with no database, with a master key. And an MCP client querying the listing against /mcp.

The result

This is what comes out on the wire:

{
 "_meta": {"litellm.ai/server_outcomes": {"probe": {"status": "ok", "tool_count": 2}}},
 "tools": [
 {
 "name": "probe-always_loaded_tool",
 "description": "tool with _meta",
 "inputSchema": {"type": "object", "properties": {}},
 "_meta": {"anthropic/alwaysLoad": true, "probe": "UPSTREAM_META_MARKER"}
 }
 ]
}

The mark arrives whole and with the correct key. The client parses it without trouble.

Tracing the code, the chain is clean on purpose. LiteLLM keeps the SDK objects as they are when it lists against the upstream, instead of rebuilding them. The name prefixing does a deep copy and mutates only the name, with a comment in the code declaring the intention to preserve every field including the _meta by avoiding mutation. The two permission filters are list comprehensions that reuse the same objects. And the name and description overrides mutate in place, so they do not lose anything either. Somebody thought about this.

One operational detail: the name arrives prefixed with the server alias, governed by the configurable separator and the short prefix mode.

The three routes where it is lost

And here is the finding that changes the previous article’s recommendation.

Proxy mode returns only its three fixed tools, with no _meta and with no upstream tool at all. Verified against the wire as well.

Tool search activated by key permissions does the same with four virtual tools.

And the REST listing builds its response object by hand and discards the field: the tool that did carry a mark comes out with the field null. The class inherits from the type that has the field; it simply is not filled in. The fix fits on one line.

Those three routes are, exactly, the three ways LiteLLM has of trimming the catalogue. The operational conclusion is uncomfortable and it is worth saying plainly: either you use the gateway’s catalogue trimming, or you use the client’s per-tool mark. They cannot be combined. The previous article recommended proxy mode as the fifth rung of the ladder; with this in hand, that rung and the fine-grained mark are mutually exclusive.

The library trap

This deserves its own section because it takes out anyone writing a server or a proxy, and it gives no warning.

The SDK’s tool model declares the field with an alias, and does not enable populating it by name. The consequences, both checked by running them:

Building with the field name populates nothing. The dictionary slips into the model’s extras and comes out serialised under the wrong key, without the leading underscore, which is a key no client interprets.

And dumping and revalidating without asking for aliases loses the field, for the same reason.

There is no exception, no warning in the log, nothing. The field disappears. If someone marks their tools and does not see them marked in the client, this is the first place to look, before the gateway.

What we have not tested

For honesty’s sake, and because it is the link that remains: we have shown that the mark survives the gateway. We have not shown that the client acts on it when the tool arrives with the name prefixed by the gateway. Claude Code’s documentation describes the per-tool mark in a single sentence and says nothing about aggregating servers or about prefixes.

It is a half-hour test for anyone who has the setup in front of them: two tools from the same gateway, one marked, and look at which one appears loaded at the start of the turn. If someone does it before I do, I am interested in the result.

What LiteLLM can and cannot do

With the above, the gateway’s inventory of levers comes out like this.

Lever	Serves external MCP clients	Preserves the `_meta`
`allowed_tools` and `disallowed_tools` per server	Yes	Yes
Name and description overrides	Yes	Yes
Tool sets per route	Yes	Yes
Proxy mode and virtual tools	Yes	No
Search by key permissions	Yes	No
REST listing	Yes	No
Semantic filter	No	Not applicable

Two clarifications on the previous article’s table, now that the code has been read again.

The allowed and disallowed lists are applied in the listing, not only on the call. They are the blunt, effective instrument, and they do not break the cache because they are static.

And the overrides have a limit that was not accounted for: they are applied in the MCP protocol listing, but they are skipped in proxy mode and not applied on the REST route. Anyone rewriting descriptions and also turning on proxy mode is not serving what they think.

What the gateway cannot do is inject _meta. There is no setting equivalent to the ones for name and description. The mark has to be put there by the origin server, which for in-house servers is trivial because we write them. To mark third-party tools you would have to patch, and it is small: one field in the type and three lines in the overrides function if configuring it by file is enough.

There is also an ordering lever that did not appear in the previous article. In the proxy’s search mode there is a core tools setting that puts them at the front of the ranking and that also does not count against the results cap. It is the closest thing to a declarative priority in the whole stack, and it lives inside the one mode that discards the _meta.

And a nuance about proxy mode’s results cap. The client cannot ask for more than five, that is true, but the operator can raise it by configuration. The previous article implied it was immovable.

The cheap lever: the description, not the name

If you cannot exclude, what is left is to bias. And bias comes in through the description.

The work that measures selection bias between functionally equivalent tools puts the combined bias of the evaluated models between 0.25 and 0.38, that is, you would have to redistribute between 25 % and 38 % of the probability mass for equivalent tools to be picked equally. On that basis, swapping two tools’ descriptions shifts selection substantially, whereas perturbations that touch only the name produce smaller and more inconsistent effects.

Another piece of work measures that a single pass of automatic description rewriting improves the metric over a large corpus, almost as much as iterative refinement, and with two orders of magnitude less time. And it adds a warning worth retaining: descriptions tuned against a fixed set of candidates do not generalise to the dynamically retrieved set. You have to optimise for the regime you serve in.

Applied: the gateway’s description override is at once token trimming, selection bias and mitigation of description change after approval. It is the lever with the best effort-to-effect ratio in the whole article, and it is static, so it does not touch the prompt cache.

Name prefixing, by contrast, costs tokens and probably does not change which tool the model goes to.

What does not work

Reordering the catalogue. A June paper measures that, when the model fails, it was already attending to the correct tool eighty per cent of the time, well above chance. The bottleneck is not at the input but in the late layers. Interventions on prompt order recover at most 23 % of the failures, against the 59 % to 91 % of those acting on the final readout. Reordering is cheap and that is why it gets recommended a lot; it is also weak.

Rules and instructions. They are text. They help and they do not decide.

Your own per-request top-K filters. This is covered in the previous article and it still holds: four out of five measured strategies perform worse than not filtering, and on top of that rewriting the tool block on every turn invalidates the entire cached prefix.

The ladder, updated

Exclude in the client. It is the only deterministic thing. --tools "" or deny by bare name for the competing native tools.
Mark your own as always loaded. At server level with alwaysLoad, or per tool with the mark in the _meta if you control the origin server. Counting on start-up waiting for the server.
Trim with static lists per server and per key, and split a catalogue by agent profile across routes.
Rewrite the descriptions with the gateway’s overrides.
Remove hot the rival sets, if the client is in-house and the model supports it.
Choose: the gateway’s catalogue trimming, or the per-tool mark. Not both.
Do not: reorder, rename, or trust prompt rules.

The risk that has to be declared

The asymmetry you build is also a surface. Whoever controls which tools head a ranking controls the selection, and that can be attacked: there is work measuring that by injecting adversarial tools at rates of 1.2 % you take over the head of the ranking between 91 % and 97 % of the time.

In a system under ENS (Esquema Nacional de Seguridad, Spain’s national security framework) this fits under change management, not access control: the set of tools a model sees and the order in which it sees them are security configuration, and today nobody versions them. Neither does the specification fix a hash of the description or of the schema, nor does the gateway. The specific Annex II codes are worth checking against the current text before taking them into a compliance document; the approach is developed in the MCP gateway article.

There is a precedent within the ecosystem itself that points the way. The MCP skills extension, which is in a final state, requires a manifest with a per-file hash, mandatory verification before use, and approval tied to the set of files and their digests, so that any change revokes the approval. And it obliges hosts to prevent two skills with the same name from silently replacing one another. It is exactly the pattern missing for tools. Anyone who needs it today has to implement it in their gateway.

Checklist

Decide the strategy before touching anything: catalogue trimming in the gateway, or the per-tool mark in the client. They are mutually exclusive.
If you pick the mark, put it in the origin server, because the gateway cannot inject it.
Build the tool object with the alias key, never with the field name, or the field disappears without warning.
Do not revalidate tool objects from a dump without asking for aliases.
Count on marking a server as always loaded making session start-up wait for that server, and on a gateway listing live against all of its upstreams.
Exclude in the client the native tools that compete, by bare name and not with a pattern in parentheses.
Rewrite the descriptions of your own tools, and do not waste time renaming.
Do not turn on proxy mode if you depend on the description overrides, because they are not applied there.
Version the exposed tool set and its order as security configuration, with a hash of the description and the schema.
If you use the gateway’s REST route for anything, know that it discards the _meta.

Traps and things that are not what they look like

MCP’s priority exists, and it is not about tools. It lives in the annotations for content, resources and prompts.
Tool annotations are hints and the schema itself says they are not reliable from untrusted servers.
The specification gives no precedence rule between servers, and it says besides that the server name is no use for disambiguating.
The tool set may indeed vary by authorisation, and that is the only blessed segmentation.
server/discover does not return tools.
A deny rule with parentheses does not remove the tool from the context, it only rejects the calls.
--tools is a command-line flag, with no equivalent in the settings file.
Marking a server as always loaded delays start-up by up to five seconds per server.
Tool(meta=...) does not populate the field and serialises it under the wrong key.
Proxy mode, search by permissions and the REST listing discard the _meta.
Description overrides are skipped in proxy mode and are not applied on REST.
Proxy mode’s cap of five results can be raised by the operator, even though the client cannot ask for more.
The core tools of the proxy’s ranking are the only declarative priority in the stack, and they live in the mode that loses the _meta.
You cannot force “any of mine from the gateway” in any API.
In OpenAI, the allowed list inside the MCP block and the identically named tool-choice type are different things.
Gemini does accept a per-tool allowlist on its remote MCP servers.
Reordering the catalogue repairs at most 23 % of the failures.

Closing

The starting question had a trap in it, and the trap is the word. When someone asks how you prioritise an MCP server, they are assuming a dial exists somewhere. It does not, and I have spent the whole article showing the places where it is not.

What there is instead is poorer and more manageable: you remove what competes, you leave your own in front, and you write the description better. Three things, none of them elegant, all three effective in that order.

What does seem to me worth taking away is the shape of the problem. The tool catalogue a model sees is security configuration in the full sense, because it determines what the agent can do and which system it is going to talk to. And today it is managed like a list of connections: things get added, it is not versioned, it is not signed, and nobody detects a change. The ecosystem has already solved that problem once, for skills, with hashes and tied approval. For tools, not yet.

In the meantime, the asymmetry has to be built by hand, server by server, and knowing that it rests on the _meta of an object that a library can empty without saying anything.

Sources

Model Context Protocol, herramientas en la revisión 2026-07-28: https://modelcontextprotocol.io/specification/2026-07-28/server/tools.
Model Context Protocol, server/discover: https://modelcontextprotocol.io/specification/2026-07-28/server/discover.
Model Context Protocol, paginación: https://modelcontextprotocol.io/specification/2026-07-28/server/utilities/pagination.
Model Context Protocol, caché de listados: https://modelcontextprotocol.io/specification/2026-07-28/server/utilities/caching.
Model Context Protocol, esquema fuente de la revisión: https://github.com/modelcontextprotocol/modelcontextprotocol/blob/main/schema/2026-07-28/schema.ts.
Model Context Protocol, extensión de habilidades: https://modelcontextprotocol.io/extensions/skills/overview.
Model Context Protocol, grupo de interés sobre agrupación de primitivas: https://modelcontextprotocol.io/community/interest-groups/primitive-grouping.
Anthropic, Claude Code y MCP (alwaysLoad, búsqueda de herramientas): https://code.claude.com/docs/en/mcp.
Anthropic, Claude Code, referencia de línea de comandos (--tools): https://code.claude.com/docs/en/cli-reference.
Anthropic, Claude Code, permisos (denegación por nombre pelado): https://code.claude.com/docs/en/permissions.
Anthropic, conector MCP y conjuntos de herramientas: https://platform.claude.com/docs/en/agents-and-tools/mcp-connector.
Anthropic, cambios de herramientas a mitad de conversación: https://platform.claude.com/docs/en/build-with-claude/mid-conversation-system-messages.
Anthropic, búsqueda de herramientas: https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool.
OpenAI, MCP y conectores (lista de herramientas permitidas): https://developers.openai.com/api/docs/guides/tools-connectors-mcp.
OpenAI, búsqueda de herramientas y caché: https://developers.openai.com/api/docs/guides/tools-tool-search.
Google, llamada a funciones en la Interactions API: https://ai.google.dev/gemini-api/docs/function-calling.
vLLM, llamada a herramientas y decodificación restringida: https://docs.vllm.ai/en/latest/features/tool_calling.html.
vLLM, propuesta de decodificación guiada por regiones y gramáticas de herramienta: https://github.com/vllm-project/vllm/issues/39848.
LiteLLM, código del gateway MCP: litellm/proxy/_experimental/mcp_server/server.py, mcp_server_manager.py, rest_endpoints.py, tool_search.py, litellm/experimental_mcp_client/tools.py: https://github.com/BerriAI/litellm.
SDK de Python de MCP, modelo Tool y serialización por alias: https://github.com/modelcontextprotocol/python-sdk.
BiasBusters, sesgo de selección entre herramientas equivalentes y peso de la descripción frente al nombre: https://arxiv.org/html/2510.00307v2.
Looking Is Not Picking, atención frente a lectura final y techo de las intervenciones sobre el prompt: https://arxiv.org/html/2606.16364v1.
A Single Rewrite Suffices, reescritura de descripciones y generalización al conjunto recuperado: https://arxiv.org/html/2606.30775.
ToolFlood, saturación adversaria de la capa de recuperación: https://arxiv.org/html/2603.13950.

Langfuse v4, day 2 (5 of 10): the worker's thirty-nine queues, and the three that decide whether you lose data

Mon, 14 Sep 2026 08:30:00 +0200

Fifth article in a series of ten about operating Langfuse v4 in production. The previous one covered migrating from version 3. This one goes inside the worker that is already running. Verified against the Langfuse 4.35.0 source, commit 2ee1908 of 14 September 2026.

TL;DR

Thirty-nine queues and thirty-two switches. The catalogue is in packages/shared/src/server/queues.ts:406. The QUEUE_CONSUMER_*_IS_ENABLED variables are read once at boot, so turning a queue off is a restart, not a hot change.

Sharding spreads nothing without Redis Cluster. The producer only applies the hash if REDIS_CLUSTER_ENABLED === "true", and that variable ships "false". Raising LANGFUSE_INGESTION_QUEUE_SHARD_COUNT on an installation with single-node Redis starts N workers and leaves N−1 with no work.

The dead-letter retrier ships disabled. QUEUE_CONSUMER_DEAD_LETTER_RETRY_QUEUE_IS_ENABLED is the only one of the thirty-two whose default is "false".

And even switched on, it does not cover ingestion. Its list holds five queues and none is ingestion-queue. An event that exhausts its six attempts sits in the failed set.

The ClickHouse writer drops rows. Three attempts with a flat hundred-millisecond delay and then out. The code comment is explicit: // TODO - Add to a dead letter queue in Redis rather than dropping. The metric to watch is called langfuse.queue.clickhouse_writer.rows_dropped.

Turn a queue’s consumer off and you stop seeing its depth. The metrics emitter only polls queues with a worker registered in that container. The queue keeps growing and the dashboard goes flat.

Project deletion runs one job every ten minutes. By limiter, deliberately, because each one scours ClickHouse.

There are two optional health probes and one of them can cause the very restart loop it is meant to fix if configured with less than sixty seconds of initial delay.

You are here: OBSERVE, day 2

The previous article left the installation migrated. This is the map of the process that does the work, and it comes before the capacity article because the queues are where saturation shows up before ClickHouse notices.

The analogy: the sorting hub

A postal sorting hub has separate belts for parcels, registered mail, returns and internal pouches. Each belt has its speed and its staff, and some are throttled on purpose because the destination cannot take more.

The job of whoever runs it is not to watch the fastest belt, it is to know which one jams first, which can stop for an afternoon without anyone noticing and which cannot stop for ten minutes. And above all, where the bin for items nobody could sort is, because every hub has one and in many of them nobody looks at it.

The Langfuse worker has thirty-nine belts. This article is the plan, with the speeds and the location of the bin.

The map of the thirty-nine

Grouped by what they do, with what stops happening if the group stops:

Ingestion (4). ingestion-queue, secondary-ingestion-queue, otel-ingestion-queue, secondary-otel-ingestion-queue. If they stop, the web keeps accepting events and uploading them to object storage, and keeps enqueueing. Nothing reaches ClickHouse. The SDK sees no error at all. This is the group that cannot stop.

Propagation (1). event-propagation-queue, the dual-write one from the previous article, with a global concurrency of one across the cluster.

Evaluation (8). trace-upsert, dataset-run-item-upsert-queue, create-eval-queue, evaluation-execution-queue and its secondary, llm-as-a-judge-execution-queue, code-eval-execution-queue, experiment-create-queue. If they stop, evaluations are neither created nor executed. Traces keep arriving. This is the group that can wait.

Deletion and retention (7). trace-delete, score-delete, dataset-delete-queue, project-delete, data-retention-queue and its processing queue, batch-action-queue. If they stop, neither UI-requested deletions nor per-project retention run. That is not an operational annoyance, it is a breach if you have a retention commitment in writing.

Exports (3). batch-export-queue, core-data-s3-export-queue, metering-data-postgres-export-queue. Downloads a user asked for stay pending.

Outbound integrations (8). The PostHog, Mixpanel and blob storage pairs, plus webhook-queue, entity-change-queue and notification-queue. In each pair, the bare queue is the scheduler that fans out per project and the processing one does the work.

Cloud billing (4). They only register with a Stripe key. On a self-hosted installation they never start.

Maintenance (4). dead-letter-retry-queue, monitor-queue, in-app-agent-run-queue and v4-legacy-api-usage-queue, which ingests nothing: it is a cron that scans ClickHouse’s system.query_log to find which projects still use the old API.

The switches number thirty-two, all QUEUE_CONSUMER_*_IS_ENABLED and all "true" except the dead-letter retrier. Four queues have no switch of their own: the model-as-judge one rides inside the evaluation switch, the two Cloud data export queues have their own variable, and the processing queues share a switch with their scheduler.

The mechanism is a top-level if around each worker registration. A false switch means the worker is never instantiated, not that the queue stops receiving work: producers keep enqueueing regardless. Turning a queue off is not consuming it, not not filling it.

The sharding that does not shard

Nine queues support sharding: the four ingestion ones, three evaluation ones, model-as-judge and trace-upsert. Each has its own shard-count variable, and they all default to 1.

Raising them looks like the obvious first move to scale ingestion. It is not, because of this (packages/shared/src/server/redis/ingestionQueue.ts:48):

 const shardIndex =
 IngestionQueue.getShardIndexFromShardName(shardName) ??
 (env.REDIS_CLUSTER_ENABLED === "true" && shardingKey
 ? getShardIndex(shardingKey, env.LANGFUSE_INGESTION_QUEUE_SHARD_COUNT)
 : 0);

The : 0 at the end is the whole story. The producer only computes a shard if Redis is in cluster mode, and REDIS_CLUSTER_ENABLED ships "false". On an installation with single-node Redis, which is most self-hosted installations, raising the shard count to four starts four workers, three of which stare at empty queues, while all traffic keeps coming through ingestion-queue.

The sharding key, when it does apply, is projectId-eventBodyId run through SHA-256, taking the first eight hex characters and a modulo. Shard names are ingestion-queue for zero and ingestion-queue-N for the rest.

Without Redis Cluster, the lever that does work is concurrency, which is a different thing.

Concurrencies, and the ones set to one on purpose

The defaults, worth knowing because several sit at one for good reasons:

Queue	Variable	Default
`ingestion-queue` (per shard)	`LANGFUSE_INGESTION_QUEUE_PROCESSING_CONCURRENCY`	20
`secondary-ingestion-queue`	`LANGFUSE_INGESTION_SECONDARY_QUEUE_PROCESSING_CONCURRENCY`	5
`otel-ingestion-queue`	`LANGFUSE_OTEL_INGESTION_QUEUE_PROCESSING_CONCURRENCY`	5
`trace-upsert`	`LANGFUSE_TRACE_UPSERT_WORKER_CONCURRENCY`	25
`monitor-queue`	`LANGFUSE_MONITOR_QUEUE_PROCESSING_CONCURRENCY`	10
evaluation and model-as-judge	several	5
`create-eval-queue`	`LANGFUSE_EVAL_CREATOR_WORKER_CONCURRENCY`	2
`trace-delete`, `score-delete`, `dataset-delete-queue`, `project-delete`	several	1
`event-propagation-queue`	fixed in code	1 cluster-wide

Deletions sit at one and carry a rate limiter on top. The project one is the most aggressive (worker/src/app.ts:234):

 limiter: {
 // Process at most `max` delete jobs per LANGFUSE_CLICKHOUSE_PROJECT_DELETION_CONCURRENCY_DURATION_MS (default 10 min)
 max: env.LANGFUSE_PROJECT_DELETE_CONCURRENCY,

One project deletion every ten minutes. If a customer asks you to delete forty projects, that is six hours and forty minutes on the clock, and rightly so, because each of those deletions is a broom sweeping through ClickHouse.

The ingestion queues, by contrast, have no limiter at all. There the brake is concurrency and the ClickHouse writer.

One BullMQ detail that recurs across queues and explains a good number of other people’s stalls (worker/src/app.ts:261):

 // The default lockDuration is 30s and the lockRenewTime 1/2 of that.
 // We set it to 60s to reduce the number of lock renewals and also be less sensitive to high CPU wait times.
 lockDuration: 60000, // 60 seconds
 stalledInterval: 120000, // 120 seconds
 maxStalledCount: 3,

An event’s path, from API to ClickHouse

It is worth following it end to end once, because there are three places it can be lost.

The web groups the batch’s events by body id, uploads one JSON file per group to object storage and aborts if that upload fails: no blob, no enqueue. Then it enqueues one job per body id on ingestion-queue.

The worker picks up the job, optionally checks a recently-processed cache in Redis (off by default, five-minute lifetime), decides whether to redirect the project to the secondary queue, and downloads the file from storage. There is a comment here worth a capacity chart on its own:

Process files in batches. If a user has 5k events, this will likely take 100 seconds.

Then it merges and writes. And writing is not writing: the service does not talk to ClickHouse, it puts the row in a buffer whose defaults are 1,000 rows, 1,000 milliseconds and 3 attempts.

That buffer is the third place things get lost, and the most serious one in the system (worker/src/services/ClickhouseWriter/index.ts:535):

 // Re-add the records to the queue with incremented attempts
 let droppedCount = 0;
 queueItems.forEach((item) => {
 if (item.attempts < this.maxAttempts) {
 entityQueue.push({ ...item, attempts: item.attempts + 1 });
 } else {
 // TODO - Add to a dead letter queue in Redis rather than dropping
 recordIncrement("langfuse.queue.clickhouse_writer.error");
 droppedCount++;
 }
 });

Once the three attempts are spent, the row is dropped. And no queue retry saves it, because the BullMQ job already completed successfully: as far as the queue is concerned, that event was processed. The internal retries are not properly exponential either: the delay is flat at a hundred milliseconds, so all three attempts happen within three hundred milliseconds and a ClickHouse that is unwell for half a second takes the batch with it.

The runbook consequence is direct. langfuse.queue.clickhouse_writer.rows_dropped has to be on a dashboard and it has to have an alert at zero. Any non-zero value is silent data loss.

Retries, and the bin nobody looks at

Each queue defines its own policy. The ones that matter:

Queue	Attempts	Backoff
`ingestion-queue`	6	exponential, 5 s
`secondary-ingestion-queue`	5	exponential, 5 s
`evaluation-execution-queue`	10	exponential, 1 s
`code-eval-execution-queue`	3	exponential, 30 s
`trace-upsert`	2	exponential, 5 s, with a 30 s initial delay
`trace-delete`, `score-delete`	2	exponential, 30 s
`monitor-queue`, `in-app-agent-run-queue`	1	no backoff, deliberately

The two single-attempt ones are a design decision, and the comment explains it well: “Postgres owns correctness (claim CAS + reconcile-on-read); BullMQ is delivery-only, so never redeliver on its own.”

And now the bin. There is no dedicated dead letter queue. What exists is BullMQ’s native failed set per queue, with its own retention, plus a queue that retries from there. That queue has two problems for whoever operates it:

First, it ships disabled. It is the only one of the thirty-two defaulting to "false".

Second, even switched on, its scope is hand-written and has five entries (worker/src/services/dlq/dlqRetryService.ts:9):

 private static retryQueues = [
 QueueName.ProjectDelete,
 QueueName.TraceDelete,
 QueueName.ScoreDelete,
 QueueName.BatchActionQueue,
 QueueName.DataRetentionProcessingQueue,
 ] as const;

ingestion-queue is not there. Ingestion events that exhaust their six attempts sit in the failed set, up to a hundred thousand of them, and nobody ever retries them. To get them out there is an admin route, POST /api/admin/bullmq with action retry, which the file itself describes as the one the managed service uses for exactly this.

The metrics, with their exact names

Each queue’s metric name is built at runtime from the queue name, turning dashes into underscores and stripping the _queue suffix. So ingestion-queue produces langfuse.queue.ingestion, and secondary-ingestion-queue produces langfuse.queue.secondary_ingestion. Sharded queues always add a shard tag.

What comes out of that:

langfuse.queue.<queue>.rate with type in request, completed, failed, error and stalled.
langfuse.queue.<queue>.time_distribution with type in wait and processing.
langfuse.queue.<queue>.depth with type in waiting, failed and active, plus an aggregate series with shard: "all".
langfuse.queue.<queue>.dlq_oldest_age, the age of the oldest failed job, which emits zero when the set empties so the dashboard resets.
From the writer: langfuse.queue.clickhouse_writer.rows_dropped, .error, .processing_time and ingestion_clickhouse_insert_queue_length.

Two things to know about those metrics. The waiting depth includes paused jobs, to match BullMQ’s counter. And the big trap (worker/src/features/queue-metrics-runner/index.ts:70):

 // Only poll queues that have registered workers. This avoids calling
 // getInstance() on queues this worker doesn't consume, which would
 // create unnecessary Redis connections and can trigger side effects

Only queues that container consumes emit metrics. If you split roles and turn a queue’s consumer off across every container, that queue stops appearing on dashboards while it keeps growing in Redis. It is the perfect way not to find out.

The metric that saturates first is still the same one: langfuse.queue.ingestion.depth with type: "waiting". The emission interval defaults to one second.

The two probes, and the one that can restart you in a loop

The worker exposes /api/health and /api/ready, identical but for one thing: the second returns 500 after SIGTERM, which is how draining works. Both always check Postgres with a SELECT 1 and Redis with a ping capped at two seconds, hardcoded.

Then there are two optional checks enabled by query parameter:

?failIfEventPropagationStuck=true looks at the propagator’s heartbeat and returns 503 if it has not been refreshed for more than 35 minutes. A missing heartbeat does not count as stuck, deliberately, so a freshly booted container is not restarted in a loop. And the condition that must be respected:

Probes using this flag MUST set initialDelaySeconds >= 60s (one cron cycle) […] A shorter delay can crash-loop the very restart this check triggers.

?failIfQueueConsumptionStuck=true is an in-memory signal of that container, with no Redis involved: it is marked on the active and completed events of any worker. It returns 503 if that container has gone 60 minutes without picking up or finishing a single job. It exists for a very specific failure mode:

After Redis lock loss BullMQ workers can wedge permanently: the process stays alive and connectivity checks pass, but no queue picks up jobs ever again.

The one-hour threshold leans on the default crons keeping a healthy worker busy at least once an hour. In multi-replica deployments each scheduler tick lands on a single replica, so the comment itself warns about raising the threshold if a replica can legitimately sit idle.

Draining, by the way, has no cap. The order is close the HTTP server, stop the periodic runners, close the BullMQ workers and then flush the ClickHouse writer, which on shutdown empties the whole queue rather than one batch. The only thing bounding that drain is the Kubernetes grace period, so it had better be generous.

The crons

Queue	Pattern	What it does
`event-propagation-queue`	`* * * * *`	propagates one staging partition
`dead-letter-retry-queue`	`0 /10 * * *`	retries failed jobs from five queues
`v4-legacy-api-usage-queue`	`/15 * * *`	scans `system.query_log` for old-API usage
`blobstorage-integration-queue`	`/20 * * *`	fans exports out per project
`cloud-usage-metering-queue`	`5 * * * *`	Cloud billing
`posthog-integration-queue`, `mixpanel-integration-queue`	`30 * * * *`	hourly fan-out
`metering-data-postgres-export-queue`	`30 2 * * *`	daily export
`core-data-s3-export-queue`, `data-retention-queue`	`15 3 * * *`	daily export and retention

And a warning the scheduler file itself writes in capitals: cron registration is fire-and-forget from the queue constructors, and a failure is only logged. A transient Redis failure at boot can leave a cron unscheduled until the next restart, without the container ever being marked unhealthy.

Splitting roles, which is what all this is for

With thirty-two switches, what you can build is one worker per family. The split that makes sense on an installation with agentic load:

# Pool 1: ingestion. This is the one that scales with traffic.
QUEUE_CONSUMER_INGESTION_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_OTEL_INGESTION_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_EVENT_PROPAGATION_QUEUE_IS_ENABLED=true # note: global concurrency 1
LANGFUSE_INGESTION_QUEUE_PROCESSING_CONCURRENCY=20
# everything else false

# Pool 2: evaluations. Scales with the number of evaluators, not with traffic.
QUEUE_CONSUMER_EVAL_EXECUTION_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_CREATE_EVAL_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_TRACE_UPSERT_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_EXPERIMENT_CREATE_QUEUE_IS_ENABLED=true

# Pool 3: maintenance. One replica is enough.
QUEUE_CONSUMER_TRACE_DELETE_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_PROJECT_DELETE_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_DATA_RETENTION_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_BATCH_EXPORT_QUEUE_IS_ENABLED=true
QUEUE_CONSUMER_DEAD_LETTER_RETRY_QUEUE_IS_ENABLED=true # switch it on

Two warnings about this split. Each worker registration opens its own Redis connection, so the connection count grows with queues times shards times replicas, and that is checked against Redis maxclients before multiplying replicas. And the moment you split roles you lose the metrics of the queues that container does not consume, so some replica needs to consume every queue or you go blind on that one.

Checklist

There is an alert on langfuse.queue.clickhouse_writer.rows_dropped, threshold zero.
There is an alert on the depth of langfuse.queue.ingestion with type: "waiting".
There is an alert on dlq_oldest_age for the queues that matter, and somebody knows how to use POST /api/admin/bullmq.
The dead-letter retrier is on in some replica, knowing it does not cover ingestion.
Nobody raised the shard count without Redis Cluster.
The optional health probes are configured with initialDelaySeconds of at least 60 seconds.
The Kubernetes termination grace period allows the ClickHouse writer to flush.
Every queue is consumed by at least one replica, so its metrics are not lost.
Redis maxclients copes with queues times shards times replicas.

Traps

Sharding without Redis Cluster does not spread load. It starts the workers and leaves them idle.

The dead-letter retrier ships disabled and does not cover ingestion. Two independent facts, both uncomfortable.

The ClickHouse writer drops rows after three attempts. With a TODO in the code admitting it and three hundred milliseconds of total headroom.

Turning a consumer off also turns its metrics off. The queue keeps growing, the dashboard stays flat.

A cron can end up unscheduled while the container stays healthy. A transient Redis failure at boot, and it is gone until the next restart.

The PostHog and blob storage integrations disable themselves when an error is classified as a customer-config fault, and if the notification also fails they stay off and silent.

A misconfigured propagation probe restarts in a loop. Less than sixty seconds of initial delay and you get the cycle.

Draining has no cap of its own. The Kubernetes grace period bounds it and nothing else.

The series: the ten articles

What goes into a trace: version 4’s data model, limits, precedences, scores, masking and indexes.
Putting LangGraph in front: instrumenting an agentic platform and the measured cost of one turn.
Two well-known agents instrumented: Open Deep Research and GPT Researcher, with the measured figures of a real request.
Migrating from version 3 to 4 with no window: the three write-mode steps, the resumable background migrations and where the rollback point of no return sits.
The worker queues (this article): the map of all thirty-nine, which pool to dedicate to each group, the per-queue switches, sharding and concurrency.
Real ClickHouse capacity and cost: how to measure bytes per observation with the system tables, the difference between the full table and the listings table, and the merge cost of full-text indexes.
Retention, deletion and data protection: why a deletion does not free disk, the mask cleaner that ships disabled, the pending deletion queue and the S3 lifecycle that has to be implemented by hand.
Backup and cross recovery: restore ordering across Postgres, ClickHouse and object storage, what each mismatch breaks, and how far event replay goes.
Saturation runbook: what to alert on from the queue metrics, the stuck probes, draining through the readiness endpoint and the dead letter queue.
Getting the data out: the blob storage integration to Parquet, batch exports and the metrics API, to build the data lake.

Sources

Langfuse 4.35.0 source, commit 2ee1908 of 14 September 2026: packages/shared/src/server/queues.ts, packages/shared/src/server/redis/, worker/src/env.ts, worker/src/app.ts, worker/src/queues/, worker/src/services/ClickhouseWriter/, worker/src/services/dlq/, worker/src/features/health/, worker/src/features/queue-metrics-runner/, worker/src/utils/shutdown.ts.
Langfuse self-hosting documentation, checked on 14 September 2026.
BullMQ: going to production, checked on 14 September 2026.

Langfuse v4, day 2 (4 of 10): migrating from 3 to 4 with no window, and where the point of no return sits

Mon, 14 Sep 2026 08:00:00 +0200

Fourth article in a series of ten about operating Langfuse v4 in production. The previous three looked at the client side: what goes into a trace, what an agent turn costs and two well-known agents instrumented. This is the first one on the server side. Verified against the Langfuse 4.35.0 source, commit 2ee1908 of 14 September 2026.

TL;DR

The default write mode is the destination, not the origin. LANGFUSE_MIGRATION_V4_WRITE_MODE ships as events_only (packages/shared/src/env.ts:337). That is right for a new installation and wrong for a migration: in that mode the traces and observations tables stop being written.

The worker refuses to start on three combinations. legacy with direct OTel writes, events_only without the preview opt-in, and events_only with OTel behaviour set to dual. The code comment names them: combinations that would silently lose data (worker/src/env.ts:722).

In events_only twenty-four REST routes return 404. Traces, observations, sessions, v1 and v2 scores, metrics, spans, generations and the dataset-run ones. Not a bug: a deliberate short circuit so stale data from tables nobody writes is never served.

History is backfilled by a chain of five background migrations, not by the ClickHouse migrations. Not one of the ten new ClickHouse migrations copies a single old row.

A failed background migration does not block the next one. The manager has no dependency model and skips them. The chain protects itself with predecessor guards that the first two steps do not declare, so a failure of step 1 does not stop step 2 from starting.

The old API’s ten-minute delay has an exact explanation. Three-minute partitions in the staging table, a lock four minutes after creation, and a per-minute cron that only touches partitions older than ten minutes, with a global concurrency of one.

That staging table has a forty-eight-hour TTL. If the propagator stops over a long weekend, those partitions disappear and the hole in events_full is permanent. No code detects it and none repairs it.

The point of no return is in Postgres, not in ClickHouse. A Prisma migration runs DROP TABLE on traces, observations and scores, and Prisma Migrate has no down migrations.

You are here: OBSERVE, day 2

The first three articles measured what an agent produces. This one starts on operating the backend that receives it, and it comes before the capacity article because you cannot size what has not been migrated yet.

The analogy: changing the track gauge without closing the line

Changing the gauge of a line in service is not done in one go. The third rail is laid, rolling stock of both gauges runs for a while, and only when every train is of the new gauge is the old rail lifted.

The Langfuse migration has exactly that shape. dual mode is the third rail: the old tables and the new ones are written at the same time. And as with the track, the danger is not the day of the change, it is at both ends. Laying the new rail and lifting the old one on the same day strands trains. And once the old rail is up, going back is not a matter of will, it is a matter of the material no longer being there.

The three-rung ladder

Everything turns on one variable with three values, defined three times over in the three packages that have to agree, with the same enum and the same default (packages/shared/src/env.ts:337, worker/src/env.ts:586, web/src/env.mjs:610):

LANGFUSE_MIGRATION_V4_WRITE_MODE: z
 .enum(["legacy", "dual", "events_only"])
 .default("events_only"),

The two helpers that read it say what each rung does better than any explanation (worker/src/env.ts:705):

export const v4WritesToEventsTable = (envValue: ParsedEnv): boolean =>
 envValue.LANGFUSE_MIGRATION_V4_WRITE_MODE !== "legacy";

export const v4WritesToLegacyTables = (envValue: ParsedEnv): boolean =>
 envValue.LANGFUSE_MIGRATION_V4_WRITE_MODE !== "events_only";

What gets written in each mode:

Table	`legacy`	`dual`	`events_only`
`traces`, `observations`	yes	yes	no
`scores`	yes	yes	yes
`dataset_run_items_rmt`	yes	yes	no
`observations_batch_staging`	no	yes	no
`events_full`	no	yes	yes
`events_core`	no	yes (through the materialised view)	yes

The middle rung is the only one where both coexist, and therefore the only place you can step back from without losing anything. The code itself offers it as the remedy when something breaks in events_only (web/src/pages/api/public/ingestion.ts:296):

As a temporary migration bridge, set LANGFUSE_MIGRATION_V4_WRITE_MODE=dual on both the web and worker services and redeploy.

The both in that sentence matters. All three packages read the variable and the code comment insists three separate times on keeping them in sync. Deploying the worker with one value and the web with another is the first way to break this.

The three combinations that refuse to start

The worker validates its configuration at boot and throws in three cases, under a comment that leaves no doubt (worker/src/env.ts:718):

 // Hard errors: combinations that would silently lose data.
 if (mode === "legacy" && otel === "direct") {
 throw new Error(
 "Invalid V4 config: LANGFUSE_MIGRATION_V4_NATIVE_OTEL_BEHAVIOUR=direct " +
 "requires LANGFUSE_MIGRATION_V4_WRITE_MODE in {dual, events_only}. " +
 "Direct OTel writes target events_full, which is not read in legacy mode.",
 );
 }

The second one bites in a real migration: events_only also requires LANGFUSE_MIGRATION_V4_ALLOW_PREVIEW_OPT_IN=true, because otherwise the web reads would target the legacy tables that mode no longer writes. The third forbids events_only with OTel behaviour set to dual.

A failed boot is the good news. A failed boot shows up in thirty seconds. The combinations it does not validate take days to surface.

What breaks in each mode

This is the table to have in front of you before moving the variable, because the two ends break different things.

In legacy the event tables are not written, so whatever depends on them disappears. Monitor routes return 404 through a dedicated middleware (web/src/server/api/trpc.ts:413). Metrics v2 is unavailable. Dashboard widgets stay at version 1 (DashboardService.ts:39). The v4 preview is forced off and cannot be switched on.

In events_only the legacy tables are not written, so twenty-four REST routes short-circuit with a 404 (web/src/features/public-api/server/createAuthedProjectAPIRoute.ts:319):

 // Short-circuit routes that read from legacy traces/observations tables
 // when the deployment is in events_only mode — those tables are no longer
 // populated, so the response would be stale or empty.

The list covers traces, traces/{id}, observations, observations/{id}, sessions, sessions/{id}, scores and scores/{id} in v1 and v2, metrics, metrics/daily, spans, generations, events, dataset-run-items and the two datasets/{name}/runs routes. If you have your own integrations against those routes, they break on cutover day, and returning 404 rather than stale data is the right call even though it stings.

One detail that is usually reported wrongly: /api/public/ingestion is never shut down. The deprecation notice says so itself (deprecations.ts:104):

This route is never removed: scores (and sdk-log) stay accepted.

In events_only that route stays alive and keeps accepting two event types (web/src/pages/api/public/ingestion.ts:274):

const EVENTS_ONLY_ALLOWED_TYPES = new Set<string>([
 eventTypes.SCORE_CREATE,
 eventTypes.SDK_LOG,
]);

The rest are rejected per event, not per batch: the batch comes back as a 207 with 400s inside. An old client sending traces and scores mixed together will see the scores go through and the rest fail, which is exactly the kind of partial failure that takes a long time to notice.

The three steps, in order

Given all of the above, the order of a no-window migration is this:

Finish the v3 background migrations before upgrading. This comes before everything else and there are no shortcuts. v4 deletes the rows of seven old migrations, and the comment explains why (prisma/migrations/20260723124010_drop_dataset_run_items_table/migration.sql:7): their worker scripts are deleted alongside, so an unfinished row would leave the manager with no script to resolve. If one was halfway, that data is lost and the row disappears quietly.
Upgrade to v4 with LANGFUSE_MIGRATION_V4_WRITE_MODE=legacy, apply the ClickHouse and Prisma migrations. Here the installation behaves like v3 and you verify everything still stands.
Move to dual. From then on both table families are written and the propagator starts. Let it run for as long as it takes and launch the historic backfill.
Once history is complete and verified, move to events_only with the preview opt-in enabled, and only then lift the old rail.

All the work is between 3 and 4, and it can take weeks. There is no rush: dual is a stable state.

The ClickHouse migrations backfill nothing

There are 96 files in packages/shared/clickhouse/migrations/canonical/, 48 up and 48 down, which have to be materialised before running (pnpm ch:migrations:materialize) and cleaned afterwards, including after a failed migration. The runner is golang-migrate.

The ten v4 ones are 0039 through 0048. The first three create events_full, events_core and the materialised view that feeds the second from the first. The three drop migrations remove event_log, project_environments and dataset_run_items. The ones in between add columns and indexes.

What matters for planning is what they do not do. None copies an old row. The materialised view only sees rows inserted into events_full after it exists. The indexes in migrations 0043 and 0047 are added without materialising historical parts, and 0047 says so in writing:

Skip indexes are added without materializing historical parts to keep this migration metadata-only.

Translated into operations: during and after the migration, searches over older data do full scans. Not a bug, the price of a migration that does not block.

And one cluster trap: migrations 0042, 0043, 0047 and 0048 carry alter_sync = 2, meaning every ALTER waits for all replicas. One slow or down replica hangs the whole migration. On a single-replica installation you never notice; on three, you do.

The five-step chain that does backfill history

The backfill is done by five background migrations running in the worker, registered in Postgres with names that carry the order inside them:

20260701_v4_step_1_create_root_spans_from_traces: creates virtual root spans in events_full from existing traces, with the identifier concat('t-', t.id). It skips traces referenced by a dataset run item, which belong to step 4. Chunked by monthly partition.
20260701_v4_step_2_rewrite_observations_to_pid_tid_sorting: copies observations into a scratch table with a different key order. Chunked by monthly partition.
20260701_v4_step_3_backfill_events_full_from_observations: the bulk of it. Chunked by ClickHouse part, not by partition, and the comment explains why that is the only safe granularity:

One todo per part keeps each INSERT bounded to a single ClickHouse part (single-digit GBs in most cases), which is the only granularity that’s safe to assume self-hoster hardware can chew through without OOM or memory-limit failures.
20260701_v4_step_4_backfill_events_full_from_dataset_run_items: walks the dataset run items by cursor and writes the root and its descendants. Batches of 200, a cap of 50,000 descendants per item, a 90-day window.
20260701_v4_step_5_drop_pid_tid_sorting_tables: drops step 2’s scratch tables.

There are only two switches for the five. The first four share LANGFUSE_BACKGROUND_MIGRATION_V4_ENABLE_HISTORIC_BACKFILL, which ships true, and the fifth has its own, LANGFUSE_BACKGROUND_MIGRATION_V4_DROP_PID_TID_SORTING_TABLES, which ships false on purpose so the scratch tables stay around until the operator confirms the backfill went well.

One migration at a time across the whole cluster. The manager takes a lock in Postgres, refreshes it with a heartbeat every 15 seconds and treats a lock older than 60 seconds as expired, all inside a serialisable transaction. Adding worker replicas does not speed the backfill up by one minute. That is the first thing that surprises anyone used to scaling out to go faster.

They are resumable, and rather better than this usually is. Progress lives in a JSONB field on the row, and on resume the worker re-attaches to the queries the previous worker left running, by query id, instead of relaunching them:

In-flight queries keep running server-side; the next run re-attaches to them via the persisted queryIds.

On how long they take, the code says nothing. Not the files, not the folder’s README, not the migrations. The only time threshold anywhere is a design one: “A good threshold is something that takes more than 5 minutes to run”. Any figure in hours or days you read elsewhere, including the one I repeated myself before checking, is not in the code.

The failure that does not block

This is the finding that changes a runbook the most. When a background migration fails, the manager marks it with failedAt and moves on. And the predicate it uses to pick the next one excludes exactly the rows carrying failedAt, so a failed migration is skipped and does not block the ones after it.

The code acknowledges the gap and explains the patch (worker/src/backgroundMigrations/utils/backfillBase.ts:107):

The BackgroundMigrationManager has no dependency model — it runs every eligible row in name order and marks each finished/failed independently, so a failed upstream step does NOT by itself stop a downstream step from running on partial data. Each downstream step calls this from its validate() […] Because the next step in turn guards on its own predecessor, a single failure transitively halts the rest of the chain.

The guard exists, but only in steps 3 and 4. Steps 1 and 2 declare no predecessor. A failure of step 1 does not stop step 2 from starting; what saves the situation is that step 3 does check step 2.

Practical consequence: the only reliable signal that the backfill is going well is to look at the background_migrations table in Postgres and check that all five rows have finishedAt and none has failedAt. And to restart a failed one you clear failedAt by hand and relaunch with --retry-failed, as the error message itself says.

Two concrete failure modes that deserve a place in the runbook:

Step 3 breaks if ClickHouse merges a part midway. The message says it all: “Part … no longer active after processing — its rows are in a merged successor part that is not in this run’s todo list. Clear failedAt and re-run with state.chunksLoaded=false”. Recovery is manual, editing the JSONB.
Step 4 stops dead on one giant trace, if it exceeds 50,000 descendants. The three ways out the error offers are raising the cap, skipping the item (and losing that trace entirely) or fixing it by hand in SQL.

And an asymmetry worth knowing before comparing data: the historic backfill does not copy trace metadata down to child spans, deliberately, whereas the live path does concatenate them. History and new data are not identical.

Where the ten minutes come from

The v3 API warns that its data runs about ten minutes behind (deprecations.ts:16):

Data on this API is delayed by about 10 minutes. Real-time is only OpenTelemetry writes plus v2 observations and v2 metrics reads.

That delay is the sum of four things, all visible in the code:

The staging table partitions on three-minute intervals.
A partition is treated as closed four minutes after creation: “We ’lock’ partitions 4min after their creation, i.e. the 15:00:00 partition should stop receiving updates at 15:04:00.”
The propagation cron runs every minute and only looks at partitions older than LANGFUSE_EXPERIMENT_EVENT_PROPAGATION_PARTITION_DELAY_MINUTES, which defaults to 10.
That cron has a global concurrency of one across the cluster and processes a single partition per invocation, so a backlog makes the delay grow rather than recover.

The way out of the delay is not to lower that variable, it is to have clients write over OTLP with direct writes, which is the real-time route. The header that enables it is x-langfuse-ingestion-version: 4, sent automatically by the Python SDK from 4.0.0 and the JavaScript one from 5.0.0, and a value above 4 is rejected with a 400.

The forty-eight-hour TTL

This is the frightening one, and it appears in no migration guide.

The staging table carries TTL toDateTime(s3_first_seen_timestamp) + INTERVAL 48 HOUR with ttl_only_drop_parts = 1. If the propagator stops for more than forty-eight hours, a long bank holiday for instance, those partitions delete themselves. The data is already in the legacy tables, so an installation in dual keeps working, but the hole in events_full is permanent, and no code detects it and none repairs it. The health probe detects stuck, not lost.

The defence is an alert, and one exists for this. The worker health endpoint accepts ?failIfEventPropagationStuck=true and returns 503 if the propagator’s heartbeat has not been refreshed for more than LANGFUSE_EVENT_PROPAGATION_STUCK_THRESHOLD_MINUTES, which defaults to 35. The comment carries a condition to read before putting it in a Kubernetes probe:

Probes using this flag MUST set initialDelaySeconds >= 60s (one cron cycle) […] A shorter delay can crash-loop the very restart this check triggers.

Another detail of the same mechanism: the propagation cursor lives in Redis, not in Postgres. If Redis is flushed, the cursor resets and every live partition of the staging table is reprocessed.

What can and cannot be undone

The ClickHouse down migrations exist, and they are honest about what they do. The one for 0039 warns in capitals:

WARNING: destructive rollback. The up migration uses IF NOT EXISTS, so this table may predate this migration […] Rolling back drops that data regardless of which path created the table.

The ones for the dropped tables recreate the schema and nothing else: “Any data held in event_log before the up migration dropped it is NOT restored.”

But the real point of no return is not in ClickHouse, it is in Postgres. There is a Prisma migration called 20260723150000_drop_legacy_tracing_tables that does exactly what it says:

DROP TABLE IF EXISTS "traces";
DROP TABLE IF EXISTS "observations";
DROP TABLE IF EXISTS "scores";

Prisma Migrate has no down migrations. It applies, and that is that.

Summarising the real reversibility: the write mode can go down from events_only to dual whenever you want, and that is the panic button. What does not come back is what was not written while you were up there: the legacy tables receive nothing retroactively. And step 5 of the chain, the one that drops the scratch tables, ships disabled precisely so you can do forensics if something went wrong.

One last note on the ground you are standing on. The comment in the export-source policy drops something worth reading twice: “legacy-writes-disabled, which in practice only surfaces on self-hosted (Cloud does not run events_only)”. The events_only path is the one you will use and not the one the vendor’s managed service uses. It is less battle-tested in production than dual.

How this looks in code

The minimum runbook, in variables:

# Step 2: upgrade to v4 behaving like v3
LANGFUSE_MIGRATION_V4_WRITE_MODE=legacy
LANGFUSE_MIGRATION_V4_NATIVE_OTEL_BEHAVIOUR=dual_write # "direct" will not start here
LANGFUSE_BACKGROUND_MIGRATION_V4_ENABLE_HISTORIC_BACKFILL=false

# Step 3: third rail. Both services, web and worker, together.
LANGFUSE_MIGRATION_V4_WRITE_MODE=dual
LANGFUSE_BACKGROUND_MIGRATION_V4_ENABLE_HISTORIC_BACKFILL=true
LANGFUSE_EVENT_PROPAGATION_STUCK_THRESHOLD_MINUTES=35

# Step 4: only once all five background_migrations rows have finishedAt
LANGFUSE_MIGRATION_V4_WRITE_MODE=events_only
LANGFUSE_MIGRATION_V4_ALLOW_PREVIEW_OPT_IN=true # without this it will not start
LANGFUSE_BACKGROUND_MIGRATION_V4_DROP_PID_TID_SORTING_TABLES=true # last of all

And the query that answers the only question that matters for weeks:

SELECT name, started_at, finished_at, failed_at, failed_reason,
 state->>'chunksLoaded' AS chunks
FROM background_migrations
WHERE name LIKE '20260701_v4_%'
ORDER BY name;

Migration checklist

Every v3 background migration is finished before the image is upgraded.
There is a Postgres and a ClickHouse backup from before the first DROP TABLE.
The write mode has the same value in the worker and in the web, and both change together.
The upgrade went to legacy first, and the service was verified unchanged.
There is an alert on ?failIfEventPropagationStuck=true, with initialDelaySeconds of at least 60 seconds.
Somebody looks at the background_migrations table every day while the backfill runs.
The in-house integrations calling the twenty-four routes that will return 404 have been inventoried.
Client SDKs are on Python 4.0.0 or JavaScript 5.0.0 at minimum before moving to events_only.
Step 5 of the chain is enabled last, and only after verifying the backfill.

Traps

The default is the end state. A v3 installation booting v4 without touching the variable lands in events_only and stops writing the old tables from the first second.

A failed background migration does not block the next one. And the first two steps of the chain declare no predecessor.

More worker replicas do not speed up the backfill. One migration at a time across the cluster, by design.

Forty-eight hours of stopped propagator is a permanent hole. No detection and no repair.

New indexes do not cover old data. Historical searches scan until somebody materialises the indexes by hand.

On a cluster, every ALTER waits for all replicas. One down replica hangs the migration.

Classic ingestion rejects per event. A mixed batch returns 207 and half of it is lost without anyone reading the response body.

History and the live path do not produce quite the same thing. Trace metadata does not descend to children in the backfill.

The route you will take is not the one the vendor takes. Cloud does not run events_only.

The series: the ten articles

What goes into a trace: version 4’s data model, limits, precedences, scores, masking and indexes.
Putting LangGraph in front: instrumenting an agentic platform and the measured cost of one turn.
Two well-known agents instrumented: Open Deep Research and GPT Researcher, with the measured figures of a real request.
Migrating from version 3 to 4 with no window (this article): the three write-mode steps, the resumable background migrations and where the rollback point of no return sits.
The worker queues: the map of all thirty-nine, which pool to dedicate to each group, the per-queue switches, sharding and concurrency.
Real ClickHouse capacity and cost: how to measure bytes per observation with the system tables, the difference between the full table and the listings table, and the merge cost of full-text indexes.
Retention, deletion and data protection: why a deletion does not free disk, the mask cleaner that ships disabled, the pending deletion queue and the S3 lifecycle that has to be implemented by hand.
Backup and cross recovery: restore ordering across Postgres, ClickHouse and object storage, what each mismatch breaks, and how far event replay goes.
Saturation runbook: what to alert on from the queue metrics, the stuck probes, draining through the readiness endpoint and the dead letter queue.
Getting the data out: the blob storage integration to Parquet, batch exports and the metrics API, to build the data lake.

Sources

Langfuse 4.35.0 source, commit 2ee1908 of 14 September 2026: packages/shared/src/env.ts, worker/src/env.ts, web/src/env.mjs, web/src/features/public-api/server/deprecations.ts, web/src/pages/api/public/ingestion.ts, web/src/features/public-api/server/createAuthedProjectAPIRoute.ts, worker/src/backgroundMigrations/, worker/src/features/eventPropagation/, packages/shared/clickhouse/migrations/canonical/, packages/shared/prisma/migrations/.
Langfuse compatibility and migration, checked on 14 September 2026.
OpenTelemetry integration migration to v4, checked on 14 September 2026.

Langfuse v4, day 2 (3 of 10): two well-known agents instrumented, Open Deep Research and GPT Researcher

Mon, 14 Sep 2026 07:00:00 +0200

Third article in a series of ten about operating Langfuse v4 in production. The first walked through what goes into a trace; the second measured what an agent turn costs over a minimal graph. This one applies the same to two agents people deploy. Code read on 14 September 2026: Open Deep Research at commit 1b7d2e8 and GPT Researcher at 6f99857, with Langfuse 4.15.2 and LangGraph 1.2.11.

TL;DR

One Open Deep Research request with the default configuration is 1,416 observations and 8.1 MB. Measured over a faithful replica of its topology with five concurrent units, six supervisor iterations and ten react calls per researcher, which are its defaults. It all lands in a single trace. And it is a floor, because on the bench the search results are three hundred characters and real ones are much larger.

None of those 1,416 is of type AGENT. Its nine nodes are called clarify_with_user, write_research_brief, research_supervisor, final_report_generation, supervisor, supervisor_tools, researcher, researcher_tools and compress_research. None contains the string the handler looks for.

The parallel fan-out does not break the trace, and the fear that it does is unfounded. Open Deep Research forwards the config to its subgraphs and GPT Researcher passes a fresh one carrying only tags. I tested both patterns: one trace in both cases, because LangChain rebuilds the config from the context variable.

GPT Researcher is two products in one repository. The multi-agent mode is LangGraph and traces like any graph. The standard mode has no graph, and there every model call opens its own trace: eight calls, eight traces, measured.

Its human node does not use interrupt(), it waits on a websocket. The node’s observation stays open while the person thinks, so its duration measures human latency. The 95th percentile of that graph means nothing unless that node is excluded.

Instrumenting both requires no change to their repositories. For Open Deep Research it is enough to import the compiled graph and pass the handler through config. For GPT Researcher’s standard mode you have to wrap with the SDK, because there is nowhere to hook callbacks.

You are here: OBSERVE, day 2

The previous article produced a formula over a toy graph: 4 + 5n observations per turn, with the byte volume growing with the square of the steps. A formula over a toy is good for understanding the mechanism and no good for sizing anything.

This article applies it to two real agents, chosen because they are well known and because they break in different ways. The series grows to ten articles and the migration one moves along one place.

The analogy: two factories, one audit

A traceability audit in a factory that runs the whole process on its own lines is tedious but complete: every station leaves a record, and the auditor reconstructs the part from beginning to end. The volume of paper is enormous and that is the only problem.

The second factory subcontracts half the process. The auditor sees material going into a building and finished product coming out, with times and quantities, and has nothing about what happens inside. There is little paper. The problem is not the paper either.

Open Deep Research is the first factory and the problem is volume. GPT Researcher, in its standard mode, is the second, and the problem is that the trace does not carry what you need to know. Both are fixable, with different tools, and confusing them leads to buying disk when what is missing is instrumentation.

The two agents, and why these

Open Deep Research (langchain-ai/open_deep_research, 12.7 k stars) is LangChain’s reference research agent. All its work goes through LangChain, so callback instrumentation sees it whole.

GPT Researcher (assafelovic/gpt-researcher, 28.4 k stars) is the best-known open source research agent and it is not from LangChain. Its multi-agent mode uses LangGraph; its standard mode, which is what the API and the web interface run, does not.

One is in-house and the other is not, which for a comparison is what matters.

Open Deep Research: the shape of the graph

Three graphs, two of them nested as subgraphs (src/open_deep_research/deep_researcher.py):

The root graph, deep_researcher (line 701), with four nodes: clarify_with_user, write_research_brief, research_supervisor and final_report_generation.
The supervisor subgraph (line 353), with supervisor and supervisor_tools, iterating between them.
The researcher subgraph (line 589), with researcher, researcher_tools and compress_research, which is a classic react loop.

The multiplication lives in supervisor_tools. When the supervisor decides to delegate, that node launches one researcher subgraph per unit and awaits them together (line 305):

research_tasks = [
 researcher_subgraph.ainvoke({
 "researcher_messages": [HumanMessage(content=tool_call["args"]["research_topic"])],
 "research_topic": tool_call["args"]["research_topic"]
 }, config)
 for tool_call in allowed_conduct_research_calls
]

tool_results = await asyncio.gather(*research_tasks)

The detail that decides whether the trace comes out whole is that config at the end: the node receives LangGraph’s configuration and forwards it to the subgraphs, so the callbacks travel and every researcher hangs off the right place.

The three ceilings in the default configuration (src/open_deep_research/configuration.py) are max_concurrent_research_units = 5, max_researcher_iterations = 6 and max_react_tool_calls = 10. They multiply each other.

What one request costs, measured

I replicated that topology with fake models, the same technique as the previous article: three graphs, the same nesting, the same fan-out with asyncio.gather, three-hundred-character search results and two-hundred-character model replies. No network and no GPU, so it reproduces anywhere.

Units	Iterations	React calls	Observations	generation	chain	Attribute KB	Largest observation
1	1	1	26	8	18	56.8 KB	3.7 KB
2	2	3	92	26	66	257.3 KB	4.6 KB
5	6	10	1,416	370	1,046	8,108 KB	11.0 KB

The last row is the repository’s default configuration. One research request, one person pressing a button once, leaves 1,416 observations and eight megabytes in the observability backend. In a single trace, under a single identifier.

Those eight megabytes are a floor, not a ceiling. On the bench each search result takes three hundred characters; a real Tavily result carrying page content takes between ten and a hundred times more, and that text enters the history and gets re-serialised in every later observation of the same researcher, which is the previous article’s quadratic growth acting on much larger text.

Put into capacity arithmetic: a hundred research requests a day, which for a small team is nothing, is 141,600 observations and on the order of a gigabyte a day uncompressed, with toy searches. With real searches, tens of gigabytes. It is another order of magnitude compared with a chat assistant, and it is the reason this article comes before the capacity one.

Zero AGENT observations

Among those 1,416 observations there is not one of type AGENT. The previous article explained why: the handler decides that type by looking for the string agent in the class path or the run name. Open Deep Research’s nine nodes are clarify_with_user, write_research_brief, research_supervisor, final_report_generation, supervisor, supervisor_tools, researcher, researcher_tools and compress_research.

They are good names. They describe what each node does and they leave out a word that adds nothing for whoever reads the code. The result is that LangChain’s reference research agent, seen from Langfuse, is a tree of chains.

It is fixable from outside, without touching the repository, by naming the subgraph invocation or renaming the two nodes that represent a model decision, which are supervisor and researcher. Without that, any dashboard grouping by observation type will not tell decision work apart from plumbing.

Instrumenting it without touching the repository

Open Deep Research ships as a LangGraph Server graph, with its langgraph.json pointing at deep_researcher. There are two ways to get the handler in, depending on how you run it.

If you invoke it from your own code, the compiled graph is importable and the config is enough:

from langfuse import Langfuse, propagate_attributes
from langfuse.langchain import CallbackHandler
from open_deep_research.deep_researcher import deep_researcher

trace_id = Langfuse.create_trace_id(seed=f"{request_id}")
handler = CallbackHandler(trace_context={"trace_id": trace_id})

with propagate_attributes(
 trace_name="deep-research",
 user_id=user,
 session_id=request_id,
 tags=["odr", "research"],
):
 result = await deep_researcher.ainvoke(
 {"messages": [HumanMessage(question)]},
 config={
 "callbacks": [handler],
 "configurable": {
 "thread_id": request_id,
 "max_concurrent_research_units": 3,
 "max_researcher_iterations": 4,
 },
 },
 )

Lowering those two ceilings from 5 and 6 to 3 and 4 is not an observability recommendation, it is a product decision that also divides the trace volume by three. Better taken at the same time.

If you deploy it on LangGraph Server there is no invocation of yours to put the config into, so the handler gets bound at compile time. The pattern is to wrap the exported graph:

# instrumented_graph.py, and point langgraph.json here
from langfuse.langchain import CallbackHandler
from open_deep_research.deep_researcher import deep_researcher

graph = deep_researcher.with_config({"callbacks": [CallbackHandler()]})

That route loses the per-request seeded trace identifier, because the handler is created once. If the agent uses interrupts, that loss matters, for the reason the previous article measured.

GPT Researcher: two products in one repository

Here is the interesting part, and it is a trap you fall into with any agent that has grown in layers.

The repository holds the gpt_researcher package, which is the engine, and the multi_agents package, which is an orchestration on top. Only the second uses LangGraph.

The orchestrator (multi_agents/agents/orchestrator.py:66) has eight nodes: browser, planner, human, researcher, writer, fact_checker, visualizer and publisher, with two conditional loops, one for human review of the plan and one for fact checking. The researcher node launches in parallel one editor subgraph per section (multi_agents/agents/editor.py:134), and that subgraph has its own revision loop between reviewer and reviser.

All of that traces like any graph. And the models are called with self.llm.ainvoke(messages, **kwargs) from the generic provider (gpt_researcher/llm_provider/generic/base.py:365), which is a LangChain chat model with no explicit config, so it picks up the ambient configuration while running inside a node. The generations show up.

The standard mode is another thing. The GPTResearcher class has no graph, no nodes and does not go through LangGraph at any point. It is the mode the API and the web interface use, that is, the one most people run.

The myth of the fresh config

Before going on, something that looks like a bug and is not has to be dismantled, because it shows up in many agents.

When the researcher node launches the section subgraphs, it does it like this (multi_agents/agents/editor.py:74):

final_drafts = [
 chain.ainvoke(self._create_task_input(research_state, query, title),
 config={"tags": ["gpt-researcher"]})
 for query in queries
]
research_results = [result["draft"] for result in await asyncio.gather(*final_drafts)]

That config is fresh and carries only tags. The method does not even receive the node’s configuration, so it could not forward it. The intuitive conclusion is that the subgraphs lose the callbacks and every section ends up in its own trace.

It does not happen. I set both patterns side by side, Open Deep Research’s forwarding the config and GPT Researcher’s passing a fresh one, and both give one trace with one root span. The reason is that LangChain does not take the config you pass as is: it starts from the ambient configuration held in a context variable and overlays the keys you bring. The callbacks are still there, and context variables are copied into the tasks asyncio.gather creates.

What does get lost are the parent’s tags, replaced by the new list. That is a filtering nuisance, not a broken trace.

The standard mode: eight calls, eight traces

With no graph there is no root run, and with no root run every model call is its own tree. Measured with eight standalone calls to a chat model, with the handler passed by hand on each one:


Observations	8
Distinct traces	8
Root spans	8
Types	8 generation

Eight traces of one observation each, with no relation between them. None of them knows it belongs to the same piece of research. And that is the favourable case, because the standard mode offers no public place to put that handler: the **kwargs reach ainvoke from create_chat_completion (gpt_researcher/utils/llm.py:120), but the path crosses several internal layers that are not meant as an extension point.

The route that works without patching the repository is to wrap from outside with the SDK and let the inner calls hang off that:

from langfuse import get_client, observe, propagate_attributes
from gpt_researcher import GPTResearcher

@observe(name="gpt-researcher", as_type="agent")
async def research(question: str, request_id: str, user: str) -> str:
 with propagate_attributes(user_id=user, session_id=request_id, tags=["gptr"]):
 researcher = GPTResearcher(query=question, report_type="research_report")
 await researcher.conduct_research()
 report = await researcher.write_report()

 client = get_client()
 client.update_current_span(
 output={"length": len(report), "sources": len(researcher.get_source_urls())},
 metadata={"gptr_costs": researcher.get_costs()},
 )
 return report

With that there is one trace per piece of research, with an agent observation at the root and the generations hanging inside, in the same tree. Searches and document chunking do not appear, because they do not go through LangChain. If they are needed, they get added with start_as_current_observation(as_type="tool") around your own calls, which is the first of the three ways out of the span filter the previous article described.

Worth noting researcher.get_costs(). GPT Researcher keeps its own cost accounting, and putting it in as metadata is the only thing that lets you reconcile it later with what Langfuse computes. When the two figures disagree, it is almost always the gateway’s model alias.

The human node that stays open

The multi-agent mode has a human node that asks for an opinion on the plan before researching. It does not use LangGraph’s interrupt(). It waits inside the node, reading from the websocket (multi_agents/agents/human.py).

The two approaches have opposite consequences for the trace. With interrupt(), the graph hands control back and the trace splits in two, which is what the previous article measured. Waiting inside the node, the trace is single but that node’s observation stays open for as long as the person takes, minutes or hours.

Neither is wrong. What is wrong is the dashboard that averages durations without knowing which of the two it has in front of it. With the wait inside the node, the 95th percentile of graph duration measures how long a human takes to answer a message, not how long the agent takes. Alert on that metric and the alert fires at lunchtime.

The practical rule is to exclude human-wait nodes by name from any latency metric, and to measure that wait separately, which tends to be an interesting product metric in its own right.

What to look at in the trace once it arrives

With both agents instrumented, and before building any dashboard, there are four checks that are done once and save weeks:

Count the observations of a typical request. If it comes out above a thousand, this platform’s problem is going to be volume and sampling has to be decided before anything else.
Look for more than one root span. More than one means there is work being traced outside the tree, and you want to know which before there are thousands.
Check that generations carry cost. If they come in at zero, it is the model alias, fixed by registering that name with its price.
Sort observations by duration and remove the ones waiting on a person. What stays at the top is where the time really is.

Checklist

The graph is invoked with a trace identifier seeded per request, not a random one.
The agent’s concurrency and iteration ceilings are set deliberately, knowing they multiply trace volume.
Nodes representing a model decision carry agent in the name, or it has been accepted in writing that there will be no observations of that type.
Work that does not go through LangChain is wrapped with the SDK, or it has been decided that it need not be visible.
Human-wait nodes are identified and excluded from latency metrics.
The agent’s own cost accounting, if it has one, goes in as metadata so it can be reconciled.
There is a measured figure of observations per request, taken from your own installation and not from this article.

Traps

A research agent’s defaults are demo values, not production ones. Five units times six iterations times ten react calls multiply, and whoever chose them was thinking about report quality, not about your ClickHouse.

One repository can hold two agents with two different instrumentation stories. Instrumenting the mode the README documents and not the one the API runs is a mistake with no symptom: the traces arrive, just of something else.

Passing a fresh config to a subgraph does not break the trace. It breaks the tags. The tree survives because the callbacks travel through a context variable.

A node waiting on a person contaminates any duration statistic. And not obviously, because the node finishes cleanly and flags no error.

Good node names and good observation classification are conflicting goals. The winner is whoever writes the agent’s code, who is normally not you.

Eight megabytes per request with three-hundred-character searches. With real results, the number your installation produces will not look like this article’s, and it will be larger.

The series: the ten articles

What goes into a trace: version 4’s data model, limits, precedences, scores, masking and indexes.
Putting LangGraph in front: instrumenting an agentic platform and the measured cost of one turn.
Two well-known agents instrumented (this article): Open Deep Research and GPT Researcher, with the measured figures of a real request.
Migrating from version 3 to 4 with no window: the three write-mode steps, the resumable background migrations and where the rollback point of no return sits.
The worker queues: the map of all thirty-nine, which pool to dedicate to each group, the per-queue switches, sharding and concurrency.
Real ClickHouse capacity and cost: how to measure bytes per observation with the system tables, the difference between the full table and the listings table, and the merge cost of full-text indexes.
Retention, deletion and data protection: why a deletion does not free disk, the mask cleaner that ships disabled, the pending deletion queue and the S3 lifecycle that has to be implemented by hand.
Backup and cross recovery: restore ordering across Postgres, ClickHouse and object storage, what each mismatch breaks, and how far event replay goes.
Saturation runbook: what to alert on from the queue metrics, the stuck probes, draining through the readiness endpoint and the dead letter queue.
Getting the data out: the blob storage integration to Parquet, batch exports and the metrics API, to build the data lake.

Sources

Open Deep Research source, commit 1b7d2e8, read on 14 September 2026: src/open_deep_research/deep_researcher.py and configuration.py.
GPT Researcher source, commit 6f99857, read on 14 September 2026: multi_agents/agents/orchestrator.py, multi_agents/agents/editor.py, multi_agents/agents/human.py, gpt_researcher/llm_provider/generic/base.py and gpt_researcher/utils/llm.py.
Own measurement of 14 September 2026 over replicas of both topologies, with LangGraph 1.2.11, langchain-core 1.6.3, Langfuse SDK 4.15.2, InMemorySpanExporter and fake models, no network.
GitHub stars for both repositories, checked on 14 September 2026.

Langfuse v4, day 2 (2 of 10): putting LangGraph in front, and what one agent turn costs in observations

Sun, 13 Sep 2026 07:00:00 +0200

Second article in a series of ten about operating Langfuse v4 in production. The first one, what really goes into a trace, walked through the data model by reading the server code. This one measures the other side. Verified against Langfuse 4.15.2 (Python SDK), LangGraph 1.2.11 and langchain-core 1.6.3, with figures taken on 13 September 2026 on a bench with no network.

TL;DR

One agent turn is four observations plus five per tool call. Measured with an in-memory exporter over a minimal ReAct graph: zero tools, 4 observations; one, 9; five, 29; ten, 54; forty, 204. The formula holds exactly across the whole range.

The byte volume grows quadratically, not linearly. The same turn goes from 12.2 KB with one tool call to 1.87 MB with forty. 88 % of those bytes is the input field, because every observation re-serialises the full message history as it stands at that moment.

Tool definitions travel attached to every model call. The handler appends them to the input field as messages with role tool. Measured: 364 bytes per tool. With twenty tools that is 7.3 KB on every single step of the loop.

The AGENT observation type is assigned by string match on the name. A node called planificador comes out as CHAIN; the same node called agente_planificador comes out as AGENT. The logic is literally looking for agent in the class path or in the node name (langfuse/langchain/CallbackHandler.py:397).

An interrupt with a resume produces two separate traces. Verified with invoke, with stream and with subgraphs, and also reusing the same handler object. The SDK ships machinery to reattach the trace, but it only fires if the root run finishes with a control-flow exception, and in LangGraph 1.2.11 the interrupt comes back through the return value.

The fix fits in one line and is measured. Passing trace_context={"trace_id": Langfuse.create_trace_id(seed=thread_id)} when building the handler leaves the interrupt and the resume in the same trace, even with different handlers and therefore with different replicas.

The span processor drops by default whatever it does not recognise. It only exports spans from the SDK’s own tracer, spans carrying some gen_ai.* attribute, or spans from a closed list of thirty-five instrumentation prefixes (langfuse/_client/span_filter.py:11). Your own code’s spans do not arrive unless they come through one of those three doors.

You are here: OBSERVE, day 2

The series was planned with eight articles and has grown to ten. The reason is ordering: the next two are about migrating and about sizing ClickHouse, and both need an input figure that nobody publishes, which is how many observations and how many bytes a real unit of work produces. A chat produces a trace with two observations. An agent produces two hundred. Sizing with the chat figure leads to a cluster that falls over the day someone connects the agent.

So before migrating, measure. And to measure there has to be in front what people actually deploy.

The analogy: the file that gets photocopied whole at every signature

A procurement file passes across six desks. At each desk somebody reads what is there, adds a sheet and passes it on.

There are two ways to leave a record of that route. The first is to write in a register which desk touched it, when, and which sheet it added. Six short entries. The second is to photocopy the whole file on the way into each desk and on the way out, and to archive the twelve copies. The register takes one sheet. The copies take, at the sixth desk, six times what they took at the first, and in total they take something on the order of the square of the number of desks.

LangChain’s callback instrumentation does the second thing. It does not record the delta of each step, it records the complete state on the way into and out of each step. With a chat that makes no difference, because the state is short and there is one step. With a forty-step agent, the archive weighs almost two megabytes per file processed.

This is not a defect that needs fixing. It is what lets you open a trace and see exactly what the model saw at step 17 without reconstructing anything. But it drives the sizing, and the number has to be known before signing the contract.

Which agentic platform is the most used

The question has two answers depending on the metric, and both are defensible.

By PyPI downloads over the last thirty days, checked on 13 September 2026:

Package	Downloads / 30 days
`langgraph`	52.3 M
`strands-agents`	35.2 M
`openai-agents`	22.9 M
`crewai`	19.1 M
`google-adk`	14.5 M
`pydantic-ai`	8.0 M
`llama-index`	5.0 M
`agno`	1.9 M
`autogen-agentchat`	0.6 M

By GitHub stars the order flips at the top: CrewAI is at 58.4 k, LangGraph at 38.1 k and the OpenAI Agents SDK at 28.6 k.

PyPI downloads are inflated by continuous integration and by images that reinstall on every build, so they indicate presence in pipelines rather than agents in production. Stars measure interest, not deployment. With both together, LangGraph is the one that shows up in more pipelines and the one with the most worked-through instrumentation path in Langfuse, so it is the one instrumented here. What follows about volume applies equally to any platform instrumented through LangChain callbacks, because the cost comes from the callback model, not from the graph.

How LangGraph gets into Langfuse

The route is worth stating, because the previous article left two open and only one is used here.

langchain-core 1.6.3 does not import OpenTelemetry anywhere. The check is direct against the installed wheel and returns not one file. LangGraph does not emit OTLP spans on its own either. What exists is LangChain’s callback system, and the Langfuse CallbackHandler hooks into it.

That handler does not talk to the classic ingestion API. It creates observations with the v4 SDK, which underneath is OpenTelemetry: LangfuseSpanProcessor sets up an OTLPSpanExporter against {base_url}/api/public/otel/v1/traces (langfuse/_client/span_processor.py:123). So it is OTLP, but generated by the SDK, not by third-party auto-instrumentation.

The difference matters for one concrete reason left hanging in the previous article. Because the data passes through the SDK calls, the classic masking function does apply: input, output and metadata all go through _process_media_and_apply_mask before becoming attributes (langfuse/_client/span.py:534). With auto-instrumentation of the OpenInference or OpenLLMetry kind, the messages travel in span events and masking does not touch them. For an agent handling customer data inside its tools, that difference decides the route.

The trade-off is that a callback handler only sees what goes through LangChain. HTTP calls a tool makes on its own, the database query, the subprocess, none of that shows up. And if you instrument your code with OpenTelemetry so that it does show up, you run into the filter described further down.

The observation type is decided by the node name

The first article insisted that v4 has ten observation types and that using the right one changes what the interface knows how to group. The LangChain handler emits six of those ten: tool, retriever, generation, agent, chain and span. It never emits event, embedding, evaluator or guardrail.

The assignment is direct for four of them: tool callback, type tool; retriever callback, type retriever; model callback, type generation. For chain callbacks, which is what every node of a LangGraph graph is, the criterion is this (langfuse/langchain/CallbackHandler.py:393):

elif callback_type == "chain":
 # Detect if it's an agent by examining class path or name
 if serialized and "id" in serialized:
 class_path = serialized["id"]
 if any("agent" in part.lower() for part in class_path):
 return "agent"

 name = self.get_langchain_run_name(serialized, **kwargs)
 if "agent" in name.lower():
 return "agent"

 return "chain"

Measured over the same graph with the node renamed and everything else identical:

Node name	Tool calls	Observations	Types
`planificador`	10	54	11 generation, 33 chain, 10 tool
`agente_planificador`	10	54	11 generation, 22 chain, 11 agent, 10 tool

Eleven observations change type because of eleven characters in a function name. This has two practical consequences. The first is that any dashboard counting AGENT observations is counting naming conventions. The second is that the interface draws the agent graph when the trace contains any observation of a type other than span, event or generation, and chain qualifies, so the drawing comes out either way. What changes is what you can filter and aggregate afterwards.

The operational recommendation is boring and it works: name the nodes that represent a model decision with a name containing agent, and leave the rest as chains. It is a naming convention promoted to data schema, which is not pretty, but it is what there is and it is stable.

What a turn costs, measured

The setup is a minimal ReAct graph: one node that calls the model, a ToolNode with one tool, a conditional edge with tools_condition and an in-memory checkpointer. The model is langchain-core’s GenericFakeChatModel, which returns a fixed list of messages, so neither network nor GPU is needed and the experiment reproduces on any laptop. The exporter is OpenTelemetry’s InMemorySpanExporter, passed to the client through the span_exporter parameter, so observations are counted with no Langfuse server in front.

Result, with the node named planificador:

Tool calls	Observations	generation	chain	tool	Attribute bytes
0	4	1	3	0	4.2 KB
1	9	2	6	1	12.2 KB
5	29	6	18	5	64.1 KB
10	54	11	33	10	173.5 KB
20	104	21	63	20	541.4 KB
30	154	31	93	30	1,108 KB
40	204	41	123	40	1,874 KB

The observation count is exactly 4 + 5n. The five per loop iteration are: the model node, the model call, the conditional edge, the tools node and the tool. The four fixed ones are the root graph, the first partial pass and the close.

The bytes do not follow that line. From 10 to 20 steps the volume multiplies by 3.1; from 20 to 40, by 3.5. That is the quadratic growth expected when each of the n steps drags along a history whose length is proportional to n.

The breakdown by field confirms it. With forty steps, out of 1,709 KB measured in that run, 1,502 KB sit in langfuse.observation.input and only 56 KB in the output. 88 % of what is ingested is the same history repeated at different stages of progress. The largest single observation in that turn is 22.2 KB, well below the 9.5 MB per OTLP span ceiling, so the problem is not the size limit: it is the aggregate.

And inside the input there is a second multiplier. The handler appends each tool’s definition to the generation’s input field, as messages with role tool (langfuse/langchain/CallbackHandler.py:1190). Measured with three-parameter tools and a one-line description:

Bound tools	Input field bytes of one generation
0	34
1	398
5	1,854
10	3,674
20	7,334

That is 364 bytes per tool, on every generation, always. An agent with twenty tools and forty steps ingests 7.3 KB × 41 generations = 300 KB in tool schemas alone, which never change over the course of the turn.

What this means for ClickHouse

A honest caveat belongs here. I compressed the full payload of those runs with ZSTD level 3, which is what events_full uses, and got ratios between 29× and 86×, improving the longer the turn. That number is inflated by how synthetic the bench is: my tool always returns the same string, so the redundancy is higher than in production.

What does hold is the shape. Repeating the history is exactly the pattern columnar compression absorbs well, so the ClickHouse disk is not where it hurts. Where it hurts is earlier: in the bandwidth towards the OTLP endpoint, in the worker CPU that decompresses and flattens, and in the ingestion queue, all of which see uncompressed bytes. The first article noted that the queue that saturates first is langfuse.queue.ingestion.depth, and this is why it saturates.

Quick sizing arithmetic, with the measured figures and an average ten-step turn:

1,000 turns a day is 54,000 observations and around 170 MB uncompressed a day.
10,000 turns a day is 540,000 observations and 1.7 GB uncompressed a day.
If the average turn goes from ten steps to twenty, observations double and bytes triple.

Sensitivity to the number of steps is the parameter to watch, more than the number of users. A prompt change that makes the agent take three more turns around the loop multiplies the observability bill with nobody touching the infrastructure. Article 6 of the series comes back to this with ClickHouse system tables to measure the real bytes per observation in your own installation, which is the only thing that replaces this estimate.

Interrupts: two traces where there should be one

A production agent with human approval uses interrupt(), hands control back, waits for a person and resumes with Command(resume=...). The operational question is whether the trace survives that pause.

The SDK ships explicit machinery for the case. There is a store of pending trace contexts keyed by thread_id, capped at 1,024 entries with eviction of the oldest (MAX_PENDING_RESUME_TRACE_CONTEXTS), and a function that recognises a Command carrying resume and recovers the stored context.

Measured, it does not fire in this combination of versions:

Case	Same trace?
Same handler object, `invoke`	No
Same handler object, `stream`	No
Interrupt inside a subgraph	No
Fresh handler for the resume	No

The reason is in the code and it is consistent. The pending context is only stored from on_chain_error when the failing run is the root one and the resulting level is DEFAULT, which is what happens with LangGraph’s control-flow exceptions (CallbackHandler.py:866). In LangGraph 1.2.11 the interrupt does not leave the root as an exception: the graph returns normally, with the interrupt in the return value. The root run ends through on_chain_end and the branch that stores the context never runs. The interrupted node’s observation does come out right: level DEFAULT and the Interrupt in the status message, not marked as an error, which is the correct behaviour.

The fix is verified and does not depend on the SDK changing:

from langfuse import Langfuse
from langfuse.langchain import CallbackHandler

trace_id = Langfuse.create_trace_id(seed=thread_id)
handler = CallbackHandler(trace_context={"trace_id": trace_id})

With that, the interrupt and the resume land in the same trace even if the handler is a different object and even if another replica serves the resume, because the identifier derives from the thread_id and not from in-memory state. It is the same technique the documentation recommends for joining several agents into one trace, applied to joining an agent with itself.

With one nuance to decide deliberately. Seeding with the bare thread_id puts every turn of that conversation into a single trace, which grows without bound while the conversation stays alive. If what you want is one trace per turn with the resume attached to the right turn, seed with thread_id plus the turn number, and leave conversation grouping to the session identifier, which is what it is for. Since v4 has no trace table, a huge trace is not a huge row: it is a filter returning many rows, and what suffers is the interface when opening it.

Thread, session, user, and what is only read at the root

The handler recognises five metadata keys: langfuse_session_id, langfuse_user_id, langfuse_trace_name, langfuse_tags and langfuse_prompt. The first four are read only when parent_run_id is None, that is, only on the root run (CallbackHandler.py:587). Putting them in the config used to invoke a subgraph does nothing.

The recommended route, and the one that works without surprises, is the context manager:

from langfuse import propagate_attributes

with propagate_attributes(
 trace_name="agent-turn",
 user_id=user,
 session_id=thread_id,
 tags=["production", "support-l1"],
):
 graph.invoke(payload, config=cfg)

What LangGraph contributes on its own arrives anyway. In the measured run these show up as observation metadata: langgraph_node, langgraph_step, langgraph_triggers, langgraph_path, langgraph_checkpoint_ns, checkpoint_ns and thread_id, plus ls_provider, ls_model_type and ls_integration. The thread_id is also promoted to trace metadata.

That is fine, but keeping in mind the first article: in events_core what is indexed is metadata_names, that is the key names, not the values. Filtering by langgraph_node = "planificador" is a scan. If you are going to segment by node routinely, the right place is the observation name, which is where the node already appears, or a tag, assuming that tags are not indexed either. The cheap field for slicing by node in v4 is the observation name itself.

The session identifier deserves an explicit decision. LangGraph’s thread_id and Langfuse’s session_id are the same concept and they do not connect on their own. Mapping them one to one is the reasonable thing, and it is also what makes the whole conversation navigable in the interface even when each turn is a separate trace.

The filter that drops your own spans

This is the finding that surprises most people while setting it up. The Langfuse span processor does not export everything that passes through the TracerProvider. It exports what satisfies one of these three conditions (langfuse/_client/span_filter.py:104):

It was created by the Langfuse SDK’s own tracer.
It carries at least one attribute starting with gen_ai.
Its instrumentation scope matches one of thirty-five prefixes in a closed list.

The list covers the expected and a bit more: openinference, litellm, haystack, langsmith, strands-agents, pydantic-ai, autogen-core, vllm and a couple of dozen opentelemetry.instrumentation.* entries.

The consequence for an agent is direct. If you instrument your tools with OpenTelemetry by hand so that the ERP call or the Postgres query shows up inside the agent’s trace, those spans are dropped silently, with no log and no error. There are three ways out, in order of cleanliness:

Create those observations with the Langfuse SDK, with start_as_current_observation(as_type="tool"), which is what the rest of this article assumes.
Put some gen_ai.* attribute on the span, which is abusing the convention to smuggle the span through, so only if the span does describe an interaction with a model.
Send those spans to your normal OpenTelemetry collector instead of Langfuse, and correlate by trace identifier. For a sovereign platform with its own collector, this is the sensible option: Langfuse keeps the model side, the collector keeps the system side, and the trace identifier joins them.

The behaviour being silent is what makes this take an afternoon to find. It belongs in the runbook.

The model and the cost with a gateway in the middle

One detail that affects anyone putting LiteLLM in front, which is the setup described in the operational pair and in GPU multi-tenancy.

The model name the handler puts on the generation comes first from the ls_model_name metadata and, failing that, from the component’s serialisation (CallbackHandler.py:1297). With a ChatOpenAI pointed at the gateway, that name is whatever alias you defined in LiteLLM, for example qwen-32b-internal. Langfuse computes cost from its own model table, which does not know that alias, so the generation arrives with tokens and without cost.

It is fixed by defining the model in Langfuse under that same name with its price per million tokens, which is also the only way for the cost to reflect what your GPU costs rather than what somebody else’s API costs. It is an afternoon of work and it stops the agentic platform’s cost panel from sitting at zero forever.

How this looks in code

The minimum to have in place for a LangGraph agent to produce useful traces in Langfuse v4:

import os
from typing import Annotated, TypedDict

from langfuse import Langfuse, get_client, propagate_attributes
from langfuse.langchain import CallbackHandler
from langchain_core.messages import HumanMessage
from langgraph.graph import StateGraph, START
from langgraph.graph.message import add_messages
from langgraph.prebuilt import ToolNode, tools_condition

# 1. Client. Masking applies because the data goes through the SDK.
def mask(*, data, **kwargs):
 return redact_pii(data)

Langfuse(
 public_key=os.environ["LANGFUSE_PUBLIC_KEY"],
 secret_key=os.environ["LANGFUSE_SECRET_KEY"],
 host=os.environ["LANGFUSE_HOST"],
 mask=mask,
 environment="prod", # lowercase, dashes, 40 chars, must not start with "langfuse"
 sample_rate=float(os.getenv("LANGFUSE_SAMPLE_RATE", "1.0")),
)

# 2. Graph. The decision node carries "agent" in its name on purpose.
class State(TypedDict):
 messages: Annotated[list, add_messages]

graph = StateGraph(State)
graph.add_node("agent_planner", model_node)
graph.add_node("tools", ToolNode(TOOLS))
graph.add_edge(START, "agent_planner")
graph.add_conditional_edges("agent_planner", tools_condition)
graph.add_edge("tools", "agent_planner")
app = graph.compile(checkpointer=checkpointer)

# 3. Per turn: deterministic trace seeded with thread and turn, session = thread.
def run_turn(thread_id: str, turn: int, text: str, user: str):
 trace_id = Langfuse.create_trace_id(seed=f"{thread_id}:{turn}")
 handler = CallbackHandler(trace_context={"trace_id": trace_id})
 cfg = {
 "callbacks": [handler],
 "configurable": {"thread_id": thread_id},
 }
 with propagate_attributes(
 trace_name="agent-turn",
 user_id=user,
 session_id=thread_id,
 tags=["agent", "support"],
 ):
 out = app.invoke({"messages": [HumanMessage(text)]}, config=cfg)
 return trace_id, out

# 4. Resume after human approval: same seed, same trace.
def resume_turn(thread_id: str, turn: int, decision):
 from langgraph.types import Command

 trace_id = Langfuse.create_trace_id(seed=f"{thread_id}:{turn}")
 handler = CallbackHandler(trace_context={"trace_id": trace_id})
 cfg = {"callbacks": [handler], "configurable": {"thread_id": thread_id}}
 return app.invoke(Command(resume=decision), config=cfg)

# 5. Score the turn. Scores do not travel over OTLP: they have their own route.
get_client().create_score(
 trace_id=trace_id,
 name="resolved",
 value=1,
 data_type="BOOLEAN",
)

Two notes on what is not in the example. sample_rate is head sampling and per trace, so it samples whole turns, which is what you want: half an agent trace is no use to anybody. And if the process is a web server, the client is built once at startup, not per request, whereas the handler can be per turn with no problem, because with the seeded identifier it no longer holds state worth preserving.

Agent instrumentation checklist

Nodes representing a model decision carry agent in the name. The rest do not.
Every turn has a deterministic trace identifier seeded with thread and turn number.
LangGraph’s thread_id goes as Langfuse’s session_id, always.
Trace attributes are set with propagate_attributes, not as metadata on a subgraph.
The gateway’s model alias is registered in the Langfuse model table, with a price.
Tools doing work outside LangChain are instrumented with the Langfuse SDK, not with bare OpenTelemetry.
There is an alert on ingestion queue depth, and the average number of steps per turn is known.
Someone has measured, on the actual installation, how many observations and how many bytes a real turn produces. The figures in this article come from a synthetic bench.
Sampling is set through an environment variable, so it can be lowered without deploying code.

Traps

Type AGENT does not mean there is an agent. It means somebody wrote agent in the name. And the other way around: a whole graph without that string produces not one agent-type observation.

Putting langfuse_session_id in a subgraph’s metadata does nothing. It is only read on the root run.

An interrupt splits the trace in two. Unless you seed the identifier. And with several replicas, the handler’s in-memory state is never going to save you, because the resume can land on another pod.

Your OpenTelemetry spans do not arrive. The default filter drops them without warning if the scope is not in the list and there are no gen_ai.* attributes.

The number to watch is not users, it is steps per turn. Volume grows with the square of the steps, and the steps are decided by the prompt, not by the infrastructure.

You store megabytes and the interface shows you a thousand characters. The materialised view’s truncation to two hundred characters and the thousand-character read ceiling are still there. You pay for the full ingestion and see a clipping, unless you open the specific observation.

Cost comes out at zero with the gateway in front. Model alias unknown to the price table.

flush_at above the queue size blows up when constructing the client. With an OpenTelemetry ValueError that does not mention Langfuse anywhere.

The series: the ten articles

What goes into a trace: version 4’s data model, limits, precedences, scores, masking and indexes.
Putting LangGraph in front (this article): instrumenting an agentic platform and the measured cost of one turn.
Two well-known agents instrumented: Open Deep Research and GPT Researcher, with the measured figures of a real request.
Migrating from version 3 to 4 with no window: the three write-mode steps, the resumable background migrations and where the rollback point of no return sits.
The worker queues: the map of all thirty-nine, which pool to dedicate to each group, the per-queue switches, sharding and concurrency.
Real ClickHouse capacity and cost: how to measure bytes per observation with the system tables, the difference between the full table and the listings table, and the merge cost of full-text indexes.
Retention, deletion and data protection: why a deletion does not free disk, the mask cleaner that ships disabled, the pending deletion queue and the S3 lifecycle that has to be implemented by hand.
Backup and cross recovery: restore ordering across Postgres, ClickHouse and object storage, what each mismatch breaks, and how far event replay goes.
Saturation runbook: what to alert on from the queue metrics, the stuck probes, draining through the readiness endpoint and the dead letter queue.
Getting the data out: the blob storage integration to Parquet, batch exports and the metrics API, to build the data lake.

Sources

Own measurement of 13 September 2026: LangGraph 1.2.11, langchain-core 1.6.3 and Langfuse Python SDK 4.15.2, with OpenTelemetry’s InMemorySpanExporter and GenericFakeChatModel, no network.
Langfuse SDK 4.15.2 source: langfuse/langchain/CallbackHandler.py, langfuse/_client/span_filter.py, langfuse/_client/span_processor.py, langfuse/_client/span.py.
PyPI downloads over the last thirty days via pypistats, checked on 13 September 2026.
GitHub stars for LangGraph, CrewAI and the OpenAI Agents SDK, checked on 13 September 2026.

Completing Keycloak for MCP: the protected resource no SDK hands you ready-made, and the enterprise extension that skips the whole flow

Sat, 12 Sep 2026 09:00:00 +0200

A follow-up to Keycloak in an AI platform, which flagged the gap: resource indicators are not supported and protected resource metadata belongs to the MCP server. This post tries to close it. Verified against revision 2026-07-28 of the specification, the official Python and TypeScript SDKs, Keycloak 26.7.3 and LiteLLM 1.102.0.

TL;DR

The specification splits three obligations and the authorisation server covers only one. The MCP server must publish its protected resource metadata. The client must send the resource indicator in both requests, and must do so even if the authorisation server does not support it. The MCP server must validate that the token was issued for it. Keycloak does not understand the indicator and considers the metadata someone else’s, so all three end up in the resource layer.

The Python SDK mounts the metadata on its own, but degraded. As soon as the resource server URL is configured, the route appears. What that automatic route publishes carries a single authorisation server, reuses as its scope catalogue the ones the middleware requires, and leaves the name and the documentation null. To publish a complete document you have to mount the route by hand.

Audience validation comes switched off in Python and does not exist in TypeScript. In Python, if the resource URL is configured and the corresponding flag is not enabled, the SDK emits a deprecation warning and behaves as if it were disabled; the docstring itself promises that version 3 will set it to true. In TypeScript the bearer verification does three things, and none of them is looking at the audience. Neither of the two ships a signature verifier with public keys.

Token passthrough is forbidden in writing, and the substitute has a name. The specification says the MCP server must not accept or relay tokens that were not issued for it, and that if it calls upstream APIs the token must be a different one. The mechanism is token exchange, which Keycloak supports in its standard version with a limitation you need to know: the audience is a client identifier, not a resource URL.

There is an official extension that changes the whole picture, and it is stable. Enterprise-managed authorisation replaces the redirect to the MCP authorisation server with an exchange at the corporate identity provider, which evaluates the policy and issues a grant based on an identity assertion. It has been stable since June 2026 and there are servers in production.

Keycloak implements it halfway and in experimental. It only knows how to act as receiver, not as issuer, behind a feature flag, and against draft 01 when the working group is on 04. The official documentation literally says not to use it in production.

LiteLLM already implements it on the client side and does not document it. In 1.102.0 there is an authentication mode towards MCP servers that runs the two stages of that grant. There is not a single documentation file that mentions it.

You are here: the resource side

The previous post walked through the identity provider. This one walks through what sits on the other side of the token, which is where the MCP specification puts almost all its normative obligations.

The natural reading order is When MCP grows up for the basic setup, Keycloak in an AI platform for the identity piece, and this one for the missing layer. The MCP gateway seen from the inside is in its own post.

The analogy: the visa and the border checkpoint

A consulate issues visas. A border post checks them. They are two different offices, and the classic mistake consists of assuming that because the visa is genuine it is good for entering through any door.

The visa has written on it which country it is good for. That is the token’s audience, and here the problem appears: the consulate we work with does not know how to write the destination the traveller asks for, because it does not understand that field on the form. It knows how to write a destination if asked for it under a different name, by way of a pre-agreed stamp, which is the workaround of scopes and the audience mapper.

And then there is the other half, the one almost nobody builds. The border post has to exist, it has to announce where it is and from which consulates it accepts visas, and it has to read the destination written on the visa before letting anyone through. If the border post limits itself to checking that the stamp is genuine and does not look at the destination, any valid visa from the same consulate is good for entering. That is exactly what the two official SDKs do by default.

Part 1. The split of obligations

From revision 2026-07-28, with the normative words as they are:

Obligation	Whose	Status with Keycloak
Publish protected resource metadata (RFC 9728)	MCP server, MUST	Out of its scope, by design
Send the resource indicator (RFC 8707) in authorisation and in token	Client, MUST, even if the authorisation server does not support it	It does not understand it
Validate that the token was issued for oneself	MCP server, MUST	The resource’s responsibility
Not accept or relay other tokens	MCP server, MUST NOT	The resource’s responsibility
Validate the issuer of the authorisation response (RFC 9207)	Client, MUST; authorisation server SHOULD emit it	Supported
Consent for each dynamically registered client	MCP proxy with a static identifier, MUST	Configurable

The most confusing part of that split is the second line. The specification obliges the client to send the parameter regardless of whether the authorisation server supports it. With Keycloak, that parameter is lost: the official documentation says it cannot recognise it, and the standard behaviour towards an unknown parameter is to ignore it. The client complies, the token comes out, and what does not come out is the correct audience. The failure appears later and somewhere else, which is the worst kind of failure.

It is worth knowing where that work stands. The matter is issue 14355 of the project, open, with a milestone in 26.8.0. There was a complete implementation in pull request 35711, with a mapper of its own and an extension point for resolving resources, which went back to draft in October 2025 because of a total lack of tests and because it was decided to redo it in phases, starting with a single resource per client. In March 2026 a new issue was opened for experimental support, also with a milestone in 26.8.0 and with no associated pull request. In 26.7.x there is no feature flag that enables it.

What is new in the revision and affects the setup

Four things changed with respect to the previous revision and are worth collecting:

RFC 9207. A new section on validating the authorisation response. The client must record the issuer from the validated metadata document, and the comparison is literal: you are not allowed to normalise case, elide the default port, add or remove the trailing slash or re-encode characters before comparing.
Offline tokens leave the catalogue. A new section on refresh tokens: MCP servers should not include the corresponding scope either in the challenge header or in the supported scopes of their metadata.
Dynamic registration becomes deprecated. It goes from optional to optional and deprecated, retained for compatibility with authorisation servers that do not support client identifier metadata documents. And a new requirement is added to registration: the application type is mandatory, and a client must not reuse credentials from another authorisation server, it has to register again.
Step-up elevation is rewritten. The menu of three server strategies disappears and is replaced by two rules: the challenge’s scope attribute describes what is needed for the requested resource, with no obligation to include what has already been granted, and scope accumulation becomes the client’s responsibility. With a new requirement for the server: it must take scope hierarchies into account, where a broad one implies the narrow ones.

Part 2. What the SDKs give, and what they do not

Here is the uncomfortable part, and it is the main reason for writing this post. Verified by reading the code of the Python SDK and of the TypeScript 2.0.0-alpha one.

Protected resource metadata

In Python the complete model exists and it mounts on its own. The metadata class carries the RFC fields: resource, authorisation servers with a minimum of one, key set URL, supported scopes, bearer methods with a default value of header, name, documentation, policy, terms, and the client certificate and DPoP fields. The handler serves the document with a one-hour cache directive, and the URL is built by inserting the well-known path in front of the resource path, as the RFC requires.

As soon as the resource server URL is configured, the route appears with no further work. But what that automatic route publishes is degraded in three points: it passes a single authorisation server, it takes as its scope catalogue the ones the middleware requires (which are not the same as the ones the resource supports), and it passes neither the name nor the documentation, which come out null. To publish a complete document you have to call the route creation function by hand.

In TypeScript it exists and does not mount on its own. The function that builds the document emits only five fields: resource, authorisation servers, supported scopes, name and documentation. It does not emit the supported bearer methods, nor the key set URL, nor anything about DPoP. And serving it is explicit: either you call the response function from your own handler, or you mount the metadata router. There is no point in the SDK that does it on its own.

Audience validation

This is the one to fix on day one.

In Python it comes disabled. The token verifier is a single-method protocol, that is to say a hole the implementer fills. The access token’s resource field is the indicator you set yourself. And the comparison only happens if you ask for it: there is a function that normalises as a URL and ignores the trailing slash, but the resource server URL reaches it empty unless the validation flag is enabled.

The detail to read twice is in the configuration: if the resource URL is set and the flag is not, the SDK raises a deprecation warning and behaves as if the flag were false. The docstring says version 3 will set the default to true. Until then, a Python server configured with authorisation accepts tokens issued for another resource, unless the implementer enables the flag or validates the audience inside their own verifier.

In TypeScript it is not validated at all. Bearer token verification does exactly three things: strip the prefix, check the required scopes, and require that the expiry is present and not past. A search for audience or key set across the server and middleware packages returns no validation. It is all left in the hands of whichever verifier you plug in.

Neither of the two ships a signature verifier with public keys. In Python the only example is introspection against the authorisation server, and its audience check sits behind a flag that also comes out false. In TypeScript the interface is empty. That is: the component that validates the token, which is the one holding up the most important normative requirement of the specification, is your own code in both cases.

The authentication challenge

Python builds the challenge with the error and its description, and adds the metadata URL only if it is configured. It returns 401 with invalid token and 403 with insufficient scope, both from the same function, so both carry the metadata URL. What it never emits is the scope parameter, which revision 2026-07-28 asks to be included, and which is exactly what the client needs in order to know what to ask for.

TypeScript does emit the scope when there are required scopes, in addition to the metadata URL, and it correctly maps invalid token to 401 and insufficient scope to 403.

That is, each SDK gets a different half right. The Python one validates better and warns worse; the TypeScript one warns better and does not validate.

Scopes, which are not per tool

In both SDKs scopes are a static list at the transport mounting level, not per tool. There is no per-tool scope declaration and no step-up elevation help on the server side. Emitting a 403 with the specific scope that call requires, which is what the specification describes, is your own code.

That is the origin of almost all the fine-grained authorisation you have to build, and it is why there is a Part 6.

Which revision each one announces

Python is already on 2026-07-28. The revision appears in the known versions and in the list of modern versions, described as the ones that use the stateless envelope per request. The server discovery method is registered with a default handler, and the method and name headers are validated against the body.

TypeScript has it, but in a separate list. The latest protocol version constant is still 2025-11-25, because that list is only the one for the initial handshake. The modern era lives in a separate module, with a comment explaining the reason: keeping them deliberately separate so that adding a revision there never leaks a modern version string into a 2025-era handshake.

And it is worth recalling the mismatch that already came up in the MCP gateway post: LiteLLM 1.102.0 still announces 2025-06-18.

Part 3. Building the protected resource

With the above, the list of what you have to write is short and concrete.

1. A token verifier that validates the signature and the audience. Against the realm’s key set, checking issuer, expiry and that the audience contains the MCP server’s canonical URL and only things that concern it. Here the side effect of the Keycloak workaround comes in: if a client asks for two scopes from two different resources, out comes a token with two audiences, because each mapper contributes its own to the array. There is no client policy condition or executor in 26.7 that limits the number of audiences per token. The practical defence is the resource’s: reject tokens whose audience includes other resources, instead of merely checking that its own is present.

2. The complete metadata document, with all the authorisation servers you accept, the real catalogue of scopes the resource understands, and the name and documentation filled in. In Python, mounting the route by hand instead of leaving the automatic one. In TypeScript, mounting it, full stop.

3. The authentication challenge with the scope. In Python you have to add it, because the SDK does not emit it. And you have to take scope hierarchies into account, which is a new requirement of the revision.

4. The decision between local validation and introspection. Validating the signature locally costs no network, but revocation does not take effect until the token expires: there are no revocation lists for signed tokens. Introspection costs a round trip per request and only confidential clients can invoke it. The reasonable combination is a short token life and introspection on the operations that change state.

5. Binding state to the user. The new revision is explicit about this, because with protocol sessions gone, state is carried in handlers that travel as an ordinary tool argument. Servers must verify every incoming request and must not treat possession of a handler as authentication; and they should bind the handler to the user on the server side, for example storing the state under a key that combines the user identifier derived from the verified token with the handler, and reject the handler if someone else presents it. It is a change in the way of working compared with the sessions of before.

Part 4. The hop from the gateway to the MCP server

The real shape in an inference platform is not client against MCP server: it is client against gateway, and gateway against MCP server. That second hop is where it is decided whether the architecture is correct.

The specification leaves no room. The MCP server must not accept or relay tokens that were not issued for it. And if it makes requests to upstream APIs, it may act as an OAuth client of theirs, but the token it uses there is a different token, issued by the upstream authorisation server, and it must not forward the one it received.

The standard mechanism for that is token exchange, and here you need to know three Keycloak details.

First, the permission model changed. The old version required fine-grained administration permissions and an explicit exchange authorisation on the target client. The standard version, supported since 26.2, does not require them: it is enough for the requesting client to be confidential and to have the corresponding switch enabled. In exchange there is a surprising condition: the subject token has to carry the requesting client in its audience, unless it exchanges its own token. That is, the gateway needs to appear in the audience of the user token, which is achieved with an audience mapper in a default scope of the gateway. Only then can it narrow the audience to that of the specific MCP server.

Second, the audience is a client identifier, not a resource URL. The parameter filters audiences, that is to say it narrows them, which is what you want. But it takes the identifier of a client registered in the realm. The practical consequence is that the MCP server has to be registered as a client, and its identifier should be its canonical URL, so that the resulting audience matches what the resource validates. It is a naming trick, and it has to be documented for whoever comes after. The documentation itself acknowledges it: token exchange does not yet support the resource parameter.

Third, the subject token type is limited. The standard version only accepts access tokens as subject.

There is also a new experimental feature in 26.7 that adds a parameterised scope type for validating whether the requesting user is authorised to act on behalf of another. Interesting for delegation, but experimental.

What the gateway does today

In LiteLLM 1.102.0 the authentication modes towards MCP servers number twelve, with a default value of none. The ones that matter for this discussion are four.

oauth2_token_exchange implements the standard exchange. It sends the correct grant type, the subject token and its type, and the audience, never the resource. The omission is deliberate according to the code itself: fabricating a target risks an invalid target error. It caches the resulting token under a key that combines the subject token with the whole configuration, and with a lifetime equal to the token’s minus one minute. And it blocks, never degrades: with no incoming token it returns 401, which the edge turns into a challenge with the resource metadata URL; a rejection from the identity provider returns 401; a gateway configuration error returns 500; a transport failure, 503.

true_passthrough and oauth_delegate forward the client’s authorisation header as is. That is token passthrough, by the name the specification gives it, and its legitimate use is narrow: when the token the client presents was already issued for the target MCP server and the gateway is a mere transport. Outside that case, it is non-compliant.

oauth2_id_jag is the surprise, and it deserves a part of its own.

Part 5. The enterprise extension, which changes the picture

In June 2026 the MCP project published an authorisation extension that solves a different problem from the one the standard flow solves, and which in an organisation with an identity provider of its own is the real problem.

The normal flow is per user: each employee authorises each client against each MCP server. It works for consumer applications and it does not work in a company, because onboarding a person requires authorising dozens of services one by one and offboarding requires revoking them one by one.

The io.modelcontextprotocol/enterprise-managed-authorization extension inverts that. It is in the stable specifications directory of the extensions repository, it comes from SEP-990, and the flow is this:

The MCP client authenticates the user against the corporate identity provider through the normal flow, and stores the identity assertion, which may be an OpenID identity token or a SAML assertion.
When the server indicates that enterprise-managed authorisation is needed, the client exchanges that assertion at the corporate provider for an authorisation grant based on an identity assertion. The provider evaluates the organisation’s policy there: group membership, roles, conditional access.
The client presents that grant to the MCP authorisation server and obtains an access token.
The sentence that defines the extension, verbatim: the user is not redirected to the authorisation endpoint of the MCP authorisation server.

The MCP authorisation server validates the signature against the corporate provider’s key set, plus audience, issuer and expiry, and uses the subject as the user’s stable identifier, with the email as a fallback for linking earlier accounts.

What that buys is exactly what a compliance file asks for: policy in a single place, an auditable decision at the identity provider, and centralised revocation that takes effect across all clients at once. The employee who loses access stops receiving grants, without touching any server.

The underlying standard is a draft from the IETF’s OAuth working group, under the name of JWT identity assertion authorisation grant, revision 04 of May 2026, signed by people from Okta, Ping Identity and an independent author. It profiles identity chaining across trust domains by combining token exchange with the JWT profile.

And here comes the problem

Keycloak implements it only halfway. It has its own documentation page, behind a feature flag. And three limitations you need in front of you before designing anything:

It only acts as receiver. It accepts assertions issued by an external provider and issues local tokens. Native support for acting as issuer, the documentation says, is not yet fully implemented. In the extension’s flow, the issuer is the corporate provider: if that provider is Keycloak, the missing piece is exactly the one you need.
It is based on draft 01 while the working group is on 04.
The official documentation says not to use it in production.

The asymmetry is striking and worth pointing out: Keycloak does not support a standard published eight years ago such as resource indicators, and does partially support a draft from a few months back.

What is already in the gateway, undocumented

In LiteLLM 1.102.0, the oauth2_id_jag mode runs the two stages of the flow. In the first it asks the corporate provider for a token exchange with the requested token type specific to this grant, sending the user’s identity token as subject and, optionally, audience, resource and scope. In the second it presents the resulting assertion to the resource’s authorisation server with the JWT bearer grant type. In both the gateway authenticates as a client with a JWT signed by private key.

Two operational warnings that come out of the code and are nowhere else:

The subject comes from the caller’s identity token, or from the one captured at login. And only the generic OIDC provider captures assertions: with Google, Microsoft or SAML there is none, and all users fail. The code raises a warning when loading the configuration.
A badly configured server refuses with a 500 instead of falling back to the static credential. It is the right decision, and it is worth knowing before it happens.

And the main warning: there is no documentation. Not a single file in the repository that mentions this mode. Anyone using it is reading code, like this post.

Part 6. Per-tool authorisation

None of the previous pieces answers the question the business ends up asking: whether this person may call this tool with these arguments. The identity provider gives roles and scopes, and with that you do not model a graph of relations, as was already argued in the previous post.

What is published today, with sources:

OpenFGA is the only one with a dedicated product guide, updated on 9 September 2026. The pattern is a tool type with an invocation relation, checked on each call, and a listing query so the client only sees the tools it may invoke. It is a pattern and a modelling language, not a reference implementation.
Pomerium ships a per-tool policy criterion, with exact, prefix, suffix and list matching.
Kong offers it as a product, with per-tool control lists.
Traefik Hub goes as far as parameter-level restrictions, which is the finest published grain.
Envoy has an MCP filter that extracts protocol attributes for fine-grained access control, declared to be in active development and with no OAuth of its own: the hook is external authorisation towards a policy engine.

The lowest-friction option in a platform that already has a service mesh is external authorisation against a policy engine. The most expressive one, the relations model.

Part 7. What still has no standard

Two gaps the specification does not cover and that have to be closed from outside.

Tool definition change. Neither revision 2026-07-28 nor the SDKs pin a hash of the description or of the input schema. The best practices page covers confused deputy, token passthrough, server-side request forgery in discovery, state handler hijacking, server confusion and validation of the authorisation URL scheme, but there is no integrity or pinning requirement. The tool that does it is the Invariant Labs scanner, under the Apache 2.0 licence, which pins hashes to detect definition change after approval and also ships a proxy mode with guardrails. It already came up in the MCP gateway post and it remains the only answer.

The public guidance that does get into this is the NSA’s on MCP security design considerations, from May 2026, done with Carnegie Mellon’s software engineering institute. Two of its observations are directly actionable: that authorisation in MCP is optional and that the specification imposes no requirement on token lifecycle management, so expiry and rotation are left in the organisation’s hands; and the warning against dynamic tool discovery with no origin verification and no authorisation checks, which is the most direct acknowledgement of the risk in a state guidance document.

Part 8. Who covers what

With all of the above, the decision table for the layer you put in front of the MCP server:

Piece	Publishes resource metadata	Validates audience	Token exchange	Per-tool authorisation
Python SDK, automatic route	Yes, degraded	Not by default	No	No
TypeScript SDK	Does not mount on its own	No	No	No
Traefik Hub (commercial)	Yes, automatic	Not documented	Not documented	Yes, down to parameter
Kong, OAuth plugin for MCP	Not documented	Yes	Not documented	Yes, per-tool lists
Pomerium	Partial	Not documented	Not documented	Yes, per-tool criterion
agentgateway (Solo.io)	Yes	Yes, by JWT policy	Yes, including the assertion grant	Yes, with expressions
mcp-context-forge (IBM)	Not in release notes	Not documented	Yes, since 1.0.6	Yes, per-role control
Envoy, MCP filter	No	No	No	Via external authorisation
LiteLLM 1.102.0	Yes, and authorisation server metadata too	Not on the incoming JWT	Yes, and the assertion grant	Yes, per-key and per-team permissions

Two warnings about that table. The Kong plugin requires a minimum version of 3.12, is in technical preview and needs a licence for its AI edition; it also introduces a breaking change in 3.13, which starts treating all traffic as MCP in order to close a possible authentication bypass. And on oauth2-proxy I found no primary documentation either for or against protected resource mode: I am not claiming it does not have it, I am claiming it is not documented where I could look.

Reference configuration

Keycloak: the per-resource scope, optional, with its audience. That it is optional and not default is the key piece, because it is what makes the client have to ask for it explicitly and makes it work as a substitute for the resource indicator:

kcadm.sh create client-scopes -r plataforma \
 -s name=mcp:inventario -s protocol=openid-connect \
 -s 'attributes."include.in.token.scope"=true'

kcadm.sh create "client-scopes/<id>/protocol-mappers/models" -r plataforma \
 -s name=aud-mcp-inventario \
 -s protocolMapper=oidc-audience-mapper \
 -s 'config."included.custom.audience"=https://mcp.ejemplo.es/inventario' \
 -s 'config."access.token.claim"=true'

And the MCP server registered as a client with its canonical URL as identifier, so that token exchange produces the right audience:

kcadm.sh create clients -r plataforma \
 -s clientId=https://mcp.ejemplo.es/inventario \
 -s enabled=true -s publicClient=false -s consentRequired=true

The gateway, with token exchange instead of token passthrough:

mcp_servers:
 inventario:
 url: "https://mcp.ejemplo.es/inventario"
 transport: "http"
 auth_type: "oauth2_token_exchange"
 client_id: "litellm-gateway"
 client_secret: os.environ/GW_SECRET
 token_exchange_endpoint: "https://sso.ejemplo.es/realms/plataforma/protocol/openid-connect/token"
 audience: "https://mcp.ejemplo.es/inventario"
 subject_token_type: "urn:ietf:params:oauth:token-type:access_token"
 token_exchange_profile: "rfc8693"

The resource verifier, which is what nobody hands you ready-made. The essential bit is the last check, the one that rejects other resources’ audiences instead of settling for finding its own:

RECURSO = "https://mcp.ejemplo.es/inventario"

async def verificar(token: str) -> AccessToken | None:
 claims = jwt.decode(
 token, await jwks(),
 algorithms=["RS256"],
 audience=RECURSO, # validates that I am there
 issuer=ISSUER,
 )
 aud = claims["aud"]
 aud = [aud] if isinstance(aud, str) else aud
 # and that nobody else is: the scopes workaround allows
 # tokens with two audiences, and that reopens the confused deputy
 if set(aud) - {RECURSO, CLIENTE_GATEWAY}:
 return None
 return AccessToken(
 token=token,
 client_id=claims["azp"],
 scopes=claims.get("scope", "").split(),
 expires_at=claims["exp"],
 resource=RECURSO,
 )

And when mounting the server, the flag that does not come set:

auth = AuthSettings(
 issuer_url=ISSUER,
 resource_server_url=RECURSO,
 validate_token_resource=True, # without this there is only a warning
 required_scopes=["mcp:inventario"],
)

Checklist

Enable audience validation in the Python SDK, or write it into the verifier if it is TypeScript. Without that, the central normative requirement is not met.
Reject tokens with other resources’ audiences, not just check that your own is present.
Mount the metadata document by hand, with all the authorisation servers and the real catalogue of scopes.
Add the scope parameter to the challenge, in Python, and account for scope hierarchies.
Register each MCP server as a Keycloak client with its canonical URL as identifier, so that token exchange produces the right audience.
Replace any token passthrough mode with token exchange, except the narrow case where the token was already issued for the target.
Check that the gateway appears in the audience of the user token, or the exchange will fail.
Bind state handlers to the user on the server side, and never treat them as authentication.
Decide between local validation and introspection knowing that the first delays revocation until expiry, and shorten the token life accordingly.
Pin a hash of the description and the input schema from outside, because the specification does not cover it.
Put a policy engine in for per-tool authorisation, and do not try to model it with roles.
If you are considering the enterprise extension, check first who issues the grant: Keycloak today only knows how to receive it, in experimental and against an earlier draft.

Traps and things that are not what they look like

The Python SDK mounts the metadata document on its own, and that is why it looks solved. What it publishes carries a single authorisation server and no name and no documentation.

The audience validation deprecation warning does not enable anything. It warns and carries on behaving as disabled.

The TypeScript SDK does not validate the audience at all. It is not configuration, it is that there is no code for it.

Python’s challenge does not carry the scope, so a client that receives a 403 does not know what to ask for.

The scopes workaround allows two audiences in the same token. Each mapper contributes its own, and there is no client policy that limits it. The defence is in the resource.

The token exchange audience is a client identifier, not a URL. If the MCP server is not registered with its URL as identifier, the resulting audience will not match what the resource validates.

Token exchange requires the gateway to be in the audience of the user token. It is the condition that wastes the most time the first time round.

Dynamic client registration is deprecated as of this revision, and the substitute, client identifier metadata documents, is experimental in Keycloak and has an open bug with documents carrying unknown fields, which blocks login with real clients.

LiteLLM’s identity assertion grant depends on the generic OIDC provider. With Google, Microsoft or SAML no assertion is captured and all users fail.

Step-up elevation is no longer the server’s responsibility. Scope accumulation moved to the client in this revision.

The authentication elevation signal goes in a 401, not a 403. Insufficient scope is 403; asking for a higher authentication level is a different thing and a different code.

Closing

The summary of this post fits in one sentence: the identity provider is the cheap half of the problem, and the expensive half is the protected resource, which no SDK hands you ready-made and which in the two official ones comes with audience validation disabled or absent.

From that comes an order of work. First the token verifier, with the audience validated in both directions, because without it everything else is decoration. Second, the complete metadata document, mounted by hand. Third, replacing any token passthrough with exchange, with the MCP server registered with its URL as identifier. Fourth, a policy engine for what roles do not model.

And a scheduling decision worth taking with the information in front of you. The enterprise-managed authorisation extension is the right route for an organisation with a provider of its own, it is stable and it has servers in production behind it. What is not ready is Keycloak as issuer. Anyone who wants that flow today has two honest options: delegate the mechanics to a gateway that implements it, or wait. Putting it into production with a feature the official documentation marks as experimental and expressly advises against is not one of them.

Sources

Especificación MCP, revisión 2026-07-28: autorización, registro de clientes y buenas prácticas de seguridad.
Extensión de autorización gestionada por la empresa y su especificación estable, del SEP-990. Anuncio de 18 de junio de 2026.
draft-ietf-oauth-identity-assertion-authz-grant, revisión 04 de 21 de mayo de 2026, grupo de trabajo OAuth del IETF.
Código de los SDK oficiales de MCP: python-sdk y typescript-sdk 2.0.0-alfa, clonados el 12 de septiembre de 2026.
Integración de Keycloak con Model Context Protocol, intercambio de tokens y concesión de autorización por aserción de identidad, consultados el 12 de septiembre de 2026.
Incidencias de Keycloak 14355, 47117 y 51039, y petición de cambios 35711.
Código de LiteLLM 1.102.0, commit 9071ca50 del 11 de septiembre de 2026.
NSA y SEI, Model Context Protocol: Security Design Considerations, mayo de 2026.
OpenFGA, autorización de servidores MCP, actualizado el 9 de septiembre de 2026.
mcp-scan de Invariant Labs, Apache 2.0.
Documentación de agentgateway, Traefik Hub, Kong y Pomerium, consultadas el 12 de septiembre de 2026.

Keycloak in an AI platform: where it fits, what it solves and the two standards MCP requires and it does not implement

Sat, 12 Sep 2026 07:30:00 +0200

This post opens the identity block of the operations track. The previous one covered the gateway side: the JWT that does not validate audience, the user who never reaches the trace. This one covers the piece on the other side. Verified against Keycloak 26.7.3, revision 2026-07-28 of the MCP specification and the text of Royal Decree 311/2022 in the BOE.

TL;DR

Keycloak 26.7.3 was released on 31 August 2026 and is still incubating at the CNCF, which it entered in April 2023. The cadence is four minor releases a year and support in practice covers a single branch: the twenty security bugs fixed in 26.7.3 were not backported to the 26.4, 26.5 or 26.6 branches at the time of release.

The Quarkus-based distribution separates build and runtime configuration. What is set in the build phase is frozen into the image, and changing it at startup forces a rebuild. In containers, the pattern is to build into the image and start with the optimised flag.

Storage is relational and only relational. The alternative engine that was in development was withdrawn in version 25.0. Since 26.0 sessions are persisted to the database by default, which changes the load profile: the dominant cost in the official benchmarks is database CPU.

Organisations solves multi-tenancy inside a realm, fully supported since 26.0. It groups members, associates email domains and links identity providers per organisation. It is the alternative to replicating realms per customer.

The measured bottleneck is password hashing. Since 25.0 the default algorithm outside FIPS environments is Argon2, with around 7 MB of memory per operation. The official sizing guide gives one vCPU for every 15 password logins per second, against 120 per second for client credentials and for refreshes.

Keycloak does not implement two standards the MCP specification marks as mandatory. Protected resource metadata belongs to the MCP server, not the authorisation server, so there is nothing to hold against it there. Resource indicators do belong to it and are not supported. Keycloak’s own documentation declares partial conformance with the last three MCP revisions for that reason. The practical substitute is scopes plus an audience mapper.

There is a recent critical bug worth looking at today. CVE-2026-18963, scored 9.1, allowed resetting any user’s password without authenticating, administrators included. Fixed in 26.7.2, released on 19 August 2026.

You are here: the cross-cutting layer

Identity is not a stage of the pipeline, it is a layer that crosses all of them. It shows up in stack hardening as one of the seven layers of defence, in the model’s chain of trust as workload identity, in the gateway as token validation, and in the post about authenticating MCP servers as authorisation server.

What was missing was the post about the piece itself, and above all the decision of where to put it and what to expect from it.

The analogy: the civil registry and building permits

A small town hall has a population register, which says who lives there, and it has permits, which say who can do what. They are two different things and it is worth not confusing them, because the register is good for the whole town while a permit is good for one specific job on one specific street.

Keycloak is the register. It knows who each person is, identifies them reliably, knows which groups they belong to and issues a credential document that expires. What it does not do is decide whether this person can modify this particular file: that is the permit, and it lives in each service.

Confusing the two produces the most expensive design mistake in platforms of this kind, which consists of putting into the token everything that will need to be decided later. The register document grows, gets handed out everywhere, and when a permission changes you have to wait for everyone’s document to expire.

There is a third element the analogy captures well. A credential document issued for presenting at the tax office should not be good for entering the municipal depot. That is the token’s audience, and it is the central subject of the second half of this post.

Part 1. What Keycloak is today

Project status

Item	Value
Latest stable version	26.7.3, released on 31 August 2026
Previous ones	26.7.2 (19 August), 26.7.0 (9 July), 26.6.0 (8 April)
Cadence	four minor releases a year
CNCF	incubating since 10 April 2023, not graduated
Support	the current branch; earlier ones do not receive fixes reliably

That last point deserves underlining because it conditions the upgrade policy. The analysis of 26.7.3 shows that its twenty security fixes were not backported to earlier branches at release time. Operating Keycloak means keeping up to date with the current branch, not staying on a “stable” version for a year.

Since 26.0 there is a backward compatibility guarantee across minor versions for features marked as fully supported, and breaking changes are opt-in.

The internal architecture in five pieces

Two startup modes. The development one enables HTTP, does not resolve the hostname strictly and disables the theme cache. The production one requires a hostname and TLS, or it will not start. Confusing them in production is the first antipattern on the final list.

The build phase. The build command materialises the optimisations into the image. Build options, such as the database engine, the enabled features or metrics, are frozen. Runtime ones, such as the database password or the hostname, are applied at startup. Changing a build option at startup forces a rebuild with the corresponding latency cost. The correct pattern in containers is to build into the image and start with the optimised flag.

Extensions via SPI. Authenticators, protocol mappers, user storage, event listeners, hashing algorithms and secret vaults. They are packaged as JARs and resolved in the build phase.

Infinispan cache. Embedded by default, with database-based discovery as the recommended mechanism. The discovery alternatives via Kubernetes DNS, TCP and UDP are deprecated. There are local caches, with default capacities of 10,000 entries in several of them, and replicated caches for sessions, client sessions, offline sessions, authentication sessions, login failures and action tokens. Remote mode, with an external Infinispan, is needed for first-generation multi-site deployment.

Relational storage. JPA over a database. The alternative engine was removed in version 25.0. The databases supported today include PostgreSQL from 14 to 18, MariaDB and MySQL in their extended support versions, SQL Server 2019 and 2022, Oracle 19c and 23.x, and the managed variants Aurora PostgreSQL and Azure SQL.

The minimum conceptual model

Before touching anything you need to be clear on eight concepts, and the order in which they relate:

Realm: the isolation boundary. Its own users, its own signing keys, its own policies. Two realms share nothing.
Client: an application that asks for tokens. Public if it cannot keep a secret, such as a browser application; confidential if it can, such as a service; and bearer-only if it merely validates tokens without requesting them.
Client scope: a reusable bundle of scopes and mappers, assigned to clients as default or as optional. It is the unit with which you bound what a token carries.
Protocol mapper: what projects an attribute, a role or an audience into the token. The audience mapper is the one that solves the central problem of this post.
Role, realm or client, and composite role, which aggregates others.
Group, hierarchical, with inherited roles and attributes.
Identity provider: federation towards another external OIDC or SAML. User federation: LDAP, Active Directory and Kerberos, with synchronisation and attribute mapping.
Authentication flow: a sequence of executions with required, alternative, conditional or disabled requirements, and subflows. It is where you build a second factor or a check of your own.

And one feature that changes architectures: organisations, fully supported since 26.0. It solves multi-tenancy inside a single realm, grouping members, associating email domains and linking identity providers per organisation, so that provider discovery is done by the user’s email domain. The classic alternative, one realm per customer, multiplies administration and shares no configuration.

Part 2. Who can authenticate against it

This is the table you need before drawing anything. The right-hand column is usually the one that decides.

Piece	Protocol	Cost
Kubernetes API	native OIDC, or the structured authentication configuration file	Free
LiteLLM	OIDC and JWT, plus UI SSO	JWT and SSO above five users: Enterprise
vLLM	static key only	No OIDC
Langfuse	generic OIDC and Keycloak provider	SSO free; per-project access control: Enterprise
Grafana	generic OAuth yes in the open edition	Team synchronisation: Enterprise
Backstage	generic OIDC and community-maintained Keycloak provider	Free
Open WebUI	OIDC, with role management by claim	Free
LibreChat	generic OIDC, with Keycloak documented	Free
Argo CD	native OIDC or Dex, with groups into its RBAC file	Free
MinIO	OIDC with role assumption via web identity	See the note below

Two clarifications that change decisions.

Kubernetes. The modern mechanism is structured authentication configuration, which allows several JWT issuers plus validations and mappings with CEL expressions. The corresponding proposal, KEP-3331, went alpha in 1.29, beta in 1.30 and is stable since 1.34. It is the route to use today, and it replaces the loose API server flags.

vLLM has no identity. All there is is a static key via environment variable. This is not a defect to fix, it is an architectural fact: the engine sits behind the gateway and its access control is network-based, not identity-based. Any design that exposes vLLM directly to users is giving up per-person traceability.

About MinIO a note of honesty is in order: the public documentation now redirects to the commercial edition, so the detail of how it maps policies from a claim is verified there and not in the open edition. Anyone depending on that mechanism should confirm it against their own version before designing on top of it.

Part 3. Humans and machines

They are two different problems and it is worth solving them separately.

People

Authorization code with PKCE, with the S256 method. The MCP specification requires it when the client is technically capable, and it goes further: it mandates rejecting the flow if the authorisation server metadata does not publish the supported challenge methods.

Workloads with no person behind them

Here the obvious option is client credentials with a Keycloak service account, authenticated with a secret or, better, with a JWT signed by private key.

But there is a less known and better option for workloads inside the cluster: federated client authentication, introduced in January 2026 under the banner of doing away with secrets. It lets a client authenticate with a token issued by an external provider, and one of those providers is Kubernetes. The announcement placed full support for the Kubernetes service account variant and for OIDC in version 26.6; the SPIFFE variant, by contrast, expressly says it will stay in preview until the SPIFFE client authentication standard is finalised at the IETF.

That means a pod can authenticate against Keycloak with the token the kubelet already projects into it, with no additional secret to rotate. It is worth confirming the exact status and the flag name against the release notes of whichever version you have deployed, because the feature is recent and has changed status between 26.5 and 26.6. It is preferable to token exchange, whose standard version is supported but is internal-to-internal only, while the old one, which covered external-to-internal and impersonation, is deprecated and in preview.

Keycloak against SPIFFE

The question always comes up and the answer is that they solve different things with a small overlap.

SPIRE attests the node and the process, and issues short-lived identity documents with no prior secret. Keycloak does not attest workloads: it trusts a credential someone had to place there, except in the federated authentication case just mentioned, where the one attesting is Kubernetes. In exchange, Keycloak federates people, issues tokens with business data and governs consent, things SPIFFE does not do.

The practical rule: SPIFFE for service-to-service identity inside the perimeter, Keycloak for anything that has a person behind it or crosses the perimeter. The SPIFFE detail is in the close of the chain of trust series.

Part 4. MCP, and the two missing standards

This part has the most consequences and is the worst documented outside the primary sources.

What the specification requires

From revision 2026-07-28, with the normative words as they appear:

The MCP server must implement RFC 9728 protected resource metadata, and the client must use it to discover the authorisation server.
The client must implement RFC 8707 resource indicators, including the resource parameter in both the authorisation request and the token request, and must send it regardless of whether the authorisation server supports it.
The MCP server must validate that the access token was issued specifically for it as the audience, and must not accept or relay any other token.
Authorisation servers and clients should support client identifier metadata documents. RFC 7591 dynamic registration becomes optional and is declared deprecated, retained for compatibility.

The revision also adds the issuer in the authorisation response, from RFC 9207, and a stateless model with state handlers where it is expressly stated that holding a state handler is not equivalent to being authenticated.

What Keycloak supports

From its own MCP integration guide:

Standard	Keycloak
OAuth 2.1 and authorisation server metadata	Supported
RFC 9207, issuer in the response	Supported
RFC 7591, dynamic registration	Supported
Client identifier metadata document	Supported, experimental, after enabling the feature
RFC 8707, resource indicators	Not supported
RFC 9728, protected resource metadata	Out of scope, it belongs to the MCP server

The conformance declared by the project itself is “partially supported, without resource indicators” for the last three MCP revisions.

Two practical conclusions:

The protected resource metadata document is published by the MCP server or a proxy in front of it, never by Keycloak. That is correct and there is nothing to fix, but you have to remember to implement it, because the client needs it to discover where to go.
Without resource indicators, the audience is bounded with scopes. The official route is to use the scope parameter instead of resource, and to hang from that client scope an audience mapper with the MCP server URL as the included custom audience.

It is a workaround, it works, and it has to be documented in the design because anyone coming after will look for the standard parameter and will not find it.

Part 5. The confused deputy

With the above on the table, the classic mistake becomes obvious on its own. A token with a generic audience turns any compromised service into a master key: whoever obtains the token a user presented to the dashboard can present it to the gateway, and from there to the MCP server.

The MCP specification is explicit in three sentences worth quoting verbatim in any architecture document:

MCP servers must not accept tokens that were not explicitly issued for them.
If the MCP server calls upstream APIs, it must not forward the token it received from the client.
MCP proxies that use static client identifiers must obtain user consent for each dynamically registered client before forwarding to third-party authorisation servers.

Translated into the specific figure this track has been dealing with: the gateway must not forward the user’s JWT to third-party MCP servers. It has to obtain a different token with the right audience, or act as an OAuth client in its own right. By default, as already documented in the MCP gateway post, the behaviour is to use the server’s own credential, which is the correct one; there are two modes that forward the caller’s token and both have a name in the specification.

Part 6. Two hours of task against five minutes of token

The specification pushes towards short-lived tokens and mandatory refresh token rotation in public clients. A two-hour agentic task comfortably exceeds the life of an access token and may also exceed session idle time if nobody refreshes.

Three real options, with their cost:

Offline tokens. They survive session expiry, and expire through their own idle timeout or through a maximum if that limit is enabled. The risk is obvious: a near-permanent credential, hard to inventory. And the MCP specification adds that MCP servers should not include that scope either in the authentication header or in the supported scopes of their metadata.

Automatic refresh in the client. It is the right answer when there is a client that can implement it, with refresh token rotation and secure storage. For an agent with no person behind it, it is better to renew via client credentials.

Long-lived virtual keys in the gateway. It works, and it is what most people end up doing. The price has to be written down in the risk analysis: identity ends up decoupled from the provider, so deactivating someone in Keycloak does not invalidate their key, and traceability back to the real person depends on the discipline with which the keys were created. The levers that compensate are expiry, automatic rotation and a mandatory alias, covered in the virtual keys post.

A note of honesty about the default values. The numbers going around (five minutes of access token, thirty of session idle time, ten hours of maximum) do not appear published in the administration guide and I did not locate them in the code. Treat them as a console observation and confirm them in your own realm before quoting them in a document.

Part 7. Sizing and operating

On Kubernetes

The operator exposes three custom resources: the server, the realm import and the OIDC client. Probes and metrics live on management port 9000, with the health and readiness paths, and metrics in Prometheus format provided the corresponding option was enabled in the build phase. That port is not exposed publicly.

For high availability there are three shapes. A single cluster spread across zones, with no additional infrastructure. First-generation multi-cluster, with an external load balancer and an external Infinispan at each site. And second-generation multi-cluster, introduced in 26.7.0 and still in preview, which connects clusters without external Infinispan, using the synchronously replicated database as the source of truth, with less cache and more database load.

The official sizing figures

From the CPU and memory sizing concepts guide, which is still published:

1 vCPU for every 15 password logins per second, tested up to 300 per second.
1 vCPU for every 120 client credential grants per second, tested up to 2,000.
1 vCPU for every 120 token refresh requests per second, tested up to 435.
Leave a 150 % CPU headroom for spikes, startup and failover.
1,250 MB of memory per pod with realm data and 10,000 cached sessions. Keycloak allocates 70 % of the limit to the heap and around 300 MB outside it, so the limit comes from subtracting those 300 MB from the expected usage and dividing by 0.7.

The stated assumptions behind those figures include Argon2 with five iterations, sessions in the database and the default cache of 10,000 entries.

The difference between 15 and 120 is the whole story of Keycloak’s performance: the cost is in password hashing. Since 25.0, the default algorithm outside FIPS environments is Argon2, with around 7 MB of memory per operation. Before it was PBKDF2, whose iterations were multiplied by ten in version 24. In FIPS mode PBKDF2 is still used.

From that comes an architectural consequence for an AI platform: machine workloads, which go through client credentials, cost eight times less than a human login. Sizing by number of services is cheap; sizing by morning login peaks is the expensive part.

The benchmark

From the 26.4 performance report, published on 1 October 2025: up to 2,000 logins per second and 10,000 refreshes per second with three pods, 74 vCPU in total and 8 GB per pod, over an Aurora PostgreSQL database, with almost linear vertical scaling.

Two useful findings from that report. With 20 ms of round-trip time between sites, the 99th percentile rises from 51 to 130 ms, and the project advises against deploying across different regions. And raising the cache from 10,000 to 200,000 entries brought the database CPU peak down from 77.77 % to 63.77 %, in exchange for somewhat longer garbage collection pauses.

Security

The urgent matter, if the deployment is below 26.7.2:

CVE-2026-18963, score 9.1, critical. A bypass of the email action token in the credential reset flow allowed resetting any user’s password with no prior authentication, administrators included. Fixed in 26.7.2, released on 19 August 2026. The temporary mitigation, if you cannot upgrade, is to disable password recovery in all realms. No known exploitation was recorded as of the end of August.

In 26.7.3 there are another nineteen, among them the missing hostname verification in the TLS certificate towards LDAP with a score of 8.8, two tenant restriction bypasses in token exchange with Microsoft and Google brokers, and two access control bypasses in organisations.

Official hardening comes down to four decisions:

Strict hostname, which is the default value since the second generation of that configuration. The server does not infer the URL from headers, which prevents poisoning email links and redirects.
Administration console on a different hostname, and network-restricted.
Proxy headers declared explicitly, trusting only a controlled reverse proxy.
Separate management port, not exposed.

Part 8. The mapping to the ENS

The codes and titles below are verified against the official text of Royal Decree 311/2022 published in the BOE, Annex II.

Code	Literal title	What a central IdP contributes
op.acc.1	Identificación	A unique identifier per entity, user or process that gains access
op.acc.2	Requisitos de acceso	Permissions are granted by role and group, not by loose credential
op.acc.3	Segregación de funciones y tareas	It supports it with roles, but the control is organisational
op.acc.4	Proceso de gestión de derechos de acceso	Least privilege and a specific remote access policy
op.acc.5	Mecanismo de autenticación (usuarios externos)	Flows and factors configurable per flow
op.acc.6	Mecanismo de autenticación (usuarios de la organización)	Reinforcements required already at the low level
op.exp.8	Registro de la actividad	It contributes part of the events, not all of them
op.exp.10	Protección de claves criptográficas	Management and rotation of the realm signing keys

Two nuances an auditor is going to look for.

The first, about op.acc.1: the text requires every entity that accesses the system, whether user or process, to have a unique identifier. That includes agents. A virtual key shared between several agents does not comply, and that is exactly what usually happens when JWT authentication falls outside the budget.

The second, about op.exp.8: the control requires recording the identifier of the user or entity associated with the event, the date and time, what information it is performed on, the type and the result. Keycloak contributes the authentication events. The usage events are contributed by the gateway and the MCP servers, and there the two already documented gaps reappear: adding and removing MCP servers generates no audit log and the trace user is set by the client.

The correspondence between control and product in this table is my own interpretation and must be validated by the certification body. The codes and titles are not: those are literal from the BOE.

Part 9. The configuration to leave written down

Three concrete pieces, so as not to stay at the level of concepts.

Hardened startup, as container arguments:

kc.sh start --optimized \
--hostname=https://sso.ejemplo.es \
--hostname-admin=https://sso-admin.interna.ejemplo.es \
--proxy-headers=xforwarded \
--health-enabled=true \
--metrics-enabled=true \
--http-management-port=9000 \
--cache=ispn --cache-stack=jdbc-ping \
--db=postgres

The metrics, health, cache and database options are build options: they go into the image, and passing them here forces a rebuild if they do not match.

The client scope with audience, which is the workaround forced by the lack of resource indicators. Created with the administration tool:

# one scope per protected resource
kcadm.sh create client-scopes -r plataforma \
 -s name=mcp-inventario -s protocol=openid-connect \
 -s 'attributes."include.in.token.scope"=true'

# the mapper that sets the token audience
kcadm.sh create "client-scopes/<id>/protocol-mappers/models" -r plataforma \
 -s name=aud-mcp-inventario \
 -s protocol=openid-connect \
 -s protocolMapper=oidc-audience-mapper \
 -s 'config."included.custom.audience"=https://mcp.ejemplo.es/inventario' \
 -s 'config."access.token.claim"=true'

The client asks for that scope and receives a token whose audience is only that MCP server. The MCP server validates the audience and rejects any other token, as the specification requires.

Realm import via GitOps, with the operator’s custom resource, so that the configuration does not live only in the console:

apiVersion: k8s.keycloak.org/v2beta1
kind: KeycloakRealmImport
metadata:
 name: plataforma
 namespace: identidad
spec:
 keycloakCRName: sso
 realm:
 realm: plataforma
 enabled: true
 # without this, the recovery email is attack surface
 resetPasswordAllowed: false
 bruteForceProtected: true
 sslRequired: all

Part 10. What Keycloak does not solve

Fine-grained authorisation over resources. The question “can this person see this particular trace” is a relation between objects, not a role. That problem is solved by Zanzibar-style engines, with OpenFGA and its object, relation and user tuples, or a policy language such as Cedar. Keycloak gives roles and scopes, and with those you do not model a graph of relations.

Machine secrets. Storage credentials, provider keys, database passwords. That is a secrets manager, covered in stack hardening.

Cryptographic workload identity with attestation. That is SPIFFE and SPIRE. In Keycloak it exists as an identity provider in preview, which is a different thing.

Authorisation inside the model. Which MCP tool an agent may invoke, and with what arguments. The specification offers scopes, the insufficient scope error and step-up elevation, but the per-tool and per-argument decision is taken by the MCP server or a policy engine, not by the identity provider.

Antipatterns

Forwarding the user’s token from the gateway to third-party MCP servers. Explicitly forbidden by the specification.
Static client identifier with dynamic registration and no per-client consent, which is the exact recipe for the confused deputy.
General scopes, of the full-access kind, and publishing the entire catalogue of supported scopes in the metadata.
Treating a state handler as authentication, which the new revision expressly forbids.
Following OAuth discovery towards any URL, which opens the door to requests towards internal addresses. It also applies to the authorisation server when it downloads a client metadata document.
Trust headers badly exposed in chat front ends. The Open WebUI documentation warns that an incorrect configuration allows authenticating as any user.
Starting in development mode in production, with the hostname not resolved strictly.
A single shared static key as the only control between the gateway and the engine.

Startup checklist

Check the version. Below 26.7.2 there is a critical password reset bug.
Build the image with the build phase done and start with the optimised flag.
Strict hostname, administration console on another hostname, separate management port and not exposed.
Decide between organisations and one realm per tenant before creating the second tenant.
One client scope per resource, with an audience mapper, because resource indicators are not supported.
In the gateway, define the two audience and issuer variables. Without them none of that is validated.
For workloads inside the cluster, look at the Kubernetes identity provider before handing out client secrets.
Size by login peaks, not by number of services: the difference is eightfold.
Export the Keycloak events to the same place the gateway’s go, because the activity logging control needs them together.
Write into the design what the system does with a two-hour task, before someone solves the problem with an offline token.

Closing

Keycloak is a mature piece that solves one bounded problem well: knowing who the caller is, and saying so verifiably in a document that expires. In a sovereign inference platform that is worth a lot, because the alternative is an inventory of static keys spread across ten services that no audit can walk through.

What is not advisable is asking more of it. Fine-grained authorisation lives in each service, workload identity with attestation lives in SPIFFE, and the decision of which tool an agent may call lives in the MCP server.

And there is one concrete gap worth noting down, because it does not close on its own: the MCP specification marks two standards as mandatory, and of the two that would fall to an authorisation server, resource indicators are not implemented. The workaround with scopes and an audience mapper works, but it is a workaround, and whoever comes after will look for the standard parameter. Leaving it written in the design saves an afternoon.

Sources

Keycloak 26.7.3, 31 de agosto de 2026, y descargas.
Configuración del servidor, caché, bases de datos y nombre de host.
Integración con Model Context Protocol, intercambio de tokens y autenticación federada de clientes, 27 de enero de 2026.
Dimensionado de CPU y memoria e introducción a alta disponibilidad.
Bancos de pruebas de rendimiento de Keycloak 26.4, 1 de octubre de 2025.
Especificación de MCP, revisión 2026-07-28, autorización y buenas prácticas de seguridad.
KEP-3331, configuración estructurada de autenticación.
Real Decreto 311/2022, Anexo II.
Registro de CVE-2026-18963 en la base de datos de Red Hat y notas de la versión 26.7.3 en GitHub.
CNCF, proyecto Keycloak.

Langfuse v4, day 2 (1 of 10): the data model changed underneath, and what really goes into a trace

Sat, 12 Sep 2026 07:00:00 +0200

First article in a series of ten about operating Langfuse v4 in production. The architecture of version 3 and its ten performance levers were already covered in Langfuse from the inside; the seam with the gateway, in the operational pair. This series is about day 2. Verified against Langfuse 4.35.0, commit of 11 September 2026.

TL;DR

Version 4 is observations first. There is no trace table. A trace is a set of observations that share an identifier, and the trace attributes live denormalised in every row: name, user, session, tags, version, release and environment (clickhouse/migrations/canonical/0039_create_events_full.up.sql). The whole intuition of “first I create the trace and then I hang things off it” stops describing the storage.

There are ten observation types, not three. Alongside the classic SPAN, EVENT and GENERATION come AGENT, TOOL, CHAIN, RETRIEVER, EVALUATOR, EMBEDDING and GUARDRAIL (packages/shared/src/domain/observations.ts:5). Using the right type changes what the interface knows how to group.

The real limits are in the code and they do not match what people assume. The body of a request to the classic ingestion route is capped at 4.5 MB (api/public/ingestion.ts:40). The OTLP body, at 512 MiB, compressed and decompressed, with a 413 when it is exceeded. An OTLP span above 9.5 MB is only written to the log, not rejected. Identifiers accept 800 characters. The trace name, 1,000.

The ingester looks for the user in five places and the session in six. With strict precedence order, starting from Langfuse’s own attributes and ending in the metadata of other frameworks. Among the session ones sits gen_ai.conversation.id, which is the OpenTelemetry standard, so a conformant instrumentation works without translation.

Scores do not travel over OTLP. Zero occurrences of the word across the 3,864 lines of the OTLP ingestion processor. They come in through the classic ingestion route, through the scores endpoint or through the interface. Any design that assumes an evaluator can write its score down the same channel as the trace is wrong.

The classic masking function does not cover what almost everyone thinks it covers. It only acts on data that passes through the SDK calls. The attributes produced by a third-party automatic instrumentation come out unmasked. There is a new function that does act on the batch attributes, but it leaves out span events, which is where the recent conventions put the messages.

Filtering by tags or by status message is a scan. In the listings table, the indexed fields are the span identifier, the trace identifier, the user identifier, the session identifier, the dates, the model and the metadata names. The name, the tags, the version and the status message have no index.

You are here: OBSERVE, day 2

The previous series about this pair covered selection and setup. The six-service architecture explained version 3; tracing with OpenTelemetry explained the pipeline; the pair with the gateway explained where the trace gets lost between the two pieces.

What is left uncovered is the work of the next three years: migrating, sizing ClickHouse, retaining and deleting, taking backups, alerting, getting the data out. This series is about that, and it starts with the question that conditions everything else, which is what gets stored.

The analogy: the archive that changed system

An archive with a folder per case file and documents inside each folder is easy to understand. You open the folder, you look at what is there. The problem shows up when you have to find every document of one kind over the last two years: you have to open every folder.

Large archives solve that the other way around. There is no folder: each document carries, copied into its header, the case reference, the client name and the date. Searching by client is immediate. Reconstructing a case file is grouping by reference. In exchange, each document weighs a little more, and what gets written into that header matters a great deal, because everything you can search for comes from there.

Langfuse v4 made exactly that change. What used to be a folder with attributes of its own now travels repeated in the header of every document. The two operational consequences of this article come from there: which fields are worth filling in and which fields are not worth filling with content.

The data model, as it is in the code

The tables

Three new pieces in ClickHouse:

events_full (migration 0039): the table of record, with the full input and output, compressed with ZSTD level 3 and indexed with full-text indexes over the lowercase versions. ReplacingMergeTree engine ordered by event timestamp and deletion flag, partitioned by month of the start time.
events_core (migration 0040): the listings table, lighter.
events_core_mv (migration 0041): the materialised view that fills the previous one, and which truncates to two hundred characters the input, the output and each metadata value.

That truncation to two hundred characters is the first thing to know. What you see in a listing is not what is stored; it is a clipping. And when reading a full trace a second cap applies on top, LANGFUSE_SERVER_SIDE_IO_CHAR_LIMIT, with a default value of a thousand (packages/shared/src/env.ts:513).

In the events_full table there is a configuration line that says a lot: index_granularity_bytes = '64Mi', with the comment that it avoids very small granules because of large rows. It is the schema itself acknowledging that the rows of this system are bulky.

The ten observation types

SPAN, EVENT, GENERATION, AGENT, TOOL, CHAIN, RETRIEVER, EVALUATOR, EMBEDDING and GUARDRAIL (domain/observations.ts:5), with their corresponding ingestion events (server/ingestion/types.ts:279).

For a platform with agents, the difference between marking a tool call as a generic SPAN or as a TOOL is the difference between being able to count tool calls per session and having to reconstruct it by hand. It is worth reviewing the instrumentation with this list in front of you.

The level is still DEBUG, DEFAULT, WARNING and ERROR (domain/observations.ts:31), with a free-form status message.

The limits, with their source

Field	Limit	File
Trace, span and observation identifiers	800 characters, no carriage return	`ingestion/types.ts:10`
Trace name	1,000 characters	`ingestion/types.ts:428`
Environment	40 characters, pattern of lowercase, digits, hyphen and underscore	`ingestion/types.ts:226`
Text-type score	500 characters	`domain/scores.ts:44`
Score config name	35 characters	`domain/score-configs.ts:11`
Classic ingestion body	4.5 MB per request	`api/public/ingestion.ts:40`
OTLP body	512 MiB, adjustable	`web/src/env.mjs:583`
OTLP span considered large	9.5 MB, only logged	`packages/shared/src/env.ts:193`
Name, status message, version, release, tags	no declared limit	`ingestion/types.ts:424`
Metadata	no declared limit	`ingestion/types.ts:425`

The last two are the dangerous ones, because they have no schema cap and the real ceiling ends up being the one on the request body. A three-megabyte metadata value passes validation and reaches ClickHouse.

A note on the langfuse prefix in the environment field: it is reserved for the product’s own internal traces and is stripped idempotently from public inputs (ingestion/types.ts:228), including the case of someone trying to sneak it in duplicated. The default value is default.

On rate limits: in the cloud there are per-plan quotas, but in self-hosted they are not applied. The service returns directly that there is no limiting (features/public-api/server/RateLimitService.ts). It is good news for bulk exports and bad news for anyone relying on that mechanism as protection.

The two instrumentation routes

What gets lost over OTLP, and it is one specific thing

The check is simple and the result is blunt: there is not a single occurrence of the word score in the OTLP ingestion processor, a file of 3,864 lines. Scores only come in through three places: the score-create event of the classic ingestion route, the POST /api/public/scores endpoint, or the interface.

This has a design consequence worth being clear about before building evaluators: the trace can travel through the OpenTelemetry collector with everything good that brings, on-disk queue, retries and sampling, but the score an evaluator puts on it has to go down a different channel and with the project keys.

Everything else does travel over OTLP. Whatever is not in the mapping lists is kept as metadata converted to a string, visible but outside the indexed fields.

The precedence chain, which is the map you need to have

Verified in packages/shared/src/server/otel/OtelIngestionProcessor.ts:

User identifier (line 2545), in this order: langfuse.user.id, user.id, the observation metadata, the trace metadata, and lastly the Vercel SDK metadata.

Session identifier (line 2565): langfuse.session.id, session.id, gen_ai.conversation.id, observation metadata, trace metadata, Vercel metadata.

Environment (line 2383): langfuse.environment, deployment.environment.name, deployment.environment; for each key the span attributes are checked first and the resource attributes after.

Model (line 2753): langfuse.observation.model.name, gen_ai.response.model, ai.model.id, gen_ai.request.model, llm.response.model, llm.model_name, model.

Level (line 1250): the native attribute, with accepted aliases where SUCCESS and OK become DEFAULT, WARN becomes WARNING and FATAL or CRITICAL become ERROR; if there is no attribute and the span status is an error, the result is ERROR.

Tags (line 3227): langfuse.trace.tags, langfuse.tags, observation and trace metadata, Vercel metadata, and tag.tags.

Input and output (line 1790): a chain of some twenty frameworks in strict order, starting from Langfuse’s own attributes and going through Genkit, the Vercel SDK, the standard inference operation details event of the conventions from 1.37 onwards, the per-role message events, the prompt and completion attributes, Logfire, MLflow, TraceLoop, OpenInference and the gen_ai.input.messages and gen_ai.output.messages attributes.

That this chain has twenty entries says something about the real state of the conventions: it was already noted that the generative AI conventions were extracted into their own repository without any published version. While that remains the case, the backend has to guess, and it guesses with a precedence list.

The practical part: if Langfuse’s own attributes are emitted, they always win. If you want backend-neutral instrumentation, gen_ai.conversation.id and deployment.environment.name are the two standard attributes that Langfuse does understand.

Metadata, and how it ends up in the table

A metadata value is flattened into two parallel arrays, one of names and one of values, with dot-separated paths and everything converted to a string (otel/utils.ts:239, applied in worker/src/services/IngestionService/index.ts:363). A nested object with twenty keys is twenty entries in each array.

There is a cap of 10,001 positions when reconstructing arrays from OTel attributes (OtelIngestionProcessor.ts:271), which is high but it exists.

Sessions, users and cardinality

A session is a text column with a Bloom filter index at 1 %. The same goes for the user. The Postgres table that accompanies sessions stores only whether it is bookmarked, whether it is public and its environment; there is no counter and no cap. There is no limit on traces per session in the code, and the documentation recommends session identifiers below 200 ASCII characters.

The cardinality difference matters and it is not intuitive: user and session are plain text, so high cardinality does not break the schema, it only inflates the Bloom filters. The environment, by contrast, is declared low cardinality. Generating a dynamic environment value per tenant degrades ClickHouse measurably.

Aggregated cost comes out of a cost details map with materialised input, output and total columns. The classic sessions endpoint is deprecated; the replacement is the version 2 observations one with a filter.

Scores: five types and three sources

The types are NUMERIC, CATEGORICAL, BOOLEAN, CORRECTION and TEXT (domain/scores.ts:45). The boolean requires a value of zero or one. The correction does not accept an associated config. The text one is capped at 500 characters.

The sources are API, EVAL and ANNOTATION. The EVAL source is reserved for the internal evaluators: the public API only accepts the other two. Annotation scores require an associated config except for the correction one.

On what costs money, a clarification that corrects a widespread belief: in 4.35.0 the entitlement for the number of model-based evaluators is unlimited on every plan, including the open edition (features/entitlements/constants/entitlements.ts:60). What is limited by plan is the number of annotation queues and the days of data access. And there is a version nuance: trace-level evaluators are discontinued in version 4.

Masking, and what it does not cover

This is the part to read twice if the deployment has to comply with anything.

The classic Python SDK function has the signature masking_function(*, data, **kwargs) and covers only the data that passes through the SDK calls: observation creation, update, and setting the trace input and output. It does not cover the raw OTel attributes produced by a third-party automatic instrumentation.

The replacement is a function that acts on the attributes and the resource attributes in the export batch. That one does reach automatic instrumentation, with a limitation declared in the documentation: it cannot change the span name, the identifiers, the parent relationship, the resource attributes, the events or the links.

And there is the gap that matters: the generative AI conventions from 1.37 onwards put messages and responses into span events. That is, the route by which prompts arrive in a modern instrumentation is precisely the one the new masking does not touch.

For a deployment with personal data, the conclusion is that client-side masking is not enough and a redaction processor has to go into the collector. It is the same reasoning that led in the previous article to recommending the collector in the middle, and it adds to the already documented problem of MCP tool arguments being written in clear, bypassing the message redaction switch.

On the server side there is no safety net: Langfuse has no server-side personal data masking. The encryption key that gets configured protects secrets, that is, model provider credentials, integration headers and SSO configurations. Trace inputs and outputs are not encrypted at application level.

Volume control

Three levers, in order of how high up they act:

In the SDK: sample_rate, or the LANGFUSE_SAMPLE_RATE variable, with head sampling at trace level. If the trace is not sampled, neither its observations nor its scores travel. LANGFUSE_TRACING_ENABLED set to false switches the whole thing off.

In the collector: probabilistic sampling and tail sampling. It is the only layer where you can decide based on the outcome, that is, keep every error and a fraction of the rest.

In the server: a deterministic per-project sampling, with a SHA-256 hash of the trace identifier (server/ingestion/sampling.ts). Deterministic per trace, so it does not cut traces in half.

What not to put into a trace

With the schema in front of you, the list is concrete.

Do not filter by tags or by status message. In the listings table the indexed fields are the span identifier, the trace identifier, the user identifier, the session identifier, the creation and update dates, the model name, the experiment identifier and the metadata names. The name, the tags, the version, the release and the status message are not. Filtering by them scans.

Do not use metadata as a store. It gets flattened into string arrays, truncated to two hundred characters in the listings table but stored whole in the full table, and on top of that it is indexed with a full-text index segmented on non-alphabetic characters. Bulky metadata multiplies the inverted index, not just the data.

Do not store full prompts without thinking about it. The input and the output carry ZSTD level 3 compression and two full-text indexes over their lowercase versions. Storing the entire context of a 142,000-token agentic call on every turn inflates the inverted index disproportionately. Client-side truncation is the cheap lever.

Do not use the environment to separate tenants. It is low cardinality, it has a 40-character cap and it isolates nothing: same project, same keys, same limits, same retention. It is there to segment dashboards. Real isolation means separate projects, as was already argued in the article on the seams.

Version 3 and the calendar date

The constant is in the code: V3_SUNSET_DATE = "2026-11-16" (features/public-api/server/deprecations.ts:11).

It is worth being precise about what it means, because the public documentation and the code do not say exactly the same thing. The code comment is literal: the classic ingestion route is never switched off and continues to accept score events; trace and observation events fail only in version 4’s exclusive write mode, not in the mixed or classic modes. And when they fail, they fail per event, not per batch.

The x-langfuse-ingestion-version: 4 header activates direct writing to the events table, which is the real-time route. Without it the deferred path is taken. Recent SDKs send it by themselves. Any value above four is rejected.

As for names, trace-level input and output attributes are deprecated in favour of the observation ones on the root span. The classic read endpoints carry a deprecation marker in the response body, and their replacements are version 2 of observations, version 3 of scores and version 2 of metrics. And dataset runs are renamed experiments.

There is a curious piece that deserves a mention because it shows up in the Redis dashboards and is misleading: a queue called version 4 classic API usage, which ingests nothing. It is a periodic task every fifteen minutes that scans the ClickHouse query log to detect which projects are still using the old API, and materialises the result into Redis for the interface’s migration notice.

How this looks in code

Two pieces, the client one and the collector one.

On the application side, what changes compared with the usual instrumentation is the observation type and where the identity comes from:

from langfuse import get_client, propagate_attributes

lf = get_client()

# identity propagates by context, not observation by observation
with propagate_attributes(
 session_id=id_conversacion, # the same across every turn
 user_id=sujeto_del_token, # from the claim, not from the body
 environment="produccion",
 tags=["soporte", "n2"],
):
 # the type matters: the interface groups by it
 with lf.start_as_current_observation(
 as_type="agent", name="resolver-incidencia"
 ) as raiz:

 with lf.start_as_current_observation(
 as_type="tool", name="buscar_en_inventario"
 ) as herramienta:
 resultado = mcp.call("buscar", args)
 # truncate here, not on the server
 herramienta.update(output=str(resultado)[:4000])

 # the score goes down its own channel, never over OTLP
 lf.score_current_trace(name="resuelta", value=1, data_type="BOOLEAN")

The as_type parameter accepts nine literals: span, generation, embedding, agent, tool, chain, retriever, evaluator and guardrail. The tenth type in the model, the event, has its own constructor because it does not open a scope.

On the collector side, the redaction processor that compensates for what the SDK masking does not reach:

processors:
 redaction:
 allow_all_keys: true
 blocked_values:
 - '\b\d{8}[A-HJ-NP-TV-Z]\b' # national ID number
 - '\b[\w.+-]+@[\w-]+\.[\w.]+\b' # email
 summary: silent
 tail_sampling:
 policies:
 - name: errores
 type: status_code
 status_code: {status_codes: [ERROR]}
 - name: resto
 type: probabilistic
 probabilistic: {sampling_percentage: 5}

exporters:
 otlphttp/langfuse:
 endpoint: http://langfuse-web.observabilidad.svc:3000/api/public/otel
 headers:
 Authorization: "Basic ${LANGFUSE_BASIC}"
 x-langfuse-ingestion-version: "4"
 sending_queue:
 enabled: true
 storage: file_storage
 retry_on_failure:
 enabled: true

The endpoint carries the base path without the signal suffix: the exporter adds its own. And the on-disk queue is the main reason for putting the collector in the middle, because the SDK’s batch processor drops when it fills up.

Instrumentation checklist

Review the type of every observation against the list of ten. Tool calls as TOOL, agents as AGENT, guardrails as GUARDRAIL.
Decide where the user identifier comes from and set it in a single place, high up the precedence chain.
Emit gen_ai.conversation.id if you want neutral instrumentation, or Langfuse’s own attribute if coupling does not matter.
Set the environment with deployment.environment.name, with few distinct values and never one per tenant.
Get scores out through their own channel, with the project keys, and do not expect them to travel with the trace.
Truncate input and output on the client before sending them, especially with long agentic contexts.
Put a redaction processor in the collector, because the SDK masking leaves out span events.
Do not filter by tags in dashboards that get queried often.
Send the ingestion version header, or use recent SDKs that send it by themselves.
Review the size of the metadata: there is no schema cap, the cap is the one on the request body.

Traps and things that are not what they look like

What you see in a listing is truncated to two hundred characters by the materialised view, and the full read has another cap of a thousand. The data is whole in the full events table.

Metadata has no size limit in the schema. A blob of megabytes passes validation.

An OTLP span over 9.5 MB is not rejected, it is only logged. There is no signal back to the client.

The environment is declared low cardinality. One value per tenant degrades ClickHouse.

Scores do not travel over OTLP. It is by absence in the code, not by configuration.

The classic masking does not cover automatic instrumentation, and the new one does not cover span events, which is where the recent conventions put the messages.

Langfuse does not encrypt trace inputs or outputs. The encryption key protects integration credentials.

In self-hosted there are no rate limits on the API. It is no use as protection.

The classic ingestion route is not switched off on 16 November, it changes behaviour depending on the write mode, and it fails per event instead of per batch.

Model-based evaluators are not limited by plan in this version, contrary to what is usually assumed. What is limited are the annotation queues and the days of data access.

The series: the ten articles

This is the first. The order is designed so that each one can be read on its own and so that together they cover the whole of day 2.

What goes into a trace (this article): version 4’s data model, limits, precedences, scores, masking and indexes.
Putting LangGraph in front: instrumenting an agentic platform and the measured cost of one turn.
Two well-known agents instrumented: Open Deep Research and GPT Researcher, with the measured figures of a real request.
Migrating from version 3 to 4 with no window: the three write-mode steps, the resumable background migrations and where the rollback point of no return sits.
The worker queues: the map of all thirty-nine, which pool to dedicate to each group, the per-queue switches, sharding and concurrency.
Real ClickHouse capacity and cost: how to measure bytes per observation with the system tables, the difference between the full table and the listings table, and the merge cost of full-text indexes.
Retention, deletion and data protection: why a deletion does not free disk, the mask cleaner that ships disabled, the pending deletion queue and the S3 lifecycle that has to be implemented by hand.
Backup and cross recovery: restore ordering across Postgres, ClickHouse and object storage, what each mismatch breaks, and how far event replay goes.
Saturation runbook: what to alert on from the queue metrics, the stuck probes, draining through the readiness endpoint and the dead letter queue.
Getting the data out: the blob storage integration to Parquet, batch exports and the metrics API, to build the data lake.

Sources

Código de Langfuse 4.35.0, commit 39e3cd7d del 11 de septiembre de 2026.
Anuncio de Langfuse v4, 17 de agosto de 2026.
Compatibilidad y migración, consultado el 12 de septiembre de 2026.
Sesiones, muestreo y enmascarado, consultados el 12 de septiembre de 2026.
Integración nativa con OpenTelemetry, consultado el 12 de septiembre de 2026.

The gateway does not live alone: the JWT that does not validate the audience, the user who never reaches the trace, and the seams you have to write by hand

Sat, 12 Sep 2026 06:30:00 +0200

Eighth article in the operational track of the control layer. The previous seven deal with the gateway from the inside: the pair with Langfuse, day 2, virtual keys, humans and agents, prefix routing, the MCP gateway and sizing for agents. This one deals with what surrounds it. Verified against LiteLLM 1.102.0 and Langfuse 4.35.0, both with a commit of 11 September 2026.

TL;DR

The gateway has five neighbours: a database, a cache, an identity provider, a trace backend and a collector. Each neighbourhood has its own physics, and these are the seven things it decides.

The proxy starts without Postgres, but then only the master key works. The DATABASE_URL variable is optional (proxy_server.py:1163); without it, any key other than the master one returns a 400 for a missing connection (user_api_key_auth.py:1886). With a database configured but down, the default behaviour is to return 503, unless general_settings.allow_requests_on_db_unavailable is switched on.

Redis is never mandatory, and that is precisely the trap. There is a circuit breaker with a threshold of five failures and a recovery of sixty seconds (constants.py:456). When it opens, the proxy fails no request: it degrades to local memory. What is lost silently is that rate limits stop being global and become per pod.

No observability backend can bring a request down. All callbacks are launched with asyncio.create_task and their exceptions are caught and logged (litellm_logging.py:3235). If Langfuse does not respond, events are discarded as the queue fills up. There is no added latency, and there is no reliable record either: the pair’s observability is a statistical figure, not an audit record.

JWT authentication validates neither the audience nor the issuer unless two variables are defined. In _build_decode_kwargs (handle_jwt.py:1003) the audience comes from JWT_AUDIENCE and the issuer from JWT_ISSUER; if they are empty, the code sets verify_aud=False and verify_iss=False and emits a warning once only. Any valid token signed by that provider, issued for another service, is accepted.

Role-based access control ships disabled. enforce_rbac has a default value of False (_types.py:4848 onwards). With that value, a token whose role resolves to nothing does not produce a 403: it carries on through the flow. Real authorisation, then, comes from team membership, if there is a team claim configured at all.

The user identifier that reaches Langfuse is not the one from Keycloak. The user_id field of the trace is filled in with user_api_key_end_user_id (integrations/langfuse/langfuse.py:604), that is, with the user field of the OpenAI body, which the client sets. The subject of the token and the owner of the key do not travel. Anyone who wants per-person traceability needs to configure end_user_id_jwt_field or make the client send that field.

Langfuse provisions nothing from a claim. There is no variable that maps an OIDC claim to an organisation or to a project. The only thing available is static assignment at sign-up with LANGFUSE_DEFAULT_ORG_ID and LANGFUSE_DEFAULT_PROJECT_ID, the same for everyone. On top of that, per-project access control sits behind the Enterprise licence in the self-hosted edition, as do audit logs and per-project retention.

You are here: between the four pieces

The LiteLLM and Langfuse pair walked through the seam between the gateway and the trace backend: integration routes, cost that arrives as zero, correlation and queues with discarding. This article widens the focus to the other three neighbourhoods, with identity at the centre, because that is the one that orders everything else and the one an auditor asks about.

The identity piece itself, that is, Keycloak, is dealt with in the next article. Here we deal with the gateway side.

The analogy: the building site with five trades

On a small building site, the problem is rarely inside each trade’s work, it is at the joins: where the plumber leaves the pipe sleeve and the bricklayer closes up the partition wall, where the electrician wants to run through the same gap. Each trade does its part well and the join ends up badly resolved because it belongs to nobody.

An inference platform has five trades. The gateway knows about tokens, the identity provider knows about people, the trace backend knows about events, the database knows about spend and the cache knows about counters. Each one has good documentation. The joins have nobody’s documentation, and that is where the user’s identity gets lost between the token and the trace.

There is one more detail of the analogy that is useful. On a building site, the part that causes the most problems is not the one that looks badly finished, it is the one that gets covered up. The same happens here: the five seams in this article fail silently.

Seam 1. What brings things down and what degrades them

It pays to have written down what happens when each neighbour falls over, because intuition fails in almost every case.

Neighbour	Does it start without it?	If it falls over
Postgres	Yes, master key only	503 by default; degraded with `allow_requests_on_db_unavailable`
Redis	Yes	Circuit breaker after 5 failures; limits become per pod
Langfuse	Yes	Events discarded as the queue fills; no impact on latency
OTel collector	Yes	Same as the previous one
Identity provider	Yes	Access to the interface and the JWT flow goes down; virtual keys carry on

Postgres

Without DATABASE_URL, _setup_prisma_client returns None without error (proxy_server.py:10375). The proxy serves /chat/completions with the master key. Any other key gets a 400. There is no global budget either, and start-up itself warns about it: Redis does not replace the database (proxy_server.py:9240).

With a database configured and down, the exception handler re-raises unless the degradation flag is active (db/exception_handler.py:56). There is also a DISABLE_PRISMA_HEALTH_CHECK_ON_STARTUP and a watcher that reconnects. The detail that matters for diagnosis: a transient database failure returns 503, not 401. If during an incident you see 401s, the problem is credentials; if you see 503s, it is the database or admission.

Redis

The circuit breaker is enabled by default, with a threshold of five failures, a recovery of sixty seconds and a minimum duration of five (constants.py:456, logic in caching/redis_cache.py:153). The socket has a timeout of 0.1 seconds (constants.py:453). With the circuit open, DualCache serves from memory without propagating an exception (dual_cache.py:215).

The consequence has to be measured, not assumed: with four workers and three replicas, a limit of one hundred requests per minute becomes one thousand two hundred. It is a failure mode that generates no errors and that can last for days without anyone noticing, until the bill arrives or the engine saturates.

The right alert is not about Redis, it is about litellm_service_latency labelled by service (integrations/prometheus_services.py:119), which exposes Redis and database latency separately.

The callbacks

Everything behind, that is, Langfuse and the collector, is launched outside the response path and its failures are caught. The classic integration uses the v2 SDK with its own background thread and a flush interval of one second, adjustable with LANGFUSE_FLUSH_INTERVAL (langfuse.py:198). There is a cap of fifty instantiated Langfuse clients (constants.py:554), because each client is a thread.

The OTLP integration uses the batch processor of the OpenTelemetry SDK (opentelemetry.py:3060) with its default values. In both cases, saturation means discarding, and discarding generates no error towards the client.

Seam 2. Identity: the three doors and what each one validates

Everything goes through a single authentication constructor (user_api_key_auth.py:1245), in this order:

Custom authentication from the enterprise edition.
Public routes.
Opaque OAuth2, if enabled, with a licence check.
JWT, if general_settings.enable_jwt_auth is active and the token has three parts.
Master key, with constant-time comparison.
Virtual key, with SHA-256 hash and lookup in the verification table.

A JWT works for /chat/completions, not just for administration: the routes allowed by default for a team include openai_routes (_types.py:4886), which contains the chat endpoint. Administration, on the other hand, is only reachable by a token with an administrator role.

The first thing to know: it is paid for

The JWT branch carries an explicit licence check, with the literal message that JWT authentication is an enterprise-only feature (user_api_key_auth.py:1416). SSO for the administration interface is free up to five billable users and requires a licence above that (ui_sso.py:976).

This conditions the architecture of any sovereign deployment that does not want to pay for a licence: the Keycloak integration is left for the administration interface and for provisioning, while inference traffic authenticates with virtual keys. Virtual keys are not worse, but they are a different thing, and it has to be said in the risk analysis: they are credentials decoupled from the identity provider, so removing a person from Keycloak does not invalidate their key.

The `litellm_jwtauth` fields that decide security

From the LiteLLM_JWTAuth class (_types.py:4848 onwards), these are the ones that matter, with their default values:

Field	Default	What it does
`enforce_rbac`	`False`	If false, an unresolved role does not produce a 403
`enforce_scope_based_access`	`False`	Without this, `scope_mappings` is not applied
`enforce_team_based_model_access`	`False`	Model access by team
`team_id_jwt_field`	`None`	Claim the team comes from
`team_id_upsert`	`False`	If false, a team that does not exist gives a 404
`user_id_upsert`	`False`	Same with the user
`public_key_ttl`	`600`	Cache of the public keys
`public_key_stale_ttl`	`3600`	Extra margin if the provider is down
`admin_jwt_scope`	`litellm_proxy_admin`	Scope that grants administration
`user_allowed_email_domain`	`None`	Restriction by email domain

The accepted algorithms are RS256/384/512, PS256/384/512, ES256/384/512 and EdDSA (handle_jwt.py:163). There is no symmetric one, which eliminates the algorithm confusion class of attack. It is a good design decision and it deserves recognition.

Key download supports several comma-separated URLs in JWT_PUBLIC_KEY_URL, resolves discovery if the URL points to the well-known OpenID configuration, retries three times with increasing waits and memoises the failure for thirty seconds (handle_jwt.py:680, :751, :88). If the provider is down, it serves the stale copy up to the sum of the two TTLs, that is, up to seventy minutes. For availability that is fine; for revocation it is not, and it has to be written down.

One detail of key rotation: with more than one key in the set, an exact match of the key identifier is required, but with a single key and no identifier it is accepted without checking (handle_jwt.py:913).

The validation that does not happen

This is the part to fix on day one. In _build_decode_kwargs (handle_jwt.py:1003), the audience is taken from the JWT_AUDIENCE environment variable and the issuer from JWT_ISSUER. If they are not defined, the code sets verify_aud=False and verify_iss=False, and emits a warning once only.

What that means in a real deployment with Keycloak: a token issued for the Grafana client, or for the Backstage one, signed by the same realm, is accepted by the gateway as a valid credential. Expiry is always validated, with zero margin. So is the signature. The audience is not.

There is a second route, issuers, with per-issuer configuration, which does validate issuer and audience unless explicitly disabled (handle_jwt.py:1186). That is the one to use when there is more than one provider.

The minimum fix is two lines of environment:

JWT_AUDIENCE=litellm-gateway
JWT_ISSUER=https://sso.ejemplo.es/realms/plataforma

And on the Keycloak side, an audience mapper in a dedicated client scope that injects that audience. The detail is in the Keycloak article, because the standard that would resolve this cleanly, resource indicators, is not supported.

From claim to permission

The administrator role comes from admin_jwt_scope appearing in the scope claim, and it returns a result without going through a team (handle_jwt.py:309, :1373). The rest is resolved by team.

If the team in the claim does not exist in the database, with team_id_upsert false a 404 is returned with the message that the team has to be created (auth_checks.py:3035). With the flag active, it is created by invoking team creation with a synthetic administrator identity (auth_checks.py:2874). That is the only automatic provisioning route that exists from Keycloak towards the gateway, and it works well. It pays to know that it creates teams with no budget and no limits, so a default value is needed in the configuration.

The vulnerability worth knowing about

CVE-2026-35030, scoring 9.1 on CVSS 3.1 and 9.4 on CVSS 4.0, advisory GHSA-jjhc-v7c2-5hh6. With JWT authentication active, the cache of the OIDC user info endpoint used the first twenty characters of the token as the key, so that two tokens with the same prefix were confused with each other. Fixed in 1.83.0.

In 1.102.0 the code uses the full SHA-256 hash (handle_jwt.py:962). A search for similar patterns across the tree finds no authentication cache indexed by token prefix: the truncations that remain are masked for logging. And it is worth recalling what has already been published about the MCP gateway: below 1.83.14 there is unauthenticated remote execution or SQL injection.

Seam 3. Identity does not reach the trace

Here lies the most expensive design flaw in the arrangement, because it is discovered at the first audit.

The user_id field of a Langfuse trace is filled in with user_api_key_end_user_id (integrations/langfuse/langfuse.py:604 and :724), which comes from the user field of the OpenAI request body or from end_user_id_jwt_field (litellm_pre_call_utils.py:1584). It is not the subject of the token nor the owner of the virtual key.

The knock-on consequences:

If the client does not send user, the trace has no user.
If the client does send user, the trace has whatever value the client cares to put there, unverified.
An agent calling with a service key leaves traces with no person behind them.

There are three ways to fix it, in order of robustness:

Configure end_user_id_jwt_field so that the end user comes from a token claim and not from the body. It is the only option in which the value is not controlled by the client, and it requires JWT authentication, that is, a licence.
Force the trace_user_id metadata field, which overwrites the field (langfuse.py:726; on the OTLP route, langfuse_otel.py:96). It can be set per team.
Contractually require the client to send user and validate it in a guardrail. It works, but it is a client declaration, not a proof.

Something similar happens towards the engine. The caller’s identity does not travel to vLLM unless litellm.add_user_information_to_llm_headers is switched on, whose default value is None (litellm/__init__.py:223). With it, x-litellm-* headers are injected with user identifier, team identifier and key hash (litellm_pre_call_utils.py:1356). Forwarding client headers requires general_settings.forward_client_headers_to_llm_api, and the authorization header is always filtered out, which is correct.

Towards MCP servers, by default the server’s own credential is used, not the user’s token. Only two modes forward the caller’s token, and doing so has a name in the MCP specification: token passthrough, which is explicitly forbidden. It comes up again in the Keycloak article.

Seam 4. Langfuse as a neighbour

What can be automated and what cannot

The exact list of variables for SSO in Langfuse 4.x, taken from web/src/env.mjs:

With the Keycloak provider: AUTH_KEYCLOAK_CLIENT_ID, AUTH_KEYCLOAK_CLIENT_SECRET, AUTH_KEYCLOAK_ISSUER, AUTH_KEYCLOAK_ALLOW_ACCOUNT_LINKING, AUTH_KEYCLOAK_CLIENT_AUTH_METHOD, AUTH_KEYCLOAK_CHECKS, AUTH_KEYCLOAK_ID_TOKEN_SIGNED_RESPONSE_ALG, AUTH_KEYCLOAK_SCOPE, AUTH_KEYCLOAK_ID_TOKEN, AUTH_KEYCLOAK_NAME.

With generic OIDC: the same set with the AUTH_CUSTOM_ prefix, plus AUTH_CUSTOM_FETCH_USERINFO and claim mapping with LANGFUSE_CUSTOM_SSO_SUB_CLAIM, LANGFUSE_CUSTOM_SSO_EMAIL_CLAIM, LANGFUSE_CUSTOM_SSO_NAME_CLAIM and LANGFUSE_CUSTOM_SSO_IMAGE_CLAIM.

Global ones: AUTH_DISABLE_USERNAME_PASSWORD, AUTH_DISABLE_SIGNUP, AUTH_DOMAINS_WITH_SSO_ENFORCEMENT, AUTH_SESSION_MAX_AGE.

And now what is missing. There is no variable that maps a claim to an organisation or a project. The only thing available is static assignment at sign-up with LANGFUSE_DEFAULT_ORG_ID, LANGFUSE_DEFAULT_ORG_ROLE with a default value of VIEWER, LANGFUSE_DEFAULT_PROJECT_ID and LANGFUSE_DEFAULT_PROJECT_ROLE (features/auth/lib/createProjectMembershipsOnSignup.ts:61). The documentation itself confirms it: enterprise SSO does not provision roles automatically at sign-up.

On top of that, in the per-edition permissions table (features/entitlements/constants/entitlements.ts:139), the open and self-hosted professional editions do not include per-project access control or the administration API. They belong to the enterprise edition, as do audit logs and per-project retention. In the open edition, every user inherits the role of their organisation.

The declarative bootstrap, which does exist

For on-premise without a licence, the workable route is bootstrapping through variables (web/src/initialize.ts), which is idempotent: LANGFUSE_INIT_ORG_ID (mandatory, if missing the rest is ignored with a warning), LANGFUSE_INIT_ORG_NAME, LANGFUSE_INIT_PROJECT_ID, LANGFUSE_INIT_PROJECT_NAME, LANGFUSE_INIT_PROJECT_RETENTION, LANGFUSE_INIT_PROJECT_PUBLIC_KEY, LANGFUSE_INIT_PROJECT_SECRET_KEY, LANGFUSE_INIT_USER_EMAIL, LANGFUSE_INIT_USER_NAME, LANGFUSE_INIT_USER_PASSWORD.

With that, one project per tenant is created from GitOps with one initialisation container per tenant and the keys in Kubernetes secrets.

Langfuse credentials per team

This part is well resolved on the gateway side and few people use it. TeamCallbackMetadata.callback_vars (_types.py:2152) accepts langfuse_public_key, langfuse_secret_key, langfuse_host and langfuse_environment, with an allow list in initialize_dynamic_callback_params.py:67. The gateway caches one logger per credential set (langfuse_handler.py:24) and, on the OTLP route, builds a full exporter per key (langfuse_otel.py:400).

That is to say: one Langfuse project per LiteLLM team is possible today, without a licence, and it is the correct way to isolate traces between tenants. What does not exist is automatic project creation, nor the correspondence between the team and the project. That is glue you have to write.

The ingestion route

Langfuse accepts only OTLP over HTTP, in http/protobuf or http/json. gRPC is not supported. The base route is /api/public/otel and the signal lives at /api/public/otel/v1/traces, verified in the repository. Authentication is Basic with the project’s key pair, plus the x-langfuse-ingestion-version: 4 header for the version 4 route.

The gateway builds it exactly like that (langfuse_otel.py:328, :347, :359) and then normalises the endpoint by adding the signal suffix (opentelemetry.py:3237). The result matches the real route, so the integration works without touching anything. The detail matters when a collector is put in the middle and somebody copies the endpoint by hand.

And the calendar warning that already appeared in the operational pair still stands: the classic callback is tied to the v2 SDK and the date marked is 16 November 2026. With a nuance worth reading before scheduling an emergency migration, which is detailed in the data model article: that date applies to Langfuse Cloud, and in the code of the self-hosted version the old route is not switched off, it changes behaviour according to the write mode.

When to put a collector in

Direct if Langfuse is the only trace destination. Via a collector if there is more than one consumer, or if any of these three things is needed that the proxy’s batch processor does not provide:

Retries with increasing waits and an on-disk queue, with the sending_queue and file_storage processors. With that, a Langfuse maintenance window stops losing traces.
Tail sampling, which lets you keep 100 % of errors and 1 % of the rest. The proxy cannot sample by outcome, because at the point it decides it does not yet know how things ended.
Attribute redaction before they leave the namespace. It is the only layer where you can delete the MCP tool arguments which, as has already been documented, are written in the clear, bypassing the message redaction switch.

Seam 5. Postgres, Redis and the memory policy

Separate Postgres, always. Each piece applies its own Prisma migrations over the public schema of the database it is given. Two Prisma clients migrating the same schema collide over table names and over migration history. On top of that, Langfuse 4.x requires Postgres 15 as a minimum, recommends 16 and wants UTC by default. The load profile is also the opposite: the gateway writes spend records at high frequency, while Langfuse does light transactional work because trace data goes to ClickHouse.

Separate Redis, and the argument is not the one it looks like. There is no key collision by design: the gateway uses hash tags of the form {api_key:...}:requests and an optional namespace, and Langfuse uses the BullMQ prefix plus its own prefix for its key cache. The decisive argument is another one: Langfuse requires maxmemory-policy=noeviction because its queues are data, not cache, while the gateway assumes a disposable cache. Sharing an instance means that a spike of rate limit keys can fill the memory and cause write errors in the ingestion queues instead of a harmless eviction.

If it is shared anyway, two minimum measures: REDIS_KEY_PREFIX in Langfuse, namespace in the gateway, and different logical databases.

And a reminder about surface area: Langfuse 4.x does not start without ClickHouse or without an S3-compatible bucket. CLICKHOUSE_URL and LANGFUSE_S3_EVENT_UPLOAD_BUCKET are mandatory and have no default value (packages/shared/src/env.ts:120 and :271). Its failure surface is far larger than the gateway’s, and that changes where to put the high availability effort.

Ports, for the NetworkPolicy

Piece	Port
LiteLLM (API, interface and metrics)	4000
Langfuse web	3000
Langfuse worker	3030
ClickHouse	8123 HTTP, 9000 native
S3-compatible storage	9000
Redis	6379
Postgres	5432
OTel collector	4317 gRPC, 4318 HTTP

The pairs to allow under a default-deny policy, following the method from hardening the stack:

litellm → postgres:5432, redis:6379
litellm → collector:4318 (or langfuse-web:3000 if going direct)
litellm → vllm:8000
litellm → keycloak:8443 (interface and JWT flow only)
collector → langfuse-web:3000
langfuse-web → postgres:5432, redis:6379, clickhouse:8123, s3:9000
langfuse-worker→ postgres:5432, redis:6379, clickhouse:8123, s3:9000
langfuse-web → keycloak:8443
ingress → litellm:4000, langfuse-web:3000
prometheus → litellm:4000

The Langfuse worker needs no inbound traffic beyond the health probe.

The glue you have to write

Put into a table, the real state of the tenant chain:

Segment	Automatic?	How
Keycloak → LiteLLM team	Yes	`team_ids_jwt_field` plus `team_id_upsert: true`
Keycloak → LiteLLM user	Yes	`user_id_jwt_field` plus `user_id_upsert: true`
Keycloak → Langfuse organisation	No	Static assignment at sign-up only
Keycloak → Langfuse project	No	Same
LiteLLM team → Langfuse project	No	`callback_vars` by hand, or an operator of your own
Keycloak user → trace `user_id`	No	`end_user_id_jwt_field`, or a contract with the client

The minimum viable architecture without a licence, then, is this:

One Langfuse project per tenant, created with the initialisation variables from GitOps.
That project’s keys in a Kubernetes secret.
A periodic job of your own that reads the teams through the gateway’s administration API and writes callback_vars with the corresponding keys.
People entering Langfuse through Keycloak, but assigned by hand to their organisation.
The end user identifier resolved by contract with the clients, and validated.

That is about one hundred and fifty lines of your own code. It pays to budget for them from the start, because the gap does not close by itself.

Reference configuration

general_settings:
 # only with a licence; without one, virtual keys
 enable_jwt_auth: true
 litellm_jwtauth:
 team_ids_jwt_field: "groups"
 team_id_upsert: true
 user_id_jwt_field: "sub"
 user_id_upsert: true
 end_user_id_jwt_field: "preferred_username"
 user_allowed_email_domain: "ejemplo.es"
 admin_jwt_scope: "litellm_proxy_admin"
 # the three that ship switched off
 enforce_rbac: true
 enforce_scope_based_access: true
 enforce_team_based_model_access: true
 allow_requests_on_db_unavailable: false
 proxy_batch_write_at: 60

litellm_settings:
 callbacks: ["langfuse_otel", "prometheus"]
 # without this, the engine does not know who is calling
 add_user_information_to_llm_headers: true
 turn_off_message_logging: true
 cache: true
 cache_params:
 type: redis
 host: os.environ/REDIS_HOST
 namespace: litellm

# the two that enable the validation that by default does not happen
JWT_AUDIENCE=litellm-gateway
JWT_ISSUER=https://sso.ejemplo.es/realms/plataforma
JWT_PUBLIC_KEY_URL=https://sso.ejemplo.es/realms/plataforma/.well-known/openid-configuration

LANGFUSE_HOST=http://langfuse-web.observabilidad.svc:3000
LANGFUSE_PUBLIC_KEY=...
LANGFUSE_SECRET_KEY=...

And on the Langfuse side:

AUTH_KEYCLOAK_CLIENT_ID=langfuse
AUTH_KEYCLOAK_ISSUER=https://sso.ejemplo.es/realms/plataforma
AUTH_KEYCLOAK_ALLOW_ACCOUNT_LINKING=true
AUTH_DISABLE_USERNAME_PASSWORD=true
AUTH_DISABLE_SIGNUP=true
LANGFUSE_DEFAULT_ORG_ID=plataforma
LANGFUSE_DEFAULT_ORG_ROLE=VIEWER
ENCRYPTION_KEY=... # 64 hexadecimal characters

Checklist

Define JWT_AUDIENCE and JWT_ISSUER on day one, or use the per-issuer configuration. Without that, neither audience nor issuer is validated.
Set enforce_rbac, enforce_scope_based_access and enforce_team_based_model_access to true, and test that a token from another client of the same realm is rejected.
Decide where the end user of the traces comes from and write it into the design. If it is not decided, traces come out with no person.
Separate Postgres. Separate Redis. If Redis is shared, different prefix and logical database, and review the memory policy.
Check that the proxy starts with Postgres down only if that is what you want, and that the on-call team can tell an admission 503 from a credential 401.
Alert on litellm_service_latency per service, not on Redis availability, because the circuit breaker hides the problem.
One Langfuse project per tenant with the initialisation variables, and the per-team keys in callback_vars.
Put a collector in the middle if you need an on-disk queue, tail sampling or attribute redaction.
Check that the gateway version is above 1.83.14 because of the earlier critical vulnerabilities.
Write into the risk analysis that virtual keys are credentials decoupled from the identity provider, and define the offboarding procedure.

Traps and things that are not what they seem

A properly signed JWT is not a JWT addressed to this service. Without the two environment variables, the audience is not checked.

enforce_rbac set to false does not mean there is no authorisation, it means that authorisation comes only from team membership, if there is a team claim. With no team claim and without the flag, a valid token reaches /chat/completions.

The stale copy of the public keys lasts up to seventy minutes, adding the two TTLs. Revoking a key at the provider does not take effect immediately.

With a single key in the JWKS and no key identifier, the identifier is not checked.

The trace user is set by the client. Any per-person spend report built on that field is a declaration, not a measurement.

Redis going down generates no errors, it generates per-pod limits. It is the most expensive of the failures that give no warning.

Langfuse does not start without ClickHouse or without an S3 bucket. Its failure surface is bigger than the gateway’s, and that changes how the high availability effort is shared out.

Langfuse’s per-project access control is paid for in self-hosted, as are audit logs. For an ENS file, that gets decided before building, not after.

Sharing Redis does not break through key collision, it breaks through memory policy. It is a failure that shows up under load and gets diagnosed badly.

Langfuse’s ingestion route is OTLP over HTTP only. A collector configured with the gRPC exporter points at somewhere that does not exist.

Closing

The five seams share a common pattern: they fail without warning. The token from another service gets in, the trace user comes from the client, the rate limit stops being global, the spend queue discards on reaching 64 MB and the Langfuse project does not create itself. None of those five things produces a visible error.

Day 2 work, then, consists above all of turning silences into signals. Two environment variables so that the audience is validated, three flags so that authorisation is real, an alert on service latency instead of on Redis availability, and an explicit contract on where the end user’s identity comes from.

And one decision worth taking early, before the budget is closed. The gateway’s JWT authentication and Langfuse’s per-project access control are both behind a licence. A sovereign platform can live perfectly well without either, with virtual keys and one project per tenant, but it is a different architecture and it has to be drawn as such from the start.

Sources

Código de LiteLLM 1.102.0, commit 9071ca50 del 11 de septiembre de 2026.
Código de Langfuse 4.35.0, commit 39e3cd7d del 11 de septiembre de 2026.
Autenticación y SSO en Langfuse, consultado el 12 de septiembre de 2026.
SCIM y API de organización, consultado el 12 de septiembre de 2026.
Integración OpenTelemetry de Langfuse, consultado el 12 de septiembre de 2026.
Postgres y caché en la documentación de self-hosting, consultados el 12 de septiembre de 2026.
Autenticación por token en LiteLLM y ajustes de configuración, consultados el 12 de septiembre de 2026.
GHSA-jjhc-v7c2-5hh6 y registro de CVE-2026-35030 en NVD.

Sizing for agents: 275 input tokens for every output token, and why the bottleneck is still decode

Sat, 12 Sep 2026 06:00:00 +0200

Seventh article in the operational track of the control layer. The fourth dealt with the coexistence of human traffic and agentic traffic on a fixed fleet. This one deals with the prior question: how big the fleet has to be, and how big the gateway in front of it. Verified against LiteLLM 1.102.0 (commit of 11 September 2026) and vLLM 0.29.0, with load data published between May and September 2026.

TL;DR

Until this year, sizing for agents meant extrapolating a chat profile and adding margin. In 2026 the first production-scale characterisations were published and the profile turned out to be something else. These are the seven conclusions.

The median prompt of an agentic call is 68,000 tokens, of which 63,000 arrive cached. The measurement covers 13.5 million sessions, 3.2 million users, 760.5 million model calls and 44.9 trillion prompt tokens in one week of June 2026. The median input-to-output ratio exceeds 275 to 1. Any calculation done on a profile of 800 prompt tokens and 250 output tokens describes a different system.

The load is prefill-dominant when counting tokens and decode-dominant when counting time. The genuinely new tokens that have to be prefilled on each turn are only between 1.5 and 7.3 times the output, not 275. With the prefix cache working, the split of execution time runs from 91 % to 98.6 % in decode. You size by memory bandwidth and by KV cache capacity, not by prefill FLOPs.

The sizing risk is not the average, it is the collapse of the prefix cache. The median hit rate within a turn is 98 %, it falls to 55 % at the boundary between turns and to 8 % when the client changes model. With dynamically retrieved context, prefix reuse drops to 1 %. One miss turns 142,000 prefix tokens into real prefill. Session affinity per replica stops being an optimisation and becomes a capacity requirement.

The intra-turn concurrency measured in production is 1.15 at the median and 1.4 at the 90th percentile. 93 % of tool batches contain a single invocation. The image of the agent opening twenty calls in parallel does not appear in the data. The real multiplier comes from the duty cycle and from the fact that 87 % of calls are started by the agent, not by the person.

The gateway is sized per worker, and each worker is a universe. One vCPU and 4 GiB per worker, with requests equal to limits. The limit counters, the caches and the cooldowns live in the worker’s memory unless there is Redis, so a limit of 100 requests per minute with four workers and three replicas allows 1,200. The Prisma pool is also per process: the number of connections open against Postgres is workers times connection_limit.

The CPU cost that grows with agentic load is the token counter. It serialises the entire tool catalogue on every request and tokenises it. The project itself measured 46, 53 and 100 ms for prompts of 50K, 75K and 100K tokens, and the operation passes through a concurrency limiter of four per worker. With agents sending long catalogues and six-figure contexts, that limit is the ceiling before the CPU is.

Autoscaling by memory does not work on this piece and the startup probe takes five minutes. The official chart deliberately leaves memory without a target because Prisma’s query engine never returns the maximum reached. That leaves CPU at 60 %, with a startupProbe of 300 seconds that makes any new replica arrive late. For agents there are two better metrics, and they are in /metrics.

You are here: the arithmetic that comes before buying

Classic capacity planning starts from a chat SLO and crosses two budgets, the VRAM one and the time one. That method is still correct. What changes with agents are the inputs: the prompt length, the number of calls per unit of human work, the split between prefill and decode, and the very definition of an SLO. This article replaces those inputs with measured figures and drags the result through to two numbers: gateway replicas and GPUs.

Out of scope is the splitting of traffic once it is built, which was already covered by the coexistence of humans and agents and prefix routing.

The analogy: the repair shop and the front desk

An appliance repair shop sizes its staff by counting faults, not phone calls. The difference matters because the person who brings in the washing machine occupies the front desk for three minutes, while the technician who takes it apart goes back to consult the machine’s data sheet fourteen times over the afternoon. If the shop manager sizes by visits to the front desk, they end up short of technicians and overstaffed at reception.

The inference platform has the same asymmetry. The person sends one turn. The agent sends fifteen calls for that turn, and each one drags along the complete file of everything done so far. The front desk is the gateway and the technicians are the GPUs, and the usual mistake is to size both with the same counter.

There is a second detail the analogy captures well. The technician does not read the whole data sheet each time: they have it open on the bench. That is the prefix cache. The day somebody moves them to a different bench between consultations, they have to fetch it from the archive again, and the work multiplies without the number of faults having increased.

Part 1. The real load profile

For two years, any capacity calculation for agents was done on assumptions. In 2026 three pieces of work appeared that measure the matter, and it is worth reading them together because they measure different scaffolds and their figures are not averageable.

The three sources

The characterisation at scale was published by a research team working on GitHub Copilot production telemetry (arXiv 2608.00101, 30 July 2026): 13.5 million sessions, 3.2 million users, 95.1 million user turns, 760.5 million model calls and 44.9 trillion prompt tokens, in one week of June 2026. It is the largest sample published to date.

The characterisation by benchmark suites comes from an architecture team (arXiv 2605.26297, 25 May 2026), which instruments five of them (ADE-Bench, DABStep, GAIA, SWE-bench Pro and Terminal-Bench 2.0) and measures the internal split of time.

The production measurement on a different scaffold was published by the vLLM project itself together with AgentX (vLLM blog, 8 September 2026).

The three disagree on turns per session: 15 calls at Copilot’s median, 43 turns in AgentX, 62.4 in SWE-bench Pro with Qwen. It is not a contradiction, they are scaffolds with different loops. The sizing lesson is that the number of calls per unit of human work depends on the client that connects, and that is why you have to measure it at home before fixing the size.

The figures you can actually use

Quantity	Measured value	Source
Median prompt per call	68K tokens, 63K of them cached	2608.00101
Input:output ratio (median)	above 275:1	2608.00101
Median session	3 user turns, 15 calls, 4.2 min	2608.00101
Mean session	6.1 turns, 40.6 calls, 62.6 min	2608.00101
Calls per turn (median)	4.5	2608.00101
Calls started by the agent	87 %	2608.00101
Range across user profiles	50× (23K to 1.1M tokens per turn)	2608.00101
Median input context	142K tokens; output 444 tokens per turn	AgentX
Accumulated context in SWE-bench Pro	mean 68.7 to 80.1K, maximum 146 to 166K	2605.26297
Reasoning tokens over the output	29.0 % to 67.6 %	2605.26297

Two operational observations come out of that and appear in no sizing guide.

The first: agents that fail accumulate up to 1.8 times more average context than those that finish well (2605.26297). The cost does not correlate with success, it correlates with difficulty and with getting stuck. A sizing done on the task that goes well falls short.

The second: the range across user profiles is 50 to 1. A user who only converses spends 23,000 tokens per turn; one who leaves the agent in a long loop spends 1.1 million. Segmenting by usage profile pays off more than any difficulty-based routing, because the variance is in the people, not in the requests.

What to measure before calculating

Three of your own figures, taken from a week of real traffic, are worth more than any table in this article:

Model calls per hour of human work, by user profile.
Prompt length distribution (median and 95th percentile), not the mean.
Aggregate prefix cache hit rate, with the time series, not the single number.

All three come from what the gateway already stores. In LiteLLM_SpendLogs you have the tokens per request and the key and team identifiers; in proxy_server_request you have the tools block of each call; and in vLLM’s /metrics you have vllm:prefix_cache_queries and vllm:prefix_cache_hits.

Part 2. Prefill, decode and the cache trap

Here is the finding that corrects almost everyone’s intuition, including mine when I started this article.

Counting tokens, agentic load is overwhelmingly input: 275 to 1. The apparently obvious conclusion would be to size by prefill capacity, that is by FLOPs. It is false.

What has to be prefilled on each turn is not the whole prompt, but what was added since the previous turn. That addition, measured, sits between 1.5 and 7.3 times the output length (2605.26297). The rest is served by the prefix cache. And the real split of the model’s execution time comes out like this: between 91.0 % and 98.6 % in decode, between 1.4 % and 9.0 % in prefill (2605.26297).

The hit rate that sustains that split is high and has been measured in three places: theoretical from 87.9 % to 99.3 % and empirical from 84.6 % to 99.5 % (2605.26297); above 96 % in production (AgentX); and a median of 98 % within a turn in Copilot (2608.00101).

Where it falls over

The same Copilot work measures the degradation, and it is the most actionable figure in the article:

98 % median hit rate within a turn. The detail matters: the first call of a turn hits around 45 %, the second rises to 86 % and from the third onwards it stabilises between 92 % and 94 %.
55 % at the boundary between turns, with idle gaps of 4.1 minutes at the median.
8 % when the client changes model.

And there is a fourth, worse regime: with dynamically retrieved context, that is when the scaffold injects search or RAG results that change on every iteration, prefix reuse falls to 1 % or less (arXiv 2608.15127, 15 August 2026).

A cache miss with a context of 142,000 tokens turns a prefill of a few thousand tokens into a six-figure one. The work multiplies by a factor of between 20 and 60 without the number of requests having changed. That is the difference between a fleet that meets its targets and one that seizes up mid-afternoon.

Two sizing rules come from that:

Without session affinity towards the same replica, the calculated sizing is worthless. LiteLLM’s concrete mechanism is session_affinity, which is not active by default and which was already covered in the prefix routing article. Without it, the calculation has to be redone assuming a prefill regime, which is between 20 and 60 times more expensive.
Changing model halfway through a session costs the entire cache. Any difficulty-based routing policy that moves a live conversation from one model group to another pays an 8 % hit rate on the next call. If you want to route by difficulty, do it at the start of the session.

Co-batching, which is the other effect nobody measures

Mixing a short request into the same batch as long-context requests degrades the short one’s inter-token latency. The measured figure: one long request in the batch raises time per token by 38.3 %; two raise it by 79.7 % (2608.15127). It is the technical reason why separating human and agent pools improves the human experience more than the capacity split suggests, and it is also the reason not to raise --max-num-seqs above what the KV budget allows.

Part 3. The KV budget, with the numbers of the agentic case

The formula does not change with respect to classic capacity planning, but the inputs do, and the result is surprising:

$$\text{KV}_{\text{bytes/token}} = 2 \times n_{\text{layers}} \times n_{\text{kv heads}} \times d_{\text{head}} \times \text{bytes}_{\text{elem}}$$

For a 70B-class geometry (80 layers, 8 KV heads with grouped attention, head dimension 128, 2-byte elements) that gives 0.3125 MiB per token. With the median agentic context of 142,000 tokens:

$$142{,}000 \times 0.3125\ \text{MiB} \approx 43.3\ \text{GiB per sequence}$$

For a 30B-class geometry (48 layers, everything else the same) that gives 0.1875 MiB per token, that is around 26.0 GiB per sequence.

It is worth pausing on what that means. An 80 GB H100 with the weights of a 70B model in FP8 already taking up space does not have 43 GiB free for a single sequence. The conclusion is harsh and has to be written down: serving real 70B-class agentic context demands tensor parallelism of at least 2 and a quantised KV cache. And the number of simultaneous sequences that fit, which is what sets --max-num-seqs, comes from dividing the available KV by that figure, not from a chat heuristic.

$$\text{sequences} \approx \frac{\text{VRAM}_{\text{KV available}}}{\text{KV}_{\text{bytes/token}} \times \text{max context}}$$

With FP8 in the KV, the divisor halves. It is the cheapest capacity lever there is in this load profile, and its cost in quality is validated with evals, not assumed.

Part 4. From people to concurrent requests

This is where most calculations overshoot, because it is assumed that an agent equals many simultaneous requests. The data says otherwise.

The concurrency of calls within a turn, measured in production, is 1.15 at the median and 1.4 at the 90th percentile, and 93 % of tool batches contain a single invocation (2608.00101). Intra-turn parallelism is small. Subagents appear in 44 % of AgentX sessions, with a median of 4 deployments among those that use them, so there is a long tail, but it is not the usual regime.

What does multiply is the duty cycle. The mean session lasts 62.6 minutes and contains 40.6 calls. At 444 output tokens and an interactivity of around 60 tokens per second, each call occupies around 7.4 seconds of decode, which gives between 5 and 7 active minutes out of those 62.6.

$$C = D \times \rho \times A$$

where $D$ is the people working, $\rho$ the duty cycle and $A$ the amplification. With $\rho \approx 0.10$ to $0.12$ derived from the previous figures, twenty developers give between 2 and 2.4 concurrent sequences in the steady state. It is the peak, not the average, that has to be sized for.

The amplification

Three factors published separately:

Tool failures: 9 % of turns trigger them, with a compute amplification of 4 times (2608.00101).
Subagents: 44 % of sessions, median of 4 deployments (AgentX).
Internal variance: the same task from the same agent varies by up to 30 times in cost within a single trace (2608.15127).

The product of the first two gives $(1 + 0.09 \times 3) \times (1 + 0.44 \times 3) \approx 2.9$. Nobody publishes that product as a single figure, so it is flagged as my own derivation, but an amplification factor of 3 is the reasonable design provision and it fits the measured variance.

For the twenty developers in the example: between 6 and 7 concurrent design sequences, with 142K contexts. At 43.3 GiB of KV per sequence in the 70B class, that is around 300 GiB of KV cache, that is four H100s dedicated to KV alone with the weights on top. With KV in FP8, half. With a 30B-class model, a third.

That calculation, done with the chat profile of 800 prompt tokens, would have given a single GPU. That is the distance between the two regimes.

Part 5. The gateway, which also has to be sized

The proxy consumes no GPU, but it has its own arithmetic and it is the part that produces the most surprises in production, because almost all of its state is per process.

A worker is a universe

The default value of --num_workers is 1 (litellm/constants.py:57). The usual startup is uvicorn directly with uvloop (proxy_cli.py:1477), and there are alternatives with gunicorn, hypercorn and granian.

What lives in each worker’s memory, and therefore multiplies:

The in-memory cache, limited to 200 elements with a TTL of 600 seconds (litellm/caching/in_memory_cache.py:27). With many deployments, the cooldowns and the counters compete for that same space and get evicted before they expire.
The key authentication cache, with a TTL of 60 seconds (proxy_server.py:1650), adjustable with general_settings.user_api_key_cache_ttl.
The limit and budget counters when there is no Redis (parallel_request_limiter_v3.py, local_only=True paths).
The router cooldowns, with DEFAULT_COOLDOWN_TIME_SECONDS = 5 (constants.py:75).
A periodic task scheduler of its own per worker (proxy_server.py:9674), coordinated between pods with a lock in Redis.

The practical consequence: a limit of 100 requests per minute with four workers and three replicas allows up to 1,200 without Redis. With Redis, the v3 limiter registers seven Lua scripts at startup (parallel_request_limiter_v3.py:608) and the limits become global.

One detail of the v3 limiter that matters with agentic load: LITELLM_TPM_TOKEN_RESERVATION_ENABLED comes enabled by default and reserves estimated tokens before the call, which adds a round trip to Redis per request. And there is an asyncio.Lock per instance that serialises the atomic batch checks (:665), documented by the code itself as a 99th-percentile risk under contention.

If Redis goes down, requests do not fail: there is a circuit breaker with a threshold of 5 failures and a recovery of 60 seconds (constants.py:456), and the proxy degrades to local memory. What is lost silently is the globality of the limits.

Postgres

The pool is also per process. The default values are database_connection_pool_limit = 10 and database_connection_pool_timeout = 60 (proxy_cli.py:62), injected as parameters of the Prisma URL. A pod with four workers opens forty connections. Three pods, one hundred and twenty. It is the fast route to exhausting Postgres’s max_connections without having served any traffic, and the reason the official chart offers PgBouncer inside the pod itself with maxDbConnections: 20 and maxClientConn: 1000.

On the request path there is only the authentication, which is cached for 60 seconds, and the rate limiting. Spend is deferred. The SpendLogs queue is capped at 64 MB and on overflow it discards the oldest elements with an error in the log (utils.py:6704). The write interval is proxy_batch_write_at, with a default value of 10 seconds plus a random spread of up to 5 (constants.py:1719, proxy_server.py:9704); the production guide recommends raising it to 60.

With a slow Postgres, the queue grows to those 64 MB and then loses billing data without returning any error to the client. It is the silent failure mode you have to alert on.

The token counter, which is the agentic trap

This is the piece that turns a correct sizing into an insufficient one, and it appears in no guide.

Before admitting a request, the proxy counts tokens. With chat load the cost is negligible. With agentic load it is not, for two reasons that add up.

The first is the prompt length. The project itself measured the admission counting cost at 46, 53 and 100 ms for prompts of 50K, 75K and 100K tokens (benchmarks, consulted on 12 September 2026), and published that with a counter written in Rust those figures fall to 4.9, 6.8 and 10.2 ms.

The second is the tool catalogue. The function that prepares the definitions serialises the entire catalogue to a pseudo-TypeScript and tokenises it (litellm_core_utils/token_counter.py:934), plus nine fixed tokens. The cost is linear in the size of the catalogue and it is paid on every request. That is exactly the pattern of an agent connected to an MCP gateway: a long catalogue, resent on every turn. The token bill of that catalogue was already covered in the MCP gateway article; here its other face appears, which is proxy CPU.

And on top of that there is an explicit ceiling. The counting runs in a thread with a capacity limiter whose default value is 4 per worker (token_counter.py:340, constants.py:407). With 100K prompts, four concurrent counts of 100 ms mean a theoretical ceiling of around 40 admissions per second per worker before the limiter starts queueing, and that is without having called the model yet.

What is cached is the tokeniser, with an LRU of 64 entries (utils.py:2206), not the result of the count.

Three levers, in order of return: raise TOKEN_COUNTER_MAX_CONCURRENT_COUNTS if CPU is spare, reduce the tool catalogue exposed per key, and watch the gateway overhead figure in /metrics to know whether this is happening.

Streaming and admission

There is no connection limit by default. --limit_concurrency exists but is optional, and it returns 503. The built-in admission control is only activated if general_settings.max_in_flight_requests_per_worker is defined (middleware/admission_control_middleware.py:285), with max_queued_requests_per_worker and admission_queue_timeout_seconds at 1.0 by default, rejecting with 503 and a retry-after: 1 header.

For agents, that header matters: an agentic client that retries without respecting it multiplies the load exactly when the gateway is saying it cannot take any more. The combination of gateway retries was already calculated by the fourth article and it reached 45 calls per turn in the worst case.

The default request timeout is 6,000 seconds (constants.py:512). The production guide recommends 600. With long agentic tasks, leaving it at 6,000 means a hung connection occupies a slot for an hour and a half.

Resources and autoscaling

The official chart recommends 1 vCPU and 4 GiB per worker, with requests equal to limits (helm/litellm-helm/values.yaml:235), and the production documentation suggests one worker per pod scaling horizontally. The project’s own high-performance benchmark, by contrast, uses four workers per pod with PgBouncer in front.

The HPA comes with targetCPUUtilizationPercentage: 60. And there is a comment in the chart worth reading twice: memory is deliberately left without a target because the resident consumption of Prisma’s query engine is a high-water mark that is never returned, so that an HPA on memory would add replicas and never remove them (values.yaml:260).

The other problem is startup. The startupProbe has failureThreshold: 30 with a period of 10 seconds, that is up to 300 seconds before the pod is considered ready (values.yaml:109). An HPA that reacts at 60 % CPU and takes five minutes to add capacity arrives late to any agentic burst.

The alternative, optional in the chart, is to scale by Prometheus metrics: targetRequestsPerSecond and targetTokensPerSecond over rate(litellm_proxy_total_requests_metric_total[1m]) and rate(litellm_total_tokens_metric_total[1m]). The chart itself warns that the token metric lags with long streaming responses, because tokens are accounted for on completion. For agents, that is almost always.

Part 6. What to watch: the metrics that work and the ones that broke

At the gateway

Metric	What for
`litellm_in_flight_requests`	Real saturation per worker. It is the scaling metric
`litellm_overhead_latency_metric`	Proxy cost per request; it rises with token counting
`litellm_overhead_with_guardrails_latency_metric`	The same including guardrails
`litellm_request_queue_time_seconds`	Arrival through to pre-call, includes authentication and ASGI queueing
`litellm_deployment_cooled_down`	Replicas out of the pool
`litellm_service_latency`	Redis and database latency, per service

The /health/backlog endpoint returns in_flight_requests, admitted_requests, queued_requests and rejected_requests (health_endpoints/_health_endpoints.py:1871).

What it does not expose: the depth of the SpendLogs queue or its bytes, the Prisma connections in use against the limit, nor the depth of the logging worker’s queue. All three are real failure modes and have to be instrumented from outside.

At the engine

Names verified in the vLLM 0.29.0 metrics documentation:

vllm:num_requests_running, vllm:num_requests_waiting
vllm:time_to_first_token_seconds, vllm:e2e_request_latency_seconds, vllm:request_queue_time_seconds, vllm:inter_token_latency_seconds
vllm:prefix_cache_queries, vllm:prefix_cache_hits, vllm:kv_cache_usage_perc

And the ones that break dashboards on upgrade:

vllm:time_in_queue_requests is duplicated by vllm:request_queue_time_seconds.
The direct prefix cache hit rate indicator disappeared. It is now computed as rate(vllm:prefix_cache_hits) / rate(vllm:prefix_cache_queries).
vllm:num_requests_swapped and vllm:cpu_cache_usage_perc are deprecated along with CPU swapping.
vllm:gpu_cache_usage_perc became vllm:kv_cache_usage_perc, which also breaks any HPA that uses it.

The SLO worth chasing

A 95th percentile of time to first token describes nobody’s experience when the unit of work is 43 turns. Coding tasks frequently exceed ten minutes and research tasks reach hours (2608.15127).

Two indicators serve better:

90th percentile of full-task latency, measured per session, not per request. It comes from grouping LiteLLM_SpendLogs by session identifier.
90th percentile of interactivity, in tokens per second per user. It is the one AgentX uses, with values between 58.3 and 74.2 in its runs.

To autoscale the engine, the official production-stack example uses vllm:num_requests_waiting with a threshold of 5, polling every 15 seconds and a cooldown of 360. With agentic load it is worth adding vllm:kv_cache_usage_perc as a second trigger, because with 26 to 43 GiB of KV per sequence the memory pressure arrives before the queue does.

Part 7. The engine levers, ordered by return

Verified against vLLM 0.29.0, released on 9 September 2026.

--long-prefill-token-threshold 512. The AgentX measurement attributes to it +93 % in tokens per GPU per second, with a 2.3-times improvement at the 90th percentile, by attacking the head-of-line blocking that a long prefill produces in front of short requests. It is the highest-return lever documented for this profile.

KV cache in FP8. It halves the memory budget per sequence, which is the limiting resource. Validate with evals.

--max-num-queued-reqs and --max-num-queued-tokens. New in 0.29. They are admission control in the engine, and they are the right mechanism to protect the SLO instead of letting the queue grow without limit.

max_num_batched_tokens downwards. High values improve time to first token; low values improve inter-token latency. In a regime where 91 % of the time is decode, inter-token latency is what is perceived. Chunked prefill is enabled by default whenever possible in the V1 engine.

Session affinity at the gateway. It is not an engine setting, but it is what holds up everything above. Without it, that 98 % hit rate turns into random distribution across replicas.

Two version warnings: Model Runner V2 becomes the default in 0.29 and V1 is deprecated with removal planned for 0.32; and starting with python -m vllm.entrypoints.openai.api_server is deprecated.

The full calculation, from start to finish

Case: twenty developers, a coding scaffold, a 30B-class model in the fleet, maximum context of 128K, KV in FP8.

1. Steady-state concurrency. $C = 20 \times 0.11 \times 3 \approx 6.6$ sequences.

2. KV budget per sequence. 0.1875 MiB per token in BF16, half in FP8, that is around 0.094 MiB. At 128K of context: around 12.0 GiB per sequence. For 6.6 sequences with margin, rounding to 8: 96 GiB of KV.

3. GPUs. The weights of a 30B in FP8 take up around 30 GB. On an 80 GB H100 with gpu_memory_utilization of 0.90 there are around 72 GB usable per card. With tensor parallelism of 2, the weights are split: around 15 GB per GPU, so around 57 GB of KV per GPU and around 114 GB in the replica. That covers the 96 GiB calculated, and TP=2 also adds memory bandwidth, which is the resource that dominates decode. Two H100s, one replica, TP=2. It is worth seeing the contrast: on a single H100 the full weights leave around 42 GB of KV, that is three and a half sequences, less than half of what is needed. With a second replica for availability and maintenance windows: four GPUs.

4. Requests per second at the gateway. 20 people, a mean session of 40.6 calls in 62.6 minutes, duty cycle included: of the order of 0.2 to 0.3 requests per second on average, with peaks of 2 to 3.

5. Gateway workers. The ceiling is not set by throughput but by token counting: four concurrent counts per worker, at 100 ms with long prompts. With peaks of 3 requests per second and six-figure prompts, two replicas of one worker each cover the case with margin for the rolling update, at 1 vCPU and 4 GiB each. Postgres will see 20 connections.

6. What needs switching on. Session affinity, admission control at the gateway and at the engine, --long-prefill-token-threshold, and Redis so that the limits are global.

The same calculation with the chat profile would have given one GPU and one worker. The distance is not in the safety margin, it is in the inputs.

A reference `config.yaml`

model_list:
 - model_name: agentes-30b
 litellm_params:
 model: hosted_vllm/qwen3-coder-30b
 api_base: http://vllm-agentes.inferencia.svc:8000/v1
 rpm: 600
 model_info:
 input_cost_per_token: 0.00000018
 output_cost_per_token: 0.00000072

router_settings:
 routing_strategy: simple-shuffle
 # without this, the capacity calculation in this article is worthless
 optional_pre_call_checks: ["session_affinity"]
 num_retries: 2
 allowed_fails: 3
 cooldown_time: 30

general_settings:
 # admission control: below this the queue grows without limit
 max_in_flight_requests_per_worker: 24
 max_queued_requests_per_worker: 48
 admission_queue_timeout_seconds: 1.0
 # the spend queue discards at 64 MB; shorter interval, less backlog
 proxy_batch_write_at: 60
 enable_drain_endpoint: true
 drain_endpoint_token: os.environ/DRAIN_TOKEN
 database_connection_pool_limit: 10

litellm_settings:
 # 6000 s by default: a hung connection holds a slot for an hour and a half
 request_timeout: 600
 callbacks: ["langfuse_otel", "prometheus"]
 cache: true
 cache_params:
 type: redis
 host: os.environ/REDIS_HOST
 namespace: litellm

Environment variables that go with it:

NUM_WORKERS=1
TOKEN_COUNTER_MAX_CONCURRENT_COUNTS=8 # default 4; raise only if CPU is spare
LITELLM_TPM_TOKEN_RESERVATION_ENABLED=true
PROXY_CONFIG_RELOAD_INTERVAL_SECONDS=30

And on the engine side:

vllm serve Qwen/Qwen3-Coder-30B \
 --tensor-parallel-size 2 \
 --max-model-len 131072 \
 --kv-cache-dtype fp8 \
 --long-prefill-token-threshold 512 \
 --max-num-queued-reqs 64 \
 --gpu-memory-utilization 0.90

Checklist

Measure three figures of your own over a week: calls per hour of human work and per profile, prompt length distribution, and the time series of prefix cache hit rate.
Calculate the KV per sequence with the model’s real geometry, taken from its config.json, not from a table.
Set --max-num-seqs from that budget, not from a chat heuristic.
Switch on session affinity before accepting the calculated number of GPUs.
Switch on admission control in both layers and check that the client respects retry-after.
Lower request_timeout from 6,000 to 600.
Raise proxy_batch_write_at to 60 and alert on the growth of the spend queue from outside, because there is no metric.
Leave the HPA on CPU at 60 %, never on memory, and assume the 300 seconds of the startupProbe when planning the margin.
Watch litellm_in_flight_requests and litellm_overhead_latency_metric; if the overhead rises with the size of the tool catalogue, the culprit is the token counter.
Go over the dashboards for the vLLM metric renames before upgrading.
Check that no routing policy moves a live session between model groups.
Redo the calculation when a new user profile arrives: the range across profiles is 50 to 1.

Traps and things that are not what they seem

The ratio of 275 to 1 does not mean the system is prefill-bound. The new tokens per turn are between 1.5 and 7.3 times the output. The time is between 91 % and 98.6 % in decode.

The aggregate prefix cache hit rate hides the problem. An aggregate 90 % may be 98 % intra-turn and 55 % at boundaries. You have to look at the series, not the number, and segment by request type.

vLLM’s hit rate counts the reprefill after an eviction as a hit. The counter exists internally and is not exposed, so the published number is optimistic under a memory-pressure regime.

Agents that fail cost more than those that succeed, up to 1.8 times more context. Sizing on the task that goes well leaves the calculation short.

The database pool is per process, not per pod. Four workers are forty connections.

The HPA on memory does not work with this piece, and not because of a configuration mistake: Prisma’s query engine does not return the memory.

The 300-second startupProbe turns any autoscaling into late reaction. For agentic peaks, it is better to over-provision than to trust the scaling.

Raising --max-num-seqs to “make use of the GPU” degrades perceived latency: two long requests in the batch raise time per token by 79.7 %.

The token counter has a limiter of four per worker and it appears in no sizing guide. With large tool catalogues and long contexts it is the first ceiling you hit.

The figures in this article come from three different scaffolds and are not averageable. Fifteen, forty-three and sixty-two turns per session are three systems, not three measurements of the same one.

Closing

The arithmetic of sizing has not changed. What has changed are the inputs, and now there are public measurements at scale instead of intuitions. Three figures sum up the change of regime: 68,000 tokens of median prompt, 87 % of calls started by the agent rather than by the person, and a time split that is decode for more than 90 % as long as the prefix cache hits.

From that comes an order of work that is not the usual one. First, session affinity, because without it the rest of the calculation describes a different system. Second, the KV budget with the model’s real geometry, which is what sets how many sequences fit. Third, the gateway, where the ceiling is not throughput but a token counter with four threads per worker.

And a warning about your own figures. The measured range across user profiles is 50 to 1. Any platform that does not segment its measurement by profile is averaging two different populations and producing a number that describes neither of them.

Sources

Agentic Coding in the Wild: Characterizing GitHub Copilot at Production Scale, arXiv 2608.00101, 30 de julio de 2026.
Agentic AI Workload Characteristics, arXiv 2605.26297, 25 de mayo de 2026.
From LLM Inference to Agentic Workloads, arXiv 2608.15127, 15 de agosto de 2026.
vLLM x AgentX: serving agentic workloads, 8 de septiembre de 2026.
vLLM metrics design y optimization guide, consultados el 12 de septiembre de 2026.
LiteLLM benchmarks y production setup, consultados el 12 de septiembre de 2026.
Código de LiteLLM 1.102.0, commit 9071ca50 del 11 de septiembre de 2026.
Autoescalado con KEDA en vLLM production-stack, consultado el 12 de septiembre de 2026.

LiteLLM's MCP gateway: the second front door, the tool catalogue nobody bills, and why filtering it can make selection worse

Fri, 11 Sep 2026 01:45:00 +0200

Sixth post in the operational track on the control layer. The previous five treat the gateway as the piece the tokens pass through: the pair with Langfuse, day 2, virtual keys, humans and agents and prefix routing. This one is about the other door. Verified against LiteLLM 1.102.0, commit of 10 September 2026.

TL;DR

The MCP gateway is not a module, it is a complete second surface. A native MCP endpoint with four route spellings, a proxy mode with three fixed tools, four REST routes, some thirty-five administration routes and a long set of OAuth discovery endpoints. It still lives under _experimental and it is nearly 20,000 lines.

I correct what I said in the previous post: access is NOT open by default. In 1.102.0, a key with no declared permissions and no team resolves zero servers. What does open things up is a per-server flag, allow_all_keys, and that flag overrides the key’s explicit scope unless you enable a setting that is off by default.

Permissions are enforced on execution, not just on the listing. Calling a tool that did not appear in tools/list returns 403. There are three chained checks before the call, including one on the allowed arguments. The same predicate governs listing and calling, by declared design.

The hierarchy is built out of intersections, with two escapes and a fail-open. The key’s access groups are additive on top of the key and team ceiling. The organisation ceiling replaces rather than intersects when there is nothing below it. And on an indeterminate error while resolving tool permissions, the code returns “unrestricted”. It is a deliberate decision and it has to be declared in the risk analysis.

Per-tool cost is worth zero and that is why its metric disappears. There are exactly two Prometheus metrics for MCP, and the spend one is only incremented if the cost is greater than zero. With the default configuration, that series never exists. There is no listing metric, no latency metric, no error metric, no server health metric.

The cost that does matter is not attributed to MCP anywhere. It is the tokens of the tool definitions, resent on every turn inside the prompt_tokens of the model call. Five typical MCP servers add up to 58 tools and some 55,000 tokens. The tool’s spend row, with its zero cost, has nothing to do with that.

And the gateway’s semantic filter does not act on the MCP door. It only applies when it is LiteLLM calling the model. A client connected to /mcp receives the whole catalogue, filter enabled or not. And even if it did act, filtering per request rewrites the cached prefix of the prompt, which is the opposite of what Anthropic and OpenAI do with their own tool searches.

The good observability ships switched off. Without LITELLM_OTEL_V2, a tool call comes out as a generic span with the model set to MCP: name. With it enabled there is a dedicated span and correct attributes, and the arguments and the result stay hidden unless you opt in. And in neither case is trace context propagated towards the upstream MCP server: the trace is cut at the gateway.

Tool arguments are written, and they are written in the clear. They go in full into the metadata column of the spend table, without passing through the switch that governs prompt storage nor through the redaction function, which only knows about messages and response. Whoever enabled redaction for GDPR probably believes they are covered.

The 1.83 branch concentrated four serious vulnerabilities, and one of them belongs to the MCP gateway itself. The endpoint for testing a connection before saving accepted the stdio transport’s command, args and environment in the request body. A low-privilege key got command execution on the proxy host.

And the specification moved underneath. The revision in force since July 2026 removed protocol sessions, the initialisation handshake and the session header. LiteLLM 1.102.0 advertises the June 2025 revision and its version enumeration does not even contemplate the two later ones.

You are here: the door nobody chose

The gateway was chosen for the reasons in the switchboard post: licence, fit inside the cluster, maturity. What was evaluated was the inference traffic.

Later, somebody connects an agent, the agent needs tools, and the same process that is already authenticated and deployed turns out to speak MCP too. The decision to turn it into the tool door is almost never taken: it is inherited. And that door has neither the same controls, nor the same observability, nor the same cost model.

When MCP grows covered the identity of your own MCP servers. MCP from the inside covered the protocol and its instrumentation. This post covers the piece in between.

The analogy: the switchboard that also hands out keys

An office building’s telephone switchboard. Its original job is to route calls: it decides who can call outside, how much each department spends, and it leaves a record of every conference call.

Over time, the switchboard operator is also given the key cabinet. They are already at reception, they already know who everybody is, it looks like the natural place. And from that moment there are two services at the same desk with very different rules.

Calls are priced by the minute and show up on the bill. Keys have no tariff, so on the department’s bill they come out at zero, and somebody might conclude that the key cabinet costs nothing. It does cost: it costs the operator’s time, it costs having to read the whole key catalogue every time somebody asks what is available, and it costs that nobody checks that the label on a key still says what it said the day its loan was authorised.

The analogy ends at the interesting spot. When somebody takes a key, the record says which key it was and what they said they wanted it for. That record is simultaneously the evidence of the action and a piece of data that perhaps should not be stored in the clear.

The real routes

The first thing that surprises you when you look at the code is how many spellings get you to the same place.

The native endpoint

The MCP sub-application is mounted at /mcp and inside it has four mounts: the root as a wildcard, /mcp, /{name}/mcp and /sse.

And there is also a hand-declared route for bare /mcp on the main application, with a comment that explains why: the mount cannot match its own bare prefix, and the resulting 307 redirect breaks MCP clients sitting behind a proxy that terminates TLS. It is the kind of detail you discover in production on a Friday.

/mcp/sse holds a surprise. It does not use the classic SSE transport: it uses the same HTTP session manager in stateless mode. The SseServerTransport object is constructed and is not referenced on any other line of the file. It is dead code, and the /mcp/sse/messages endpoint that this object advertises does not exist as a working route.

The /{name}/mcp route resolves in a documented order: server alias, comma-separated list with a maximum of 16 elements, tool set, access group. If nothing matches, 404.

And /mcp/proxy exposes a fixed surface of three tools. It is the progressive disclosure pattern, and it will come up again further down because it is the most important mitigation in this post.

The REST routes

Four, under the /mcp-rest prefix:

Method	Route	Authentication
GET	`/mcp-rest/tools/list`	Virtual key
POST	`/mcp-rest/tools/call`	Virtual key
POST	`/mcp-rest/test/connection`	Key plus administrator role
POST	`/mcp-rest/test/tools/list`	Key plus administrator role

The variants without -rest that appear in the internal types have no router registering them. The /mcp/* wildcard absorbs them as JSON-RPC.

The last two are the stars of the security section.

Administration and discovery

Some thirty-five routes under /v1/mcp, almost all of them with virtual key authentication. Three details worth knowing:

GET /v1/mcp/registry.json carries no authentication dependency. It returns 404 if the public registry is not enabled, which is the real protection.

POST /v1/mcp/server requires the proxy administrator role. POST /v1/mcp/server/register does not: it is the proposal path, which requires a key with a team, explicitly forbids the stdio transport and leaves the entry pending review.

And on top of that there is a long set of OAuth .well-known endpoints, with dynamic client registration, authorisation, token, revocation and introspection, plus their per-server variants.

Configuration and transport

The mcp_servers schema in config.yaml has fifty-odd fields, all of them optional. A warning for anyone coming from earlier versions: spec_version no longer exists in 1.102.0.

The default transport when omitted is http, that is, streamable HTTP. The other two values are sse and stdio. The string streamable_http is not valid in the configuration file, although it is accepted as a synonym when importing connector files.

Tool prefixing uses the format {server}{separator}{tool} with - as the default separator, configurable through an environment variable read at import time. The prefix chosen is the alias, or the name, or the identifier, in that order. There is an optional short mode that derives three characters from the hash of the server identifier and resolves collisions by rehashing.

Access control, and the correction to the previous post

In the humans and agents post I closed the list of traps with this one: “access to MCP servers is open by default if no level defines a list”. Verified in 1.102.0, that is false.

The function that resolves the allowed servers does the opposite. If the team declares nothing, it inherits from the key; if the key declares nothing either, the base set is empty and the scope is marked as unbounded with zero servers. There is no branch that expands to “the whole registry” other than the administration role one.

What does open things up, and this is the important part, is something else.

`allow_all_keys`

It is a per-server flag, in configuration, in the API and in the database, defaulting to false. When set to true, that server joins the set of open servers and any key can see it.

The detail to know: by default it overrides the key’s explicit scope. A key that declares access only to server A will also see server B if B has the flag. The key’s scope is only respected if you enable general_settings.mcp_allow_all_keys_respects_mcp_scope, which is off by default.

There is a second opening case: anonymous subjects, with no user identifier, role or key, additionally receive the servers configured with authentication delegation towards the upstream and the pass-through ones. It is consistent with the design of those two modes, and it is a surface you have to know about.

`require_key_mcp_access_defined`

It lives in general_settings, defaults to false, and needs no licence. What it inverts is only the empty-key inheritance towards the team: with the flag on, a key with no list of its own gets an empty set instead of inheriting the team’s.

It does not affect keys that do declare servers, which still intersect with the team. And it does not close off the access group grants, which are additive and are the escape route you have to audit separately.

There is a sibling, require_end_user_mcp_access_defined, with the same shape and applied to the end user.

The hierarchy

The function’s docstring is normative and says that all the rules are intersections. In detail:

Key and team: intersection when both declare. Inheritance in one direction when one is empty.
Key access groups: additive union on top of the previous result.
End user: intersection if it declares anything.
Agent: intersection.
Internal user: ceiling that only narrows.
Organisation: ceiling that intersects if there are restrictions below, and that replaces by becoming the ceiling when there are none.

For subjects with no key, admitted by SSO or by the gateway, the model changes to a per-source union instead of an intersection.

The fail-open

This is the point you have to take to the risk analysis and not leave in a technical note.

On an indeterminate failure while resolving the tool permissions of a key or a JWT, the code returns “unrestricted”, that is, all tools allowed. The organisation ceiling is skipped in the same situation. For subjects with no key the behaviour is the opposite, fail-closed.

There is a well-resolved intermediate case: when a permission is named but cannot be read, a specific error is raised and it is denied in both cases.

The doctrine is documented in the class docstring, which is appreciated. And it remains a behaviour that, in a system under the ENS, you have to declare in writing instead of discovering it during an audit.

What is done well

Permissions are enforced on the call, not just on the listing. The execution path chains three checks before contacting the server: allowed or forbidden tools, per key and team tool permission, and validation of the arguments against the allowed ones. All three return 403.

Listing and calling share the same predicate, and the docstring declares it as an explicit invariant. A client that calls a tool directly when it did not appear in its listing receives 403, it does not execute. The same over REST and over the responses API route.

The x-mcp-servers and x-mcp-access-groups headers only narrow, and they fail closed if they do not resolve.

Header forwarding towards the upstream is an explicit per-server allowlist, with a centralised decision about when to strip the caller’s Authorization, and a protection against cross-forwarding: in a multi-server listing, the global header is withheld if more than one server would consume it. That is RFC 9700 applied.

The bill that never shows up

Here is, in my opinion, the most important thing in this post, and it has nothing to do with security.

Every tool the gateway aggregates takes up space in the model’s context. Name, description and parameter schema travel in the request’s tools block, on every turn, and they are paid for as input tokens of the real model.

The figures published by Anthropic in November 2025 are the best public reference there is. Five connected MCP servers add up to 58 tools and some 55,000 tokens. The breakdown: GitHub, 35 tools and some 26,000 tokens. Slack, 11 tools and some 21,000. Sentry, five and some 3,000. Grafana, five and some 3,000. Splunk, two and some 2,000.

With tool search enabled, that set drops from some 72,000 to some 8,700 tokens, and the usable context goes from 122,800 to 191,300.

The second cost of a large catalogue is not economic and is not billed: it is that the model chooses worse. It has its own section further down, because of the two it is the one that decides whether the agent is any good.

How to measure it with what is already there

The gateway stores enough data, even if it does not cross-reference it for you.

In the listing’s spend row, with call type list_mcp_tools, it enriches the metadata with allowed_server_count, tool_count_total, per_server_tool_counts and per_server_list_outcomes.

In the model call rows, the proxy_server_request column keeps the request body with the tools block inside it.

The practical method: pair the listing rows with the LLM rows of the same session by session_id, read the total number of tools from the first, and pass the tools array of the second through litellm.token_counter(model=..., tools=...), which accepts that argument. Multiply by the number of turns, because the block is resent on all of them.

For the relative cost, a quicker comparison: mean prompt_tokens of keys with MCP enabled against those without, same model and same period.

Tool selection

The previous section treats the catalogue as a bill. This one treats it as what determines whether the agent gets it right. This is ground where the talk runs well ahead of the measurement, so it is worth separating what has been measured from what gets repeated.

How many tools a model can take

Anthropic’s official figure is that Claude’s ability to pick the right tool degrades when going from 30 to 50 available tools. The same documentation gives the inverse criterion, which is the more useful one: the full catalogue with nothing else is the right option with fewer than ten tools, when all of them are used on every request, or when the definitions add up to fewer than a hundred tokens.

There is also an administrative ceiling before the semantic one. VS Code with Copilot cuts off at 128 tools per request and returns an error when you exceed it.

The full curve gets cited far more than it has been measured. The soundest primary data I found is from May 2025: varying the number of candidates from 1 to 11,100 over a registry of more than 4,400 servers, below some 30 tools the success rate is above 90 %, between 31 and 70 intermittent failures appear, and past the hundred mark the degradation becomes dominated by retrieval. It is a qualitative heat map, not a numeric table.

The figure that circulates most (78 % with ten tools, 40 % with a hundred, 13.6 % with seven hundred) comes from a conference talk with no paper, no repository and no published methodology. Do not take it to a client presentation.

And there is a warning that affects everything else. An audit of 496 tasks from four tool calling benchmarks found 18.5 % misalignment between the label and reality. In the BFCL v4 subset, 80 % of the failures assigned to the agent came from fragile state comparisons. In one of the benchmarks with an automatic judge, 23 identical re-runs gave a range from 57.9 % to 76.8 %, almost 19 points of spread, enough to reorder the ranking. Any absolute figure in this section deserves a generous margin of error.

How it fails

The work with the cleanest taxonomy evaluates 36 servers, 220 tools and 1,000 tasks against twenty frontier models, with an average of 15.2 tools exposed per task of which 4.1 are relevant.

Failure mode	Share
Not using a tool when one was needed	10.5 %
Choosing the wrong tool	9.0 %
Malformed parameters	6.9 %
Not recovering from an error	10.3 %
Cognitive failures, not tool failures	63.3 %

The interesting part is in the per-model breakdown: the dominant failure mode changes with the model, not with the size of the catalogue. In one of them, 40.1 % of the failures are simply not calling any tool. In others the malformed parameter dominates. In another, the wrong tool. There is no canonical failure to attack.

On tools that do the same thing, which is the real case of a gateway with fifteen servers, there is a good measurement. A study with ten groups of five functionally equivalent tools and a thousand query pairs puts the selection bias between 0.3 and 0.4, that is, you would have to redistribute between 30 and 40 % of the probability mass for equivalent tools to be chosen equally. The same work shows that a biased continued training takes the selection of a specific endpoint from 0.6 % to 12.8 %.

That study contributes another figure that reorders priorities: shuffling a tool’s description moves the selection distribution substantially, whereas changing only the name has minimal and inconsistent effects. Applied to the gateway’s server-tool prefixing: it costs tokens and probably does not change which tool the model goes to. There is no study isolating the effect of name spacing, and it is a real gap in the literature.

If the description is what carries weight, the state of the estate is the problem. An audit of 856 tools from 103 MCP servers found undeclared limitations in 89.8 %, missing usage guidance in 89.3 % and opaque parameters in 84.3 %. Only 2.9 % were free of problems across all their components.

The optimum is not a number

The May 2026 work I cited above proposes a chance-corrected metric and uses it to size the list adaptively. Over 370 functions it reaches 90.3 % coverage while seeing an average of 7.4 tools, against 90.8 % seeing a fixed 50.

But the same paper publishes its own counterexample, and it is honest to cite it: over another benchmark of 3,251 tools, the fixed K of 5 wins in aggregate, 64.7 % against 61.9 %. The adaptive advantage concentrates in the hard queries, where the right tool sits between sixth and twentieth by similarity: there it finds 16.7 % of the cases where the fixed K finds 0 %.

The March 2026 work over 121 tools puts the plateau at K=3, with 97.1 % that does not improve going up to K=5. With the caveat that it measures retrieval against labels, not model accuracy.

And a July 2026 work gives the formulation that seems to me the right one: ranking by score is inconsistent with optimal acquisition when costs are heterogeneous, so the optimal K is a function of cost, not a constant. Trimming from 7 to 4.4 exposed tools while keeping success is possible; fixing a universal number is not.

Filtering can make the result worse

This is the part that appears in no product blog. A June 2026 benchmark with three registry sizes (25, 100 and 250 tools), six filtering methods, seven models and 26,460 runs:

Strategy	Success	Visible tools	Tokens
Expose the whole catalogue	32.1 %	125.00	56,062
Keyword, top-5	22.1 %	4.80	3,200
Keyword, top-10	22.4 %	9.54	5,356
State-aware	24.0 %	25.95	13,522
Full causal path	24.0 %	26.24	13,697

Four of the five filtering strategies perform worse than not filtering. The fifth, which does win comfortably, exposes 0.99 tools per step and requires hand-written precondition and effect contracts for every tool, plus a search of the causal path to the goal state. It is a planner with the goal given, not a retrieval filter, and it is not comparable with anything a gateway offers.

There is more evidence in the same direction. A work over a corpus of 43,000 tools measured that the agent’s completion rate drops when using retrieved sets against hand-annotated sets: the filter introduces its own recall ceiling. And top-K breaks the tasks that need several tools: on a benchmark of 16,464 endpoints, a fine-tuned retriever reaches a Recall@3 of 68.6 % but only 39.7 % of the queries have their complete tool set within the first three. Almost 29 points of gap between “the one I needed is there” and “all the ones I needed are there”.

Filtering is not free on security either. A March 2026 work attacks the retrieval layer by injecting adversarial tools that saturate the top-k: with injection rates of 1.2 % it manages to dominate the top-k between 91 % and 97 % of the time. It is an attack surface that exposing the whole catalogue does not have.

The most elegant counterpoint comes from Anthropic’s own numbers. With the same catalogue of more than fifty tools, the gain from tool search drops from 25 points on one model to 8.6 on the next, because the unfiltered baseline rises from 49 % to 79.5 %. The margin left by filtering shrank to a third in one generation of models. Any architecture decision taken here has a short shelf life.

The prompt cache trap

Here is what makes a homemade top-K filter a bad idea even if it worked.

Tool definitions do not go just anywhere in the prompt: they go at the beginning. Anthropic documents the order of the cacheable prefix as tools, then system, then messages, with a hierarchy where each level rests on the previous one. And the invalidation rule is explicit: modifying tool definitions, be it names, descriptions or parameters, invalidates the entire cache, and a change at one level invalidates that level and all subsequent ones.

OpenAI says the same and adds a detail: the prefix must remain unchanged, and order counts. The same set of tools served in a different order is a cache miss.

With vLLM serving an open model there is no tool block of its own. The tools parameter is serialised by the chat template inside the first system message, behind the system prompt text. For automatic prefix caching the effect is identical, because it is still near the absolute start of the prompt.

The consequence is the one that matters. A filter that returns a different set of tools on the next turn does not just invalidate the tool block: it invalidates everything behind it, which is the system prompt and the whole conversation, which is exactly the part that grows with the turns. At list prices, a prefix token goes from costing 0.1 times when read from cache to 1.25 times when written. An agent with sixty thousand tokens of stable prefix goes from some six thousand equivalent tokens per turn to some seventy-five thousand, plus the prefill that gets paid again in latency.

The two big providers have separately arrived at the same solution, and it is the opposite of filtering in the prefix. Anthropic excludes deferred tools from the prefix and, when the model discovers one through search, injects it inline inside the conversation: “the prefix stays intact, so prompt caching is preserved”. OpenAI: “all tools are loaded at the end of the model’s context window (…) this allows the cache to be preserved from one request to the next”. Anthropic’s operational recommendation is the stable core pattern: leave the three to five most used tools undeferred.

The closest academic work, from January 2026, states the problem without measuring it and recommends exactly that, keeping a fixed set of reusable general-purpose functions and implementing dynamic capability through code generation.

The July 2026 MCP specification picks up both halves of the matter. Servers “should return the tools from tools/list in a deterministic order to allow caching in the client and improve the hit rate of the model’s prompt cache”. And it adds mandatory ttlMs and cacheScope in listing results, so that the client can freeze the catalogue for a known window instead of re-querying it and propagating any variation to the prompt.

Nobody has publicly measured the cost in euros of this effect. The only attempt I found is a self-published audit of ten thousand turns that puts it at 2.4 % to 3.5 % of cache misses, with non-reproducible methodology. It is a gap in the literature, and the order of magnitude in the previous paragraph comes from applying list prices, not from a measurement.

And now, LiteLLM

With all of the above in mind, the gateway’s levers read differently.

The semantic filter does not act on the MCP endpoint. It is registered as a generic LiteLLM callback, and its only dispatcher is the proxy’s precall hook, which additionally discards anything that is not completion, acompletion or aresponses. The tools/list path does not invoke that hook at any point; there is a comment in the code that admits it in passing, when explaining why the listing’s JWT signature is done separately. An external MCP client connected to /mcp receives the whole catalogue, filter enabled or not.

Lever	Serves external MCP clients	Breaks the prompt cache
Per-server `allowed_tools`	Yes	No, it is static
Name and description overrides	Yes	No, it is static
Selection by route or header	Yes	No, if it is stable per agent
`/mcp/proxy` and virtual tools	Yes	No, the surface is fixed
Semantic filter	No	Yes, it rewrites the block per request

When it does act, the filter has details you need to know before enabling it. It embeds only each tool’s description, with the name as a fallback if it is missing; the parameter schema does not go in. Of the query it embeds only the last user message: on turn twelve of an agent conversation, that might be “yes, go ahead”. The tool embeddings are cached and indexed at start-up; the query’s is computed on every request, with a synchronous call inside a coroutine.

Its failure modes are almost all open, which is the reasonable thing: if the embedding model does not respond, if nothing passes the threshold, if the names do not match the catalogue, it returns all the tools. The exception is context overflow, which fails closed with a 400, and which if it happens while building the index during start-up is memoised and makes all subsequent requests fail until you restart. The defaults are text-embedding-3-small, top_k of 10 and a threshold of 0.3, under litellm_settings.mcp_semantic_tool_filter.

/mcp/proxy does serve external clients, and that is where its real value lies. It replaces the catalogue with three tools with opaque identifiers. How search_tools searches deserves a line, because it is not what you would assume: by default it is neither a regular expression nor a semantic match, it is a keyword counter, how many tokens of the query appear as a substring in the concatenated name and description. It only moves to embeddings if you configure a model in mcp_tool_search. The result cap is hard-coded at five in this mode, and the client cannot ask for more. In exchange, call_tool goes through the same permission checks as a normal call plus a validation of the argument schema that the normal route does not do at that point.

There is a related variant: with mcp_tool_search_enabled in the key’s permissions, the catalogue is replaced by four virtual tools, and there the search one does expose top_k to the client.

The catalogue order is not deterministic, and that breaks the client’s cache without anybody filtering anything. The list of allowed servers comes from iterating a Python set of strings, so with the random hash seed the order differs between processes, between uvicorn workers and between restarts. There is no sorting anywhere in the listing path, and there is no tool cache: every tools/list queries the upstreams live, so the order within each server also depends on what the upstream returns that time. The gateway does not offer today the guarantee that the July specification recommends. The fix, sorting the catalogue before serving it, fits on one line.

Nor is there any pagination towards the client. The catalogue is returned whole, always.

And the lever almost nobody uses is the static one. Every MCP server accepts tool_name_to_display_name and tool_name_to_description, and those overrides really do replace what is sent to the client in tools/list, not just what the administrator sees in the interface. The tools/call routing translates from the new name to the real one before any permission check. With that you can trim a four-hundred-token description written by a third party, rename an ambiguous tool so it does not compete with a similar one, and do it without touching the prompt cache, because it is static and holds for all turns.

It is also the rug pull mitigation left pending from the security section: if the description the model sees is the one the operator wrote, whatever the upstream server changes stops reaching the context. Two limits. They are configured through the database or the management API, not through config.yaml. And the reverse name translation is by exact equality and is not disambiguated between servers, so two servers with the same display name route unstably, made worse by the non-deterministic order of the previous paragraph.

Alongside that, per-server allowed_tools remains the blunt and effective instrument: it filters the listing the client sees, reduces tokens, and is static.

The ladder, in order

Trim. Per-server allowed_tools, plus a per-key allowlist. Cheap, static, does not break the cache and reduces the permission surface at the same time.
Rewrite the descriptions with the overrides. That is where the evidence is, because the description carries weight and the name almost none, and along the way it pins the contract against the rug pull.
Split by route. Each agent to /{server}/mcp or to a tool set, never to the aggregated endpoint. Stable per agent, so the cache survives.
Sort the catalogue before serving it if you control the deployment, while the gateway does not do it.
/mcp/proxy when the catalogue goes past thirty or fifty tools. It is progressive disclosure and it works with external clients, which is more than you can say for the semantic filter.
The semantic filter, only when the one calling the model is LiteLLM. With a generous top_k, knowing that it embeds only the last user message and that context overflow is a closed and sticky failure.
What not to do: a home-grown top-K filter that rewrites the tools array on every turn. It costs the whole prefix and, according to four of the five measured strategies, probably makes selection worse.

The declared cost, which is worth zero

The calculation has four precedence steps: cost set by a post-call hook, per-tool cost in tool_name_to_cost_per_query, server default cost, and 0.0.

Without configuring mcp_server_cost_info, all MCP traffic is accounted at zero and consumes neither key nor team budget.

There are exactly two Prometheus metrics for MCP:

litellm_mcp_tool_calls_total
litellm_mcp_tool_call_spend_metric

Both with the same eight labels: tool name, server name, key hash, key alias, team, team alias, user and end user.

And the second is only incremented if the cost is greater than zero. With the default configuration, that time series never appears in Prometheus. It is not that it is worth zero: it is that it does not exist.

Nor is there a tools/list metric, or MCP latency, or MCP errors, or upstream server health. For that you are left with the generic request metrics.

In the reports, /spend/calculate has no MCP branch: it accepts a model and messages, or a completion response, and it always calls the LLM cost calculation. The per-team report does include MCP calls, grouped by the model column, which for these rows is worth MCP: tool_name. That is: every tool appears as if it were a model, with zero input and output tokens. Only the sessions endpoint separates the two call types explicitly.

Observability, and the trace that gets cut

There are two OpenTelemetry stacks in LiteLLM and which one runs depends on an environment variable.

Without LITELLM_OTEL_V2=true, the old logger runs, and it has no MCP handling: the only appearance of MCP in the whole file is the metadata key in a list. The tool call comes out as a generic litellm_request span with the model set to MCP: name.

With v2 enabled there is a dedicated span, with a dedicated role, client kind and a parent in the proxy’s request span. The span name is tools/call tool-name. The attributes are correct and follow the conventions:

Attribute	Value
`gen_ai.operation.name`	`execute_tool`
`mcp.method.name`	`tools/call`
`mcp.session.id`	MCP session identifier
`gen_ai.tool.name`	Tool name
`gen_ai.tool.call.arguments`	Only with content capture enabled
`gen_ai.tool.call.result`	Only with content capture enabled
`server.address`, `server.port`	From the upstream URL, redacted
`litellm.cost.total`	The cost that is probably worth zero

Content capture is off by default. Without it you see neither the arguments nor the result in the trace.

A note for anyone working with semantic conventions: mcp.tool.name does not exist. The MCP registry has exactly four attributes (mcp.method.name, mcp.session.id, mcp.protocol.version and mcp.resource.uri), all four in development, and the tool name goes in gen_ai.tool.name. LiteLLM gets it right. And the GenAI conventions were extracted into their own repository which as of today has no release at all, so there is no version number to cite.

In Langfuse, the vendor mapper returns an empty dictionary for any span that is not an LLM call. An MCP call appears as a raw span with its attributes, with no native input, output or model fields.

The trace is cut at the gateway

Incoming context is read. The traceparent the client puts in params._meta, following the specification’s propagation proposal, is extracted and turned into a link on the span, never into its parent. The reason is documented in the code: parenting to the remote trace would leave the span hanging off a trace whose root never reaches the backend. The client’s baggage is discarded on purpose to avoid forging identity attributes.

Outgoing context is not propagated. There is no propagator injection call anywhere in the tree. The headers that go out towards the upstream MCP server are built with authentication and the authorised extra headers, and nothing else.

The operational consequence: there is no end-to-end trace. If the upstream MCP server is instrumented, its spans live in another trace and there is no way to correlate them automatically.

Auditing and the conflict with privacy

A tool call leaves a normal spend row, with call type call_mcp_tool, spend equal to the cost per query, model MCP: name, zero tokens in the three token columns, a dedicated mcp_namespaced_tool_name column, and the usual key, team, user and session fields.

And in the metadata column, in JSON, the complete structure of the call: name, arguments, result, server, namespaced name, MCP session identifier, authentication mode and server resource with the URL redacted.

The problem

That structure is written into the metadata column without passing through the privacy switch. The assignment is direct and is not conditioned by the function that governs whether prompts and responses are stored, which does control the messages and response columns.

And the redaction function does not cover it either: it only touches messages and response. A search for MCP in the redaction file returns nothing.

The practical result: an operator who enables turn_off_message_logging for GDPR keeps sending the full arguments of every tool to every logging destination they have configured. To the SIEM, to Langfuse, to S3.

If an argument carries a national ID number, an address, a case number or a medical record, it has been exported.

There is an aggravating factor that does not depend on LiteLLM. The revision of the specification in force introduces an extension that allows mirroring tool arguments into HTTP headers visible to load balancers, proxies and web application firewalls. The specification recommends not marking sensitive parameters that way, and it is a recommendation, and the upstream server decides.

The fit with the ENS

The conflict is real and has a familiar shape: activity logging (op.exp.8) requires the call to be traced and non-repudiable; protection of personal data (mp.info.1) requires that log not to become a secondary repository of personal data with a different purpose. The tool argument is simultaneously the evidence of the action and the data. Redacting it destroys non-repudiation; storing it in the clear creates the repository.

The defensible way out you can build today: a hash of the argument in the audit log, the argument in the clear with encryption (mp.info.3), separate retention and its own access control. The hash sustains non-repudiation and the clear text is only opened under procedure.

There are two other measures this material touches directly. Change management (op.exp.5) because registering an MCP server and, above all, changing the description of an already approved tool are changes to the security configuration. And maintenance and updates (op.exp.4) because, as you can see in the next section, an old version of the gateway is a non-compliance with a name and a number.

Two honesty warnings. The codes follow the numbering of Annex II of RD 311/2022 and you have to check them against the text in force before citing them in a compliance document. And on NIS2: as of the date of this post the directive is still not transposed in Spain, with the draft bill on cybersecurity coordination and governance going through the process and a reasoned opinion from the Commission over the delay. What is enforceable today in the public sector and its suppliers is the ENS.

What is not logged

An administrative gap: registering, modifying and deleting MCP servers generates no audit log. There are three TODO comments in the management code that say so, one of them with an if is_audit_logging_enabled(): pass. In the same deletion block there are two more TODOs about not cleaning up orphaned permissions on keys and teams.

Security

The vulnerability you need to know about

CVE-2026-42271, published on 8 May 2026. Affects LiteLLM from 1.74.2 up to versions prior to 1.83.7. CVSS 3.1 of 8.8.

The POST /mcp-rest/test/connection and POST /mcp-rest/test/tools/list endpoints, which exist to preview an MCP server before saving it, accepted the stdio transport’s command, args and env in the request body. A low-privilege key executed arbitrary commands on the proxy host. It also affects Red Hat OpenShift AI on several branches.

It is the story of this post condensed into one line: the gateway you put in to contain MCP was the hole.

It does not come alone. The 1.83 branch concentrates three more serious ones:

CVE	Component	CVSS	Fixed in
`CVE-2026-42208`	SQL injection in key validation, with no prior authentication	9.8	1.83.7
`CVE-2026-35030`	OIDC userinfo cache keyed on `token[:20]`, collision and impersonation	9.1	1.83.0
`CVE-2026-40217`	Remote execution through bytecode rewriting in the guardrails test endpoint	8.8	no version recorded

Plus two privilege escalations fixed in 1.83.10 and 1.83.14.

The operational conclusion is short: a deployment below 1.83.14 is exposed to remote execution or to an unauthenticated SQL injection. The current branch is 1.102.0.

And a note for context: the whole MCP ecosystem has the same pattern. The unauthenticated remote execution in MCP Inspector, the command injection in mcp-remote when connecting to an untrusted server, DNS rebinding disabled by default in the Python and TypeScript SDKs until late 2025, the directory escapes in the reference servers. Connecting to an untrusted MCP server compromises the client, not just the other way round.

The rug pull nobody covers

Documented since April 2025: a server can change a tool’s description after the client has approved it. The description is what the model reads to decide when and how to use it, so changing it is changing the behaviour without asking for permission again.

The specification in force does not address it. It defines list change notification and cache freshness fields, which are recency mechanisms, not integrity ones. The closest normative warning is that clients must treat tool annotations as untrusted unless they come from trusted servers.

LiteLLM does not cover it either. The only “pinning” in the code is of the configuration’s server identifier, so that it does not change when you edit the YAML. There is no hash or comparison of the description or the input schema between successive listings. A search for related terms in the MCP module: nothing functional.

The control exists outside the gateway. mcp-scan, from the same researchers who documented the attack, implements description hash pinning precisely to detect this.

In ENS terms it is an unauthorised change to the security configuration that today goes undetected. And the home-made fix fits in a cron job: store the hash of the description and schema of every approved tool, compare against tools/list periodically, alert on the difference.

Three more holes in the code

No URL validation towards the upstream. A new server’s url field has no validator. The SSRF protection utilities exist in the tree and are used in OAuth discovery and in downloading OpenAPI specifications, but not in the egress of tools/call. A server pointing at a private range or at the cloud metadata IP is accepted as is.

The stdio transport is local execution on the proxy host. There is a command allowlist: npx, uvx, python, python3, node, docker, deno, extendable through an environment variable. The code’s own comment admits the residual risk: the permitted runtimes can execute code through arguments. With docker and npx in there, it is de facto arbitrary execution for any proxy administrator.

Two server modes skip the gateway’s authentication. Pass-through and authentication delegation to the upstream completely omit virtual key validation. They are well hardened (fail-closed if the target is mixed or unresolvable, strict comparison against truthy values, explicit blocking of the client credentials flow so as not to lend out the proxy’s service account), and the result is still that those servers have no gateway access control.

The limit that truncates in silence

The tools/list pagination stops and returns what it has accumulated in three cases: repeated cursor, one thousand pages, or expiry of a global deadline. The docstring declares it: a slow or faulty upstream produces a partial catalogue instead of an error.

The first two log a warning. The third, the deadline, exits without logging anything at all. A slow server makes part of its tools disappear from the catalogue with no trace left behind.

Correction on the slot leak

In the previous post I took issue 34534 at face value, according to which every MCP tool call acquires a max_parallel_requests slot and does not release it. Tracing the 1.102.0 code, that diagnosis no longer holds: the release path exists, on both success and failure, and the limiter is registered as a litellm callback and not just as a proxy hook.

With one honest caveat: I found no regression test covering the full cycle for MCP, and the definitive confirmation is a real proxy with max_parallel_requests: 2 and five MCP calls in a row. If anybody reproduces it today, I would like to know.

The specification moved underneath

The MCP revision in force is 2026-07-28, published on 28 July 2026, and in between there was another one, 2025-11-25. The changes in the latest one are not cosmetic:

Protocol sessions and the session header are removed. Cross-request state becomes an explicit identifier in the tool arguments.
The initialisation handshake is removed. Every request carries version and capabilities in _meta. MCP becomes stateless.
server/discover appears as a mandatory call.
Elicitation changes shape: out go server-initiated requests, in comes a multi-round pattern where the server returns an “input required” result and the client retries with the answers.
ping, logging/setLevel and the roots change notification are removed. SSE stream resumption is removed.
Standard HTTP headers are mandatory on POST, which lets a gateway or a web application firewall decide without parsing the JSON.
On authorisation: mandatory issuer validation in the client, and dynamic client registration is deprecated in favour of client identifier metadata documents.
Roots, sampling and logging are deprecated, with a minimum window of twelve months.

LiteLLM 1.102.0 advertises 2025-06-18. Its version enumeration has three values and none of them is later than that date. The literal is sent in the initialize against the upstream.

One nuance that is not entirely bad: the version the gateway advertises to its clients is not hard-coded in LiteLLM, the SDK negotiates it. And there is partial implementation of the intermediate revision, because the elicitation and sampling handlers reference it in their docstrings. That is: partial implementation of one revision while advertising an earlier one.

In practical interoperability it does not hurt yet, because the client ecosystem is still mostly on the 2025 revisions. It hurts in planning: July’s stateless redesign invalidates the gateway’s session model, and that is work ahead.

Two ecosystem notes for anyone choosing. There are clients that speak remote streamable HTTP natively (Claude and Claude Code, ChatGPT with OAuth and dynamic registration, VS Code with Copilot, Cursor, Zed, Kiro) and others that still demand stdio or a local bridge with mcp-remote, among them Windsurf, Cline and Continue.

Tool execution from the proxy itself

Besides serving MCP outwards, the proxy can consume it itself. It is enabled with an mcp block in the body, with server_url equal to the litellm_proxy sentinel, and it works both on /v1/responses and on /chat/completions.

The flow: it separates the gateway’s tools, lists them, filters by allowed, deduplicates, translates them into OpenAI function format and concatenates them. It calls the model. If there are tool calls and auto-execution is permitted, it runs them against the MCP servers. It builds the follow-up and makes one second call.

Two limits you need to know before designing on top of this.

The loop is a single hop. There is no multi-step iteration. The calls from the first response are executed and the follow-up is returned. A plan that chains tools requires the client to call again.

Auto-execution fails closed and as a block. It requires all MCP references to have approval set to “never”; a single one requiring approval disables auto-execution for the whole request.

And a third one, which is the one that bites: the multi-turn state of /v1/responses is rehydrated from the spend table. The query is literal, it looks up the session of the previous request_id and retrieves all the rows of that session ordered by time.

With disable_spend_logs: true nothing is written, the query comes out empty and the history is lost in silence, with no 4xx error. The code itself acknowledges it: it does not expect retries because deployments that do not write spend logs have nothing to wait for. It is a coupling between a performance option and a conversation feature that nobody expects to find.

Health and day-to-day operation

Health checks exist, they are on demand and they are not periodic. GET /v1/mcp/server/health accepts repeated identifiers and returns per-server status. The implementation opens a client and runs an empty operation inside the session, with a ten-second timeout.

It skips the check entirely if the server requires per-user authentication, and those always return unknown status.

The surprising part: the status, last check and error columns have existed in the database since a May 2025 migration, and the check function returns an object without writing to the database. Status is recalculated on every request. There is no periodic MCP health job.

When an upstream server is down, the behaviour is to degrade and not block. The listing fans out in parallel and each failing server contributes an empty list plus a result classified into six categories. Those results reach the client in the response’s _meta under a vendor key, without filtering the upstream’s text or URL. The reason for the design is in the code: an empty contribution with no signal makes a broken upstream indistinguishable from a healthy server with no tools.

There is no cache of the previous list. A server that is down disappears from that turn’s tools.

On registering and deregistering without restarting, there are two worlds. Database servers can be created, edited and deleted through the API, and every operation reloads the registry with an atomic swap. Propagation between replicas goes through Redis publication on the configuration change channel, if there is Redis, and through polling every 30 seconds if there is not.

Servers declared in config.yaml are loaded once, at start-up. There is no re-read. Changing them requires a restart.

And there is a precedence trap between the two worlds, with two different warnings in the code. A database row with the same identifier as a configuration entry hides the configuration one entirely. And the other way round, a configuration identifier that matches the name or alias of a database server captures its permissions, because identifiers are resolved before names. Both cases emit a log warning, once.

A hardened `config.yaml`

general_settings:
 # Access is not open by default, but this closes the empty-key
 # inheritance towards the team, which is the one that surprises people.
 require_key_mcp_access_defined: true
 require_end_user_mcp_access_defined: true
 # Without this, allow_all_keys overrides the key's explicit scope.
 mcp_allow_all_keys_respects_mcp_scope: true

litellm_settings:
 # Does not cover MCP arguments. See the auditing section.
 turn_off_message_logging: true
 callbacks: ["langfuse_otel"]

mcp_servers:
 inventario:
 url: https://mcp-inventario.interna.svc/mcp
 transport: http
 auth_type: bearer_token
 auth_value: os.environ/MCP_INVENTARIO_TOKEN
 # Explicit allowlist. Also enforced on tools/call.
 allowed_tools:
 - buscar_articulo
 - consultar_stock
 # Never true on a server with data.
 allow_all_keys: false
 access_groups: ["operaciones"]
 timeout: 20
 max_concurrent_requests: 8
 mcp_info:
 description: "Inventario de almacen, solo lectura"
 # Without this, MCP spend is zero and its metric does not exist.
 mcp_server_cost_info:
 default_cost_per_query: 0.0005
 tool_name_to_cost_per_query:
 consultar_stock: 0.0002

And in the process environment:

# Without this there is no MCP span, only a generic litellm_request.
LITELLM_OTEL_V2=true
# Full block hashes, not truncated, if correlating
# with engine events.
LITELLM_MCP_CLIENT_TIMEOUT=45
LITELLM_MCP_TOOL_LISTING_TIMEOUT=20

For clients, the rule is not to use the aggregated endpoint. Each agent points at /{server}/mcp or at a tool set, or at the three-tool proxy mode if the catalogue is large.

Checklist

Check the version. Below 1.83.14 there is remote execution and an unauthenticated SQL injection with an assigned CVE.
Inventory which servers have allow_all_keys and enable the setting that makes it respect the key’s scope.
Audit the access groups separately, because they are additive on top of the key and team ceiling.
Declare in writing the fail-open behaviour on tool permission resolution errors.
Measure the real cost: number of tools per server from the listing row, and tools block tokens from the stored request body.
Choose a catalogue strategy before connecting the third server: three-tool proxy mode, semantic filter, or selection by route. Not the aggregated endpoint.
Declare mcp_server_cost_info even with a token value, or the spend metric will not exist.
Enable the OTel v2 logger, and decide deliberately whether to enable content capture.
Assume the trace is cut at the gateway and plan correlation with the upstream by other means.
Treat the spend table’s metadata column as a personal data repository: encryption, its own retention and separate access control.
Set up the rug pull check, because nobody does it: hash of the description and schema of every approved tool, periodic comparison against the listing.
Restrict egress towards MCP servers at the network level, because the gateway does not validate the destination URL.
Do not enable the stdio transport unless you need it, and if you do, review what is on the command allowlist.
If you use /v1/responses with multi-turn state, do not enable disable_spend_logs.

Traps and things that are not what they seem

MCP access is not open by default in 1.102.0. I got that wrong in the previous post. What opens things up is per-server allow_all_keys.
allow_all_keys overrides the key’s explicit scope unless you enable a setting that ships disabled.
require_key_mcp_access_defined only closes key-to-team inheritance. Access group grants remain additive.
On an indeterminate failure resolving tool permissions, the code fails open for keys and JWTs.
The default cost of an MCP call is 0.0, and the spend metric is only emitted if the cost is greater than zero, so the series does not exist.
MCP calls appear in the per-team report as if they were models, named MCP: tool and with zero tokens.
The OTel logger with MCP support is off by default. Without the environment variable there is no MCP span.
Trace context is not propagated towards the upstream MCP server. There is no end-to-end trace.
mcp.tool.name does not exist in the semantic conventions. The name goes in gen_ai.tool.name.
Message redaction does not cover tool arguments. They are written in the clear into the spend table.
Registering, modifying and deleting MCP servers generates no audit log. There are three TODOs in the code.
The semantic tool filter is not applied on tools/list. Only on /chat/completions and /v1/responses.
The catalogue order is not deterministic, because it comes from iterating a set of strings. It breaks the client’s prompt cache even though nobody filters anything.
search_tools in proxy mode counts keywords by default, it uses neither regular expressions nor embeddings, and its cap of five results is hard-coded.
Description overrides really do replace what the client sees, and they are the only catalogue lever that does not break the cache. They are only configurable through the database or the API.
spec_version no longer exists in the configuration schema.
/mcp/sse does not use the SSE transport, and the messages endpoint its object advertises is dead code.
Listing pagination truncates in silence when the deadline expires, without leaving a single log line.
A database row hides a configuration entry with the same identifier, and the other way round a configuration identifier can capture the permissions of a database server.
Servers declared in config.yaml require a restart. Only the database ones reload hot.
The health check does not write its result to the database even though the columns exist, and there is no periodic job.
The gateway does not validate the upstream MCP server’s URL. The anti-SSRF utilities exist and are not used on that path.
docker and npx are on the stdio command allowlist.
The /v1/responses tool loop is a single hop, and disable_spend_logs: true breaks its multi-turn behaviour with no error.
LiteLLM advertises the June 2025 revision of the protocol, two behind the one in force.

Closing

There is a thread running through the six posts in this track that is more visible here than anywhere else. The gateway is chosen for one job and ends up doing five, and each new job arrives with its own defaults, which are almost never the ones you would have chosen.

In the case of MCP the asymmetry is twofold. On one side, the part that looks dangerous is better built than you would assume: access control is enforced on execution and not just on the listing, the hierarchy is built out of intersections, header forwarding is an allowlist, and default access is closed, contrary to what I myself wrote two days ago. On the other, the part nobody looks at is worse: tool arguments come out in the clear bypassing the privacy switch, spend is worth zero and its metric never gets to exist, the trace is cut, and a change to the description of an approved tool goes undetected by anybody in the whole chain, neither the specification nor the gateway.

The architecture decision that comes out of this is that the MCP door has to be treated as what it is, an interconnection of systems with third parties, and not as one more function of the LLM proxy. That means its own inventory, formal registration, change control over every tool’s description, network isolation towards the upstream and a log designed knowing that it is going to contain personal data.

And it means accepting that the cost you really pay for connecting fifteen MCP servers is in no column of the spend table. It is spread across the prompt_tokens of every model call, on every turn, and across the accuracy you lose when the catalogue is too large for the model to choose well. That number can be computed with what the gateway already stores. The first step is wanting to compute it.

Sources

LiteLLM, MCP Gateway: https://docs.litellm.ai/docs/mcp.
LiteLLM, código: litellm/proxy/_experimental/mcp_server/server.py, mcp_server_manager.py, rest_endpoints.py, auth/user_api_key_auth_mcp.py, cost_calculator.py, faults/list_outcomes.py, litellm/proxy/management_endpoints/mcp_management_endpoints.py, litellm/proxy/spend_tracking/spend_tracking_utils.py, litellm/litellm_core_utils/redact_messages.py, litellm/integrations/otel/, litellm/responses/litellm_completion_transformation/session_handler.py: https://github.com/BerriAI/litellm.
LiteLLM, incidencia 34534, fuga de slot de concurrencia en llamadas MCP: https://github.com/BerriAI/litellm/issues/34534.
NVD, CVE-2026-42271, ejecución de comandos por los endpoints de prueba de MCP: https://nvd.nist.gov/vuln/detail/CVE-2026-42271.
NVD, CVE-2026-42208, inyección SQL sin autenticación en la validación de clave: https://nvd.nist.gov/vuln/detail/CVE-2026-42208.
NVD, CVE-2026-35030, colisión de clave de caché de userinfo OIDC: https://nvd.nist.gov/vuln/detail/CVE-2026-35030.
NVD, CVE-2025-49596, ejecución remota en MCP Inspector: https://nvd.nist.gov/vuln/detail/CVE-2025-49596.
NVD, CVE-2025-6514, inyección de comandos en mcp-remote: https://nvd.nist.gov/vuln/detail/CVE-2025-6514.
NVD, CVE-2025-66416, protección de DNS rebinding desactivada por defecto en el SDK de Python: https://nvd.nist.gov/vuln/detail/CVE-2025-66416.
Invariant Labs, MCP Security Notification: Tool Poisoning Attacks (tool poisoning, rug pull, shadowing): https://invariantlabs.ai/blog/mcp-security-notification-tool-poisoning-attacks.
Invariant Labs, Introducing mcp-scan (fijado de herramientas por hash): https://invariantlabs.ai/blog/introducing-mcp-scan.
Model Context Protocol, changelog de la revisión 2026-07-28: https://modelcontextprotocol.io/specification/2026-07-28/changelog.
Model Context Protocol, changelog de la revisión 2025-11-25: https://modelcontextprotocol.io/specification/2025-11-25/changelog.
Model Context Protocol, Security Best Practices (confused deputy, indicadores de recurso, validación de audiencia): https://modelcontextprotocol.io/docs/2026-07-28/tutorials/security/security_best_practices.
NSA, Model Context Protocol: Security Design Considerations for AI-Driven Automation, mayo de 2026: https://media.defense.gov/2026/Jun/02/2003943289/-1/-1/0/CSI_MCP_SECURITY.PDF.
CISA y agencias aliadas, Careful Adoption of Agentic AI Services, mayo de 2026: https://www.cisa.gov/resources-tools/resources/careful-adoption-agentic-ai-services.
Anthropic, Advanced tool use (coste en tokens de las definiciones, búsqueda de herramientas, precisión de selección): https://www.anthropic.com/engineering/advanced-tool-use.
Anthropic, Code execution with MCP: https://www.anthropic.com/engineering/code-execution-with-mcp.
Anthropic, Prompt caching (jerarquía tools → system → messages e invalidación): https://platform.claude.com/docs/en/build-with-claude/prompt-caching.
Anthropic, Tool search tool (herramientas diferidas fuera del prefijo, umbral de 30 a 50 herramientas): https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool.
OpenAI, Prompt caching (el orden de las herramientas cuenta): https://developers.openai.com/api/docs/guides/prompt-caching.
OpenAI, Tool search (las herramientas descubiertas se cargan al final del contexto): https://developers.openai.com/api/docs/guides/tools-tool-search.
vLLM, Automatic Prefix Caching: https://docs.vllm.ai/en/latest/features/automatic_prefix_caching.html.
Microsoft, Use tools in chat (límite de 128 herramientas por petición en VS Code): https://code.visualstudio.com/docs/copilot/agents/agent-tools.
RAG-MCP, prueba de estrés de 1 a 11.100 servidores candidatos, mayo de 2025: https://arxiv.org/abs/2505.03275.
Benchmarking the Benchmarks, desalineación del 18,5 % entre etiqueta y realidad, junio de 2026: https://arxiv.org/html/2607.02577v1.
MCP-Atlas, taxonomía de fallos sobre 220 herramientas y veinte modelos, febrero de 2026: https://arxiv.org/html/2602.00933v3.
BiasBusters, sesgo de selección entre herramientas equivalentes y peso de la descripción frente al nombre: https://arxiv.org/html/2510.00307.
MCP Tool Descriptions Are Smelly, auditoría de 856 herramientas de 103 servidores, febrero de 2026: https://arxiv.org/html/2602.14878v1.
ToolMenuBench, seis estrategias de filtrado y 26.460 ejecuciones, junio de 2026: https://arxiv.org/html/2606.15508.
Retrieval Models Aren’t Tool-Savvy (ToolRet), corpus de 43.000 herramientas, marzo de 2025: https://arxiv.org/abs/2503.01763.
Tools Are Not Islands, brecha entre Recall@3 y conjunto completo, julio de 2026: https://arxiv.org/html/2607.25718.
ToolFlood, saturación adversaria de la capa de recuperación, marzo de 2026: https://arxiv.org/html/2603.13950.
Scores Are Not Decisions, el K óptimo como función del coste, julio de 2026: https://arxiv.org/html/2607.27083v1.
Don’t Break the Cache, caché de prompts en tareas agénticas de horizonte largo, enero de 2026: https://arxiv.org/html/2601.06007v2.
How Many Tools Should an LLM Agent See? A Chance-Corrected Answer, mayo de 2026: https://arxiv.org/html/2605.24660v1.
Semantic Tool Discovery for LLMs, marzo de 2026: https://arxiv.org/abs/2603.20313.
OpenTelemetry, convenciones semánticas de GenAI y MCP: https://github.com/open-telemetry/semantic-conventions-genai.
Boletín Oficial del Estado, Real Decreto 311/2022, Esquema Nacional de Seguridad: https://www.boe.es/buscar/act.php?id=BOE-A-2022-7191.
IBM ContextForge, gateway y registro MCP: https://github.com/IBM/mcp-context-forge.
Docker MCP Gateway: https://github.com/docker/mcp-gateway.
agentgateway: https://github.com/agentgateway/agentgateway.

Prefix routing: what LiteLLM does not do, who does, and how much it really improves

Fri, 11 Sep 2026 00:30:00 +0200

Fifth article in the operational track of the control layer. The pair with Langfuse covered observability, the proxy’s day 2 covered availability, virtual keys covered governance and humans and agents covered the coexistence of two classes of traffic. Here a claim of mine from June gets taken apart. Everything that follows is verified against LiteLLM 1.102.0 (commit of 10 September 2026), vLLM 0.29.0 and the main branches of production-stack, llm-d and Dynamo from the second week of September 2026.

TL;DR

LiteLLM does not do prefix-aware routing, neither in the open version nor in the paid one. It is not a licensing question. A search across the whole tree for kv_aware, cache_aware, radix, prefix_cache and block_hash returns zero matches in the router. Nor is there any call from the router to /metrics or to any vLLM endpoint. In June I wrote the opposite in the inference router article and I was wrong.

What LiteLLM has is affinity, which is a different thing. Four pre-call checks pin a request to a replica by session identifier, by key hash, by previous response identifier or by exact hash of a marked prefix. None is active by default. Affinity distributes conversations; prefix routing distributes prefixes shared between different conversations, which is the case that multiplies throughput.

The only one that looks at the prompt is fragile in three ways. If the client does not send cache_control, the key comes out null and the check does nothing. If the cut point goes in the last turn, which is what the automatic injection does by default, the key changes on every turn and it never hits. And the pin’s TTL is 300 seconds hardcoded in two lines, while the provider offers one-hour caches.

The engine does not answer questions, it publishes events. There is no vLLM endpoint you can ask whether it has a prefix. What there is, is a ZMQ socket that emits stored and removed blocks in msgpack, switched off by default, with silent discarding when the consumer does not keep up. A router that wants to be exact has to replicate the engine’s hash in full: chained SHA256, deterministic seed, block granularity and the extra keys in their order.

And by default it does not publish the hits, only the additions. The index of a router that only listens drifts away from the engine’s real LRU without noticing. To see the hits you have to ask for it request by request with kv_cache_report_mode: full.

The three routers that do do it are very different from each other. The vLLM production-stack one indexes chunks of 128 raw characters in a trie that is never purged, models no eviction and dies on every restart. The llm-d one offers an approximate mode and an exact event-driven one, all its plugins are in alpha or beta, and its default tokeniser does not produce real tokens. NVIDIA Dynamo’s keeps an event-driven radix tree with a weighted cost function, and ships with the parameter that avoids starving a freshly scaled replica switched off.

The big improvements only appear in the best possible case. The two- and three-digit numbers are published by the projects themselves, with synthetic shared prefixes and a round-robin balancer as the reference. The most complete independent measurement, from Meta, obtains 2.3 times more throughput and also identifies load regimes where affinity leaves capacity at between half and two thirds of the reference. Another, on a multi-region deployment with 89 % reuse, measures single-digit improvements.

The correct architecture has two layers, not one. LiteLLM on top for identity, budget and audit, pointing at a single destination per model group. The KV-aware router underneath, inside the engine’s domain. Mixing it into a single piece is not an option that exists today.

You are here: between the gateway and the engine

This article occupies the gap between two layers that already have an article of their own. Above, the L7 switchboard and the gateway that has been operated since the first article in this track. Below, the fundamentals of the KV cache and hit rate engineering.

The gap is the piece that decides which replica each request goes to, knowing what each replica has in memory. It is the only function in the stack that cannot be resolved in the gateway or in the engine separately.

The analogy: the library with four counters

A library with four counters. Each librarian has in front of them a pile of the books they have consulted lately. When a reader comes in asking for the third volume of an encyclopaedia, if that volume is in the pile at counter two, serving them there takes seconds. At any other counter someone has to go down to the store room.

A normal balancer sends the reader to the counter with the shortest queue. It is the right decision if all the counters are interchangeable, and it is the wrong decision here, because they are not: what distinguishes them is not the queue, it is the pile.

Session affinity resolves one specific case of this. The returning reader is always sent to the same counter, because they will probably ask for the fourth volume of the same encyclopaedia. It works, and it does not resolve the general case: when a hundred different readers ask for the same third volume, session affinity spreads them across the four counters and four librarians go down to the store room for the same book.

Prefix routing is looking at what the reader asks for before assigning them a counter. It has two costs that the analogy makes visible straight away. The first is that somebody has to keep track of what is in each pile, and that bookkeeping takes work. The second is that if everybody asks for the third volume, counter two becomes the library’s only desk while the other three look on.

The correction: what I wrote in June

In the inference router article, on 2 June 2026, I closed the prefix routing section with this sentence: “vLLM Production Stack router implements it natively. NVIDIA Dynamo too. LiteLLM in its enterprise version has a beta.”

The first two hold up. The third is false and was no less false in June. While preparing the humans and agents article the first hint appeared, and full verification against 1.102.0 confirms it.

Case-insensitive search over litellm/, enterprise/ and gateway/, for the terms kv_aware, kv-aware, cache_aware, cache-aware, radix, prefix_cache, prefix-cache, kv_cache, block_hash, prefix_hash, prefix_score and longest_prefix: zero matches. Search for vllm inside litellm/router.py, litellm/router_strategy/ and litellm/router_utils/: zero. Search for num_requests_waiting, gpu_cache_usage or any scraping of the backends’ /metrics from the router: zero. LiteLLM’s router never asks the engine anything.

Search for premium_user in the files where those checks would live: a single result, and it is about something else. There is no paid version with this feature. The only routing piece that is behind a licence is per-tag budgets, in litellm/router_strategy/budget_limiter.py.

Where the error came from. LiteLLM’s documentation has an entry on dynamic TPM and RPM allocation marked as beta and enterprise, and a pre-call check called prompt_caching. Skim-read and added together, they give the impression of prefix routing in beta. Read in the code, prompt_caching is something else, and it deserves a section of its own because it is still recommended for this problem.

What LiteLLM does have: four affinities

The optional_pre_call_checks field accepts eight values. None is enabled by default: apply_default_settings passes an empty list. One of the eight, forward_client_headers_by_model_group, is declared in the type and in the interface schema and is consumed by no branch of the code. It is a dead value.

The four that pin a request to a replica are these.

Check	What it pins	Where it stores it	TTL
`session_affinity`	Session identifier	Redis with a Lua script, mirrored in memory	`deployment_affinity_ttl_seconds`, 3600 by default, refreshed
`deployment_affinity`	API key hash	Same	Same
`responses_api_deployment_check`	`previous_response_id`	Same	Same
`prompt_caching`	Exact hash of a marked prefix	`DualCache`	300 s, hardcoded

The other four are router_budget_limiting, encrypted_content_affinity, enforce_model_rate_limits and the dead value.

`prompt_caching`: why it is no use for this

The key is a SHA256 over the serialisation of the cacheable prefix, and “cacheable prefix” means everything up to and including the last block marked with cache_control: {"type": "ephemeral"}. Three consequences, checked by running the functions in isolation over the repository’s code.

If the request carries no cache_control, the extraction returns an empty list, the key comes out null and the check pins nothing. Automatic injection of cut points is only enabled with litellm.enable_anthropic_prompt_caching, which defaults to false, and only for Anthropic and Bedrock providers.

If the cut point is in the system prompt, the key does not change when turns are added and the pin works. This is the good case and it requires the client to mark it deliberately.

If the cut point is in the last message, the key changes on every turn and the pin never hits. And this is exactly the default case of the automatic injection, which places one point on the system and another at index -1. Since the second is the last, the cacheable prefix becomes the whole conversation.

On top of that there is the TTL. Two lines with ttl=300, one of them with the comment # store for 5 minutes. There is no parameter to change it. Issue 28427 has had it open for months with no response from the maintainers, and it points to the mismatch with the one-hour ephemeral caches that LiteLLM itself knows how to request via cache_control.ttl: "1h".

There is a nuance the issue does not capture and that makes diagnosis worse: in the default case the pin does not even last five minutes, because the key changes first.

`session_affinity`: the one to use

For conversations, this is the right piece and it is reasonably built. The session identifier is looked for in this exact order of headers:

x-litellm-trace-id
x-litellm-session-id
Any x-<vendor>-session-id, with the regular expression ^x-.+-session-id$ and a value of at least eight alphanumeric characters
Only if the User-Agent is Codex’s: session-id, session_id, thread-id, conversation_id
Plain x-session-id, for opencode

And if nothing matches, two fallbacks: a metadata.user_id in Anthropic format, and the W3C baggage with session.id=. The third point is the one that captures Claude Code with nothing configured.

Storage is a Lua script in Redis that does GET, SET NX EX and EXPIRE, so the first writer wins and the TTL is refreshed on every request that confirms the pin. It is an inactivity TTL, not a total session duration. Without Redis it degrades to a pod-local reservation, which is atomic but not shared.

A detail of the pipeline order that already came up in the previous article and still holds: the affinity checks run inside async_get_healthy_deployments before tag filtering. If the pin returns a single deployment and that deployment does not satisfy the request’s tag, the later filter removes it and the request fails. Affinity gets no vote on the tags.

The routers that do read the prompt, and why they are no good

LiteLLM has four pieces that read message content: AutoRouter with semantic matching, RequestComplexityRouter, AdaptiveRouter and QualityRouter. They all live in async_pre_routing_hook, which runs before looking for healthy deployments.

The reason they do not solve this problem is in what they choose. All four choose a model group, not a replica. They decide whether an easy question goes to the thirty-billion-parameter model or to the small one. None has access to the replica list or knows what is in each one’s memory.

What did land: admission control

A minor correction to the previous article, which placed this in the 1.101 branch as an upcoming feature. In 1.102.0 it is already there, in litellm/proxy/middleware/admission_control_middleware.py, with the three settings in general_settings: max_in_flight_requests_per_worker, max_queued_requests_per_worker and admission_queue_timeout_seconds, the last with a default value of 1.0 seconds.

It stays inactive as long as the first one is not declared. It exposes an authenticated /health/backlog and the three metrics litellm_admission_admitted_requests, litellm_admission_queued_requests and litellm_admission_rejected_requests_total, the last with a reason label. It is a per-process semaphore, not coordinated across pods.

What the engine exposes upwards

Before looking at the routers that do do this, you have to understand what raw material they work with. The natural question would be whether vLLM has an endpoint you can ask about a prefix. It does not.

The only cache router in the entry points is vllm/entrypoints/serve/dev/cache/api_router.py, and it only exposes destructive writes: POST /reset_prefix_cache, /reset_mm_cache and /reset_encoder_cache. All three are behind VLLM_SERVER_DEV_MODE=1 and do not exist in a normal deployment. There is no read.

What there is, is an event stream.

The ZMQ stream

It is enabled with --kv-events-config, and it is off by default. The configuration accepts a publisher (null or zmq), an endpoint (by default tcp://*:5557, which binds), an optional replay endpoint, the number of batches retained for that replay (10,000), and a high water mark limit of 100,000.

That limit is the one that matters operationally: above it, ZMQ discards events if the consumer does not keep up. Silently.

The transport is a PUB socket with msgpack serialisation and three-part frames: topic, eight-byte sequence number and payload. The sequence number lets a subscriber ask for replay from where it left off through the optional ROUTER socket. With data parallelism the port shifts by rank, so you have to subscribe to 5557+rank.

Three event types. BlockStored carries the hashes, the parent hash, the token identifiers, the block size, the LoRA adapter name, the extra keys, the medium (GPU, CPU or storage) and a session identifier. BlockRemoved carries only hashes. AllBlocksCleared carries no payload and is emitted when the prefix cache is reset.

They are published once per scheduler step, only if there are events. With the feature switched off the cost is zero. With it switched on, the full token identifiers of each block travel over the wire, so bandwidth scales with tokens per second and not with requests per second.

An example in vLLM’s own repository declares itself experimental on its eighth line.

The hash that has to be replicated

An exact router does not receive the engine’s state, it reconstructs it. For that it has to compute the same hashes, and the engine computes them like this:

# vllm/v1/core/kv_cache_utils.py
def hash_block_tokens(hash_function, parent_block_hash, curr_block_token_ids, extra_keys):
 if not parent_block_hash:
 parent_block_hash = NONE_HASH
 return hash_function((parent_block_hash, tuple(curr_block_token_ids), extra_keys))

It is chained: each hash identifies the complete prefix up to that boundary, not the isolated block. The default algorithm is SHA256, configurable with --prefix-caching-hash-algo among four values.

Five things a router has to get right for its hashes to match the engine’s:

The seed. Since 0.29, NONE_HASH is deterministic from a fixed seed, so two different processes produce the same hashes with nothing configured. It is a silent change of semantics from earlier versions, which required setting PYTHONHASHSEED. With the xxhash algorithms it is still needed.

The block size. By default 16 tokens, but the attention backend can override it if the user did not pass --block-size: 256 with sliding-window sparse attention, 64 with ROCm’s AITER, 64 or more with FlashAttention on XPU. A router must not assume 16, it has to read it from the event.

The hashing granularity, which is not necessarily the physical block size. There is --prefix-match-unit, which with several cache groups resolves to the greatest common divisor of the cacheable groups’ sizes.

The extra keys, in their order: LoRA adapter name (by name, not by identifier), multimodal identifiers with their relative offset inside the block, cache_salt only in block zero, and the hash of the prompt embeddings. Sampling parameters do not go in, which is correct: the prefix KV does not depend on the temperature.

And truncation. By default the hashes travel over the wire as 64-bit integers truncated from a SHA256, not as the full bytes. It can be disabled with VLLM_KV_EVENTS_USE_INT_BLOCK_HASHES=0. With the default value, a router has a theoretical collision risk that the engine does not have.

The hits are invisible by default

This is the detail that takes the most effort to discover and has the most consequences. The events are incremental: only what gets newly cached is announced. A reused block generates no event.

A router that only listens sees the additions and the removals, but does not see the hits. Its notion of what is recent and what is cold drifts away from the engine’s real LRU with nothing to warn it.

A full mode exists, and it is per request. sampling_params.extra_args["kv_cache_report_mode"] = "full" makes BlockStored events be emitted for blocks that were hits too. The function’s docstring says so plainly: it generates events so that external consumers, “for example a gateway”, know which blocks were reused. That is to say: for this to work, the router has to inject a parameter into every request it forwards.

The metrics, and what broke

For aggregate measurement two counters remain: vllm:prefix_cache_queries_total and vllm:prefix_cache_hits_total, both in tokens and not in requests, with model and engine labels. There are two equivalents for the external connector, which allow a hit in HBM to be distinguished from one brought in from CPU or NVMe.

And there are two removals that break existing configurations. vllm:gpu_prefix_cache_hit_rate no longer exists: zero appearances in the repository. It was deprecated in 0.8, hidden in 0.9 and removed in 0.10. And vllm:gpu_cache_usage_perc was renamed to vllm:kv_cache_usage_perc, which breaks Grafana panels and autoscalers that depended on the old name.

The hit rate has to be derived in PromQL with rate() over the two counters. The one that appears in the text log is something else: it uses a window of the last thousand requests.

And it has a bias you need to know about before taking decisions with it. When a request is evicted, its blocks return to the free queue keeping the hash, so the later reprefill finds them again and counts as a hit. A high hit rate can mean useful reuse between users or it can mean the engine is evicting and recomputing in a loop. The internal statistics separate the two things with a preempted field, which is not exposed as a Prometheus metric.

The last token

A small detail with an effect on long conversations: the maximum hit length is the request’s token count minus one. There is always something to recompute in order to get logits. Since the reservation requires alignment to the block size, in practice that can force recomputing the whole final block. With blocks of 16 it is noise. With a backend that prefers 256, it is up to 256 tokens recomputed per turn.

The ones that do do it

vLLM production-stack

Eight routing logics in a single --routing-logic selector: roundrobin, session, kvaware, loadaware, prefixaware, disaggregated_prefill, disaggregated_prefill_orchestrated and priority. Two of them, loadaware and priority, are not in the Helm chart’s enum and are only enabled through extraArgs.

The naming trap is that prefixaware and kvaware are radically different things.

prefixaware is a hash trie in the router’s process. It chunks the prompt string every 128 characters, not tokens, and hashes each chunk with 64-bit xxhash. Its own docstring says so: the chunk size is “in number of characters”. That 128 is hardcoded, there is no flag or Helm value to change it.

Four consequences come out of that. --prefix-min-match-length is measured in characters and is quantised to multiples of 128. A shared prefix that differs by one character inside the first chunk breaks the whole match, because there is no partial match inside a chunk. For chat, it concatenates the content of all the messages with newlines and without applying the chat template, so the indexed string is not what the engine tokenises. And the tie-break between tied replicas is random.choice, without looking at load.

The trie has no notion of saturation. A popular prefix concentrates all the traffic on the same replica indefinitely. The docstring assumes the simplified model and declares it: “we assume there is no prefix cache eviction”.

It has no pruning either. HashTrie only has insertion and longest prefix lookup; there is no deletion method. When a replica disappears, its URLs stay forever in every node’s sets, and filtering happens at query time. It works, and the trie grows monotonically: each distinct prompt adds its length divided by 128 permanent nodes, with no bound, no TTL and no LRU. The router’s memory limit in the chart is 1,000 MiB. By contrast, the round-robin router in the same file does bound its caches to 1,024 entries.

And it dies on every restart: it is in-process memory state, with no snapshot and no initial load.

kvaware and loadaware are better and have a different price. They do not index themselves: they ask the LMCache controller, which knows about real tokens. That requires lmcache==0.3.11 and vllm==0.13.0 pinned exactly, and the official router image drags in the whole of vLLM. The limitation to read before planning anything: they only look at request_json.get("prompt", ""), with an open TODO for chat completions. With /v1/chat/completions, both tokenise an empty string.

loadaware is the only one of the three that breaks affinity on saturation, with the formula score = relative_match - beta * relative_load and beta = 1.0 by default. The authors’ own reading: a replica at twice the average load loses the equivalent of a full cache hit. The guide suggests 0.25 to favour cache and 2.0 to favour balancing.

On published results, the documentation’s benchmarking file literally says that the measurement platform is coming soon. The project blog’s two posts claim between three and ten times lower response latency, and only one declares a partial configuration: Llama 3.1 70B on four nodes with tensor parallelism 2 over 80 GB A100s, inputs of 9K tokens and outputs of 10, against AIBrix 0.2.0 and a flat deployment. Charts with no labelled points, no absolute figures and no declaration of which routing logic was used in each.

llm-d

There has been a reorganisation here that invalidates a good part of the third-party documentation. The Endpoint Picker code left the Gateway API Inference Extension repository and was merged into llm-d’s, which has been renamed to llm-d Router. The Kubernetes project keeps the InferencePool API and a lightweight reference EPP that does nothing about prefixes. If you look for KV-aware routing in the extension’s repository, today it is not there.

The InferencePool is at v1 and is GA. The other two APIs moved to the llm-d.ai group at v1alpha2, and when they coexist with the old ones the EPP prefers the new ones and ignores the old.

Scoring is a weighted sum. The default weights, when no configuration is passed: queue 2, KV utilisation 2, prefix 3. Any scorer referenced without a weight counts as 1.0. There is no configuration through environment variables: the old ENABLE_*_SCORER no longer exist, everything goes in an EndpointPickerConfig in YAML.

The prefix scorer consumes an attribute published by a producer, and there are two producers.

The approximate one computes the hashes in the EPP itself without talking to the engine, and keeps a local LRU per pod and block pair that is populated after the decision. It is an approximation by construction. It has a hard floor: a configured block size below 64 tokens is raised to 64 at request time, because the index stores one entry per pod and block and going down to 16 would multiply memory by four.

The exact one builds the real index from the engine’s ZMQ events, with one subscriber per pod that is installed and torn down with the endpoint’s registration and deregistration. It has adapters for vLLM and SGLang, deduplicates evictions by reference counting, and the replay buffer that allows the index to be rebuilt after an EPP restart requires vLLM 0.26.0 or above.

The trap is in the tokeniser. The EPP tokenises through a plugin, and the default backend is estimate: byte packing with no tokeniser, roughly 4 bytes per token, with identifiers that do not correspond to real engine tokens. The exact scorer needs real tokens, and for that you have to explicitly configure the vllm backend, which does HTTP against the engine’s /v1/completions/render endpoints. If it is omitted, the auto-created estimate satisfies the dependency and the system degrades silently.

On maturity, the file itself declares it: all in-tree plugins are in alpha or beta, and they will be promoted to stable when the project approaches 1.0. An alpha plugin configured without the corresponding flag makes the EPP fail to start.

And on replicas, the warning is explicit: active-active mode should be avoided with approximate prefix routing, because the EPP replicas do not share state and each one only sees the traffic it has served itself, which significantly degrades the hit rate. The exact index is safe in high availability, and even so the reference guides fix a single replica because the accounting of in-flight requests is local to the process.

The cost is the best documented in the whole field, and it deserves a table. Simulator with Qwen3 8B, 100 pods, 100K input tokens:

Requests/s	Cap on tokens to match	Peak CPU	Peak memory
5.0	4,096	1.19 cores	0.26 GiB
5.0	100,000	3.82 cores	0.65 GiB
98.7	4,096	35.17 cores	2.46 GiB
98.8	100,000	46.50 cores	3.41 GiB

With outputs of 10K tokens at 50 requests per second, 32.53 cores and 12.54 GiB. The maintainers’ sizing rule is half a core to one core per request per second on large agentic loads. And idle consumption scales with the number of pods because of continuous scraping: with 100 pods, about 7.5 cores with no traffic.

NVIDIA Dynamo

A precise event-driven index, with a global radix tree in each frontend’s process. Since all the replicas consume the same event plane, there is no need to synchronise routers with each other for the prefix state.

The cost function is explicit and weighted. It discounts from the prefill the blocks that overlap in GPU, host and disk with different weights (1.0 for the overlap credit, 0.75 for host, 0.25 for disk), and adds the potential decode blocks and the active requests.

There is a parameter to look at before deploying: --router-kv-overlap-score-credit-decay is at 0 by default, that is, disabled. Its description says what it is for: preventing loaded, cache-rich replicas from winning again and again while the freshly autoscaled ones receive too little traffic. With the default value, a new pod has no cache, therefore never wins the scoring, therefore never generates cache.

The approximate mode exists (--no-router-kv-events, with a TTL of 120 seconds) and the documentation says it is not the recommended route in production.

Dynamo is the most honest project in its measurement methodology: its comparison guide recommends contrasting --router-mode random against kv over the public Mooncake trace from FAST'25, with a declared cache ratio of 59 %, and it generates the table locally instead of publishing a number.

The rest of the field

SGLang has renamed its router to sgl-model-gateway, version 0.3.2 within SGLang 0.5.19. Its cache_aware policy is an approximate radix tree over raw characters, with switching to shortest queue when the system is unbalanced according to two combined thresholds. One detail that can cost an afternoon: the Rust defaults and the Python CLI defaults do not match, and on the maximum tree size they differ by four orders of magnitude (10,000 nodes against 2^26).

AIBrix 0.7.0 offers a standard mode with a local hash table and a KV event synchronisation mode behind a flag and a build tag, with vLLM 0.7.0 or above.

Ray Serve LLM has PrefixCacheAffinityRouter with an explicit alpha API warning, a character-based prefix tree in a decoupled actor, and its imbalance threshold at infinity by default, that is, out of the box it never breaks affinity on load.

Envoy AI Gateway 1.1.0 implements none of this and delegates to InferencePool and the EPP. KServe the same, with its LLMInferenceService at v1alpha1. Mooncake Conductor is still an unimplemented proposal.

Comparison table

Project	Version and date	Index	Where it lives	Kubernetes	Declared maturity
production-stack `prefixaware`	main branch, 09-Sep-2026	Approximate, character trie, no eviction	Router process	Optional	WIP in the README
production-stack `kvaware`/`loadaware`	same	Exact via LMCache controller	LMCache controller	Optional	No chat completions
llm-d Router approximate	v0.10.0, 17-Aug-2026	Approximate, LRU per pod	EPP process	Yes	Beta, active-active discouraged
llm-d Router exact	same	Exact through ZMQ events	EPP process	Yes	Beta
NVIDIA Dynamo	v1.4.2, 27-Aug-2026	Exact, event-driven radix tree	Frontend process	No	GA with experimental subsystems
SGLang `sgl-model-gateway`	0.3.2, 03-Sep-2026	Approximate, character radix	Gateway process	No	Default policy
AIBrix	v0.7.0, 16-Jun-2026	Both	Gateway plugin	Yes	Synchronisation behind a flag
Ray Serve LLM	Ray 2.58	Approximate, characters	Ray actor	No	Declared alpha
Envoy AI Gateway	v1.1.0, 21-Aug-2026	None of its own	Delegates to EPP	Yes	Stable 1.x API

How much it really improves

Here two types of source have to be separated.

The projects’ own numbers. Red Hat published the llm-d case in May: Qwen3 32B, eight vLLM pods over 16 H100s with tensor parallelism 2, synthetic shared-prefix load with 150 groups of five prompts, 6,000 tokens of system prompt, 1,200 of question and 1,000 of output. Result: up to 109 % more throughput and up to 99 % less TTFT, around 200 concurrent users within SLO against around 20. The repository’s result files give more detail: peak throughput from 6,986 to 14,892 output tokens per second, p90 TTFT from 135.5 seconds to 0.26.

That same file publishes the regression, and that is why it is worth reading: p50 inter-token latency rises by 22.4 % with vLLM and by 45.4 % with SGLang. The trade-off is declared: routing by affinity concentrates more concurrent work on the pods with a warm cache.

It is the best possible case. A synthetic shared prefix and a reference that is a flat Kubernetes Service.

The independent measurements. There are three and they say different things.

CacheRoute, from Meta, August 2026, is the most complete. Llama 3.3 70B in fp8, 30 targets with tensor parallelism 2 over 60 H100s, a semi-synthetic trace from production telemetry with 128,824 business keys, and all the alternatives reimplemented in the same harness instead of comparing against a dumb balancer. It obtains 176 requests per second with a p99 SLO of 3.5 seconds against 76 for the best competitor, and a hit rate of 93.2 % against 72.0 %.

And it publishes two negative findings that nobody else publishes. There are load regimes where affinity reduces capacity to between half and two thirds of the reference, to the point that the authors make a mirrored-traffic trial mandatory before deploying. And growing the fleet can cool a prefix even though total cache capacity grows, because the time between revisits stretches beyond the eviction window. Scaling horizontally can make the hit rate worse.

GORGO, June 2026, measures on a multi-region deployment with a dataset where intra-user prefix reuse is 89.4 % and the average prompt is almost 18,000 tokens. An improvement of 6.9 to 15.5 % in p95 TTFT. With extremely high reuse, the improvement is single-digit, because network latency and queuing dominate.

LAAR, from IBM Research Tokyo, April 2026, is the uncomfortable counterpoint: in its comparison, session affinity routing was the worst of the baselines, and at 64K contexts a load-based routing gives lower absolute latency at the cost of fewer hits.

The joint reading fits in three conditions. The big improvements require high, concentrated prefix reuse, long prompts, and a weak reference. Take away any of the three and the numbers deflate.

The five failure modes

Hot spot from affinity. The busiest bucket inherits the load’s skew, and tail latency starts following the hottest target instead of the fleet average. Mitigations exist in the four main products and in three of the four they ship switched off: Dynamo’s credit decay at 0, Ray’s imbalance threshold at infinity, and production-stack’s prefixaware with none at all.

Starvation of the freshly scaled pod. A new pod has no cache, so it never wins the scoring, so it never warms up. Dynamo documents it as the reason for the decay parameter. I found no primary documentation from any of the projects about the specific interaction with KEDA.

Cost of the index. The llm-d numbers in the previous table are the best public reference that exists. Raising the cap on tokens to match from 16,384 to 400,000 can more than double the EPP’s CPU with little traffic.

Traffic with unique prefixes. Each project degrades to something different. SGLang routes to the smallest tree, which is not the same as the least loaded. Ray falls back to power of two choices. Dynamo has an explicit preset, --load-aware, to say “I want the load model without reuse”. And production-stack’s prefixaware keeps inserting into the trie, that is, keeps paying the cost while receiving nothing.

Sibling bursts. Parallel sampling, best of N, or an agent that opens five branches at once. With event-driven routing, no engine has yet emitted the block stored event, all the siblings score zero overlap, and the prefix is precomputed on every replica. Dynamo has a prediction TTL for this; the others do not.

The architecture that comes out of all of the above

Two layers, with the responsibilities separated.

On top, LiteLLM, with what it does do well: non-forgeable identity through the virtual key, budgets, spend logging, concurrency limits, fallback between providers and API unification. For the gateway, each model group has a single destination: the URL of the router below.

model_list:
 - model_name: qwen-30b
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://kv-router.inferencia.svc:8000/v1
 api_key: os.environ/VLLM_KEY

router_settings:
 routing_strategy: simple-shuffle
 num_retries: 1
 optional_pre_call_checks: ["session_affinity"]
 deployment_affinity_ttl_seconds: 3600

general_settings:
 max_in_flight_requests_per_worker: 64
 max_queued_requests_per_worker: 64
 admission_queue_timeout_seconds: 1.0

Below, the KV-aware router, inside the engine’s domain, talking to the vLLM replicas and consuming their events.

Four consequences of this separation that have to be accepted before building it.

Per-replica visibility in LiteLLM is lost. For the gateway there is a single deployment, so its cooldowns, its per-deployment counters and its strategies stop meaning anything. Observability of which replica served what goes downstairs.

Retries stack up. The client SDK retries, LiteLLM retries, and the router below may have its own failover. In the previous article 45 calls per turn came out with three layers; with four it is worse. Lowering num_retries in the gateway to 1, or to 0, is the sensible decision when there is a router below that already retries.

LiteLLM’s session affinity stops making sense for the KV, because it no longer chooses a replica. It is kept if it is used for something else, such as pinning a conversation to a specific provider.

And the choice of the router below conditions the rest of the cluster far more than the gateway’s does. llm-d requires Kubernetes 1.32 or above, Gateway API, the inference extension and a conformant Gateway. Dynamo does not require Kubernetes. production-stack has a Helm chart and works outside it. That decision is the expensive one, not the gateway’s.

When not to build this

Three cases where the correct answer is not to do it, at least not yet.

A single replica per model. With one replica, the engine’s prefix cache already does all the work and there is nothing to route. That is the case for a good part of the four H100 deployments with one large model and tensor parallelism.

Low or scattered prefix reuse. Before building anything, the measurement is rate(vllm:prefix_cache_hits_total[5m]) / rate(vllm:prefix_cache_queries_total[5m]) per replica, with a mental correction for the bias from reprefills after eviction. If it is already high with blind distribution, reuse is concentrated inside each conversation, and there LiteLLM’s session affinity costs one line of configuration and resolves the case.

Traffic where the network dominates. The GORGO result is the warning: with enormous prompts and a distributed deployment, the improvement is eaten up by queuing and network latency.

The case where it does pay off is specific: several replicas of the same model, a large system prompt shared between different users, or a RAG with a corpus of documents that repeat across requests. There the prefill saved is real and it is most of the work.

Checklist

Measure the current hit rate per replica with the two vLLM counters before touching anything, and check whether the engine is evicting with vllm:num_preemptions_total.
Check that prefix caching is switched on. It is no longer --enable-prefix-caching that switches it on: the default value is derived from the model and the log of the decision is at DEBUG level. What is actionable today is --no-enable-prefix-caching to switch it off.
Audit the prompts before routing: a timestamp or a session identifier at the start of the system prompt breaks the match from block zero and no routing layer fixes it.
If the KV event stream is enabled, size the bandwidth by tokens per second and watch discarding at the high water mark. With data parallelism, subscribe to one port per rank.
If an exact router is chosen, verify the vLLM version against the one the replay buffer requires.
If llm-d is chosen, explicitly configure the vllm tokenisation backend. The default estimate does not fail, it degrades.
If production-stack is chosen with kvaware or loadaware, check first whether the traffic goes through /v1/chat/completions, because today it does not work there.
Review the load-based affinity-breaking parameter, because in three of the four main projects it ships disabled.
Lower num_retries in LiteLLM when there is a router below that already does failover.
Trial with mirrored traffic before changing production. It is the explicit recommendation of the only independent measurement that went looking for bad regimes, and found them.

Traps and things that are not what they seem

LiteLLM has no prefix routing in any version. Neither the open one nor the paid one. I got it wrong in June.
prompt_caching does nothing if the client does not send cache_control, and with the default automatic injection it fails on every turn because it marks the last message.
That pin’s TTL is 300 hardcoded seconds, incompatible with the one-hour caches that LiteLLM itself knows how to request.
production-stack’s prefixaware indexes characters, not tokens, in non-configurable chunks of 128, and its trie is never purged and models no eviction.
production-stack’s kvaware and loadaware do not support chat completions. They tokenise an empty string.
The default tokeniser in llm-d’s EPP is a byte estimate, and with it the exact scorer degrades silently.
llm-d’s active-active mode is discouraged with approximate prefixes, and the reference guides fix one replica even with the exact one.
Dynamo’s overlap credit decay ships at zero, which can leave a freshly autoscaled replica with no traffic.
The SGLang router’s default values differ between the Rust and the Python CLI, by as much as four orders of magnitude.
vllm:gpu_prefix_cache_hit_rate no longer exists and vllm:gpu_cache_usage_perc is now called vllm:kv_cache_usage_perc. Old panels and autoscalers are measuring nothing.
The hit rate counts the reprefill after eviction as a hit. A high number can be reuse or it can be thrashing.
KV events do not publish the hits by default, only the additions, unless full mode is requested request by request.
The hashes travel truncated to 64 bits unless it is disabled with an environment variable.
The default block size is 16, and the attention backend can change it without anyone asking.
The Endpoint Picker left the Gateway API Inference Extension repository. Looking for it there today leads nowhere.

Closing

June’s error has a reading that goes beyond the erratum. It came from reading a product’s documentation and not its code, and from the fact that LiteLLM’s documentation uses the word “caching” in a pre-call check that does something else. The distance between what a name suggests and what a function does is, at this point in the track, the most repeated pattern of the whole series.

The architectural conclusion is that KV-aware routing is not a gateway function. It is not one today in LiteLLM and probably should not be: it requires keeping an index synchronised with each engine’s memory, replicating its hash function, consuming an event stream and sizing CPU per request per second. That belongs to the engine’s domain, not to that of identity and budget.

What remains is a two-part decision. The first is whether the traffic deserves it, and that is answered with two Prometheus counters and without installing anything. The second is which of the three routers fits the cluster you already have, and there the deciding variable is not the hit rate but whether you are willing to bring in Gateway API and the inference extension, or to pin vLLM 0.13.0, or to operate a Dynamo frontend.

And the honesty the material demands: the only independent measurement that specifically went looking for regimes where this makes things worse found them, and they are not marginal. Half to two thirds of reference capacity is a serious regression. That no project publishes that side of the curve does not mean it does not exist in theirs.

Sources

LiteLLM, Router - Load Balancing (estrategias, comprobaciones previas, afinidad): https://docs.litellm.ai/docs/routing.
LiteLLM, código: litellm/router.py, litellm/router_utils/prompt_caching_cache.py, litellm/router_utils/pre_call_checks/prompt_caching_deployment_check.py, litellm/router_utils/pre_call_checks/deployment_affinity_check.py, litellm/proxy/litellm_pre_call_utils.py, litellm/proxy/middleware/admission_control_middleware.py: https://github.com/BerriAI/litellm.
LiteLLM, incidencia 28427, TTL de afinidad por caché de prompts fijado a cinco minutos: https://github.com/BerriAI/litellm/issues/28427.
vLLM, código: vllm/v1/core/kv_cache_utils.py, vllm/v1/core/block_pool.py, vllm/v1/core/kv_cache_manager.py, vllm/distributed/kv_events.py, vllm/config/kv_events.py, vllm/v1/metrics/loggers.py, vllm/v1/metrics/stats.py: https://github.com/vllm-project/vllm.
vLLM, KV offloading usage (niveles CPU, filesystem y objeto; compartición entre procesos): https://docs.vllm.ai/en/latest/features/kv_offloading_usage.html.
vLLM, notas de la versión 0.29.0 (semilla determinista, seguimiento de propiedad en eventos, flags de admisión): https://github.com/vllm-project/vllm/releases/tag/v0.29.0.
vLLM, RFC 16669, publicación de bloques KV y métricas: https://github.com/vllm-project/vllm/issues/16669.
vLLM production-stack, código: src/vllm_router/routers/routing_logic.py, src/vllm_router/prefix/hashtrie.py, src/vllm_router/parsers/parser.py: https://github.com/vllm-project/production-stack.
LMCache Lab, resultados de production-stack frente a AIBrix (parte interesada): https://blog.lmcache.ai/en/2025/03/06/open-source-llm-inference-cluster-performing-10x-faster-than-sota-oss-solution/.
llm-d Router, código y documentación de operaciones (pesos por defecto, dimensionamiento del EPP, aviso de activo-activo): https://github.com/llm-d/llm-d-router.
llm-d, guía de enrutado preciso por prefijo y resultados publicados (parte interesada): https://github.com/llm-d/llm-d.
llm-d Router, incidencia 1290, estado de prefijo no compartido entre réplicas del EPP: https://github.com/llm-d/llm-d-router/issues/1290.
Red Hat, Same 16 GPUs, twice the users (parte interesada): https://www.redhat.com/en/blog/same-16-gpus-twice-users-inference-aware-routing-llm-clusters.
Kubernetes SIG Network, Gateway API Inference Extension (traslado del EPP y del BBR): https://github.com/kubernetes-sigs/gateway-api-inference-extension.
NVIDIA Dynamo, diseño y ajuste del router KV: https://github.com/ai-dynamo/dynamo.
SGLang, sgl-model-gateway: https://github.com/sgl-project/sglang/tree/main/sgl-model-gateway.
AIBrix, enrutado por caché de prefijo: https://github.com/vllm-project/aibrix/blob/main/pkg/plugins/gateway/algorithms/prefix_cache_readme.md.
Ray Serve LLM, enrutado consciente de prefijo (API en alfa): https://docs.ray.io/en/latest/serve/llm/user-guides/prefix-aware-routing.html.
Huang Cheng (Meta), CacheRoute, agosto de 2026: https://arxiv.org/html/2608.19677.
Yuan et al., DualMap, febrero de 2026: https://arxiv.org/abs/2602.06502.
Ricci Toniolo et al., GORGO, junio de 2026: https://arxiv.org/html/2602.11688.
Yoshimura, Chiba y van de Beek, Accuracy Is Speed (EuroMLSys ‘26), abril de 2026: https://arxiv.org/html/2604.15732v1.

Humans and agents on the same gateway: why `priority` does not prioritise, the 429 that brings down the whole pool, and the two ways to separate traffic

Thu, 10 Sep 2026 08:00:00 +0200

Fourth article in the operational track of the control layer. The pair with Langfuse covered observability, the proxy’s day 2 covered availability and virtual keys covered governance. Here comes the new tenant: agents, which speak the same protocol as humans and behave in a different way. Everything that follows is verified against LiteLLM 1.100.1 and vLLM 0.29.0, both from the first ten days of September 2026.

TL;DR

The day a team connects an agent to the corporate gateway, the platform changes regime without anyone touching a YAML. These are the seven conclusions.

LiteLLM’s priority parameter prioritises nothing under normal conditions. It is still marked as beta and its queue is only consulted when all the deployments in the group are in cooldown. With free capacity, the only thing it produces is a response header. On top of that, the success branch does not pop the item off the queue, so the list grows indefinitely; and on /chat/completions the value 0, which is the one used by all the documentation examples, is falsy in Python and disables queuing.

Separating traffic is done with two pools, not with priorities. The non-forgeable way is model_group_alias at key or team level: the client always asks for qwen-30b and the credential decides whether that resolves to qwen-30b-interactivo or to qwen-30b-agentes. The way that does not touch clients is tag_regex against the User-Agent, with a pattern like ^User-Agent: claude-code\/, which serves to classify load and not to control access, because that header is set by the client.

The useful admission control arrives in the 1.101 branch, with max_in_flight_requests_per_worker and a per-process queue that returns 503 with retry-after. It rejects before authenticating, so what it discards does not appear attributed to any key.

max_parallel_requests is only applied at virtual key level. The team, user, end customer and organisation endpoints accept the field and store it in the database, and nobody reads it. global_max_parallel_requests is a no-op unless the old limiter is reactivated, and general_settings.max_parallel_requests is read by no code path.

vLLM with --scheduling-policy priority does not push ahead anyone who is already generating. The only thing that changes is who it evicts when KV cache runs short. If the agents have filled max_num_seqs, a chat with the highest priority waits just the same. And the two knobs that in 2025 allowed short prompts to get ahead of long ones, max_num_partial_prefills and max_long_partial_prefills, disappeared from the code in 0.27.

The retry product reaches 45 calls per agent turn. Three from the client SDK times three from the router times up to five fallback groups. The proxy sets max_retries=0 on the SDK it uses itself, but it cannot touch the client’s. The timeout is per attempt, not per request, and LiteLLM’s Timeout comes out as a 408, which is retryable.

A single 429 takes a deployment out of the pool for five seconds, without going through the allowed_fails threshold. With an agentic burst against a fleet of four replicas, the whole pool enters cooldown and the gateway returns a 429 of its own to everybody, humans included.

You are here: the gateway layer with two classes of traffic

In the seven-layer stack this still lives in the gateway layer, with one difference compared to the previous three articles. There the variable was load: more requests, more teams, more budget to share out. Here the variable is the shape of the load.

A gateway sized for three hundred engineers using a chat works. The same gateway, with thirty of those engineers running coding agents, serves a fraction of the human traffic with the previous latency and the rest degrades. The GPUs stay just as busy, the utilisation metrics stay good, and the support team gets complaints from people waiting nine seconds for a paragraph.

The analogy: the registry desk and the courier with two hundred files

A registry office serves people in order of arrival. Citizens come in one at a time, present a piece of paper, and leave. The system is fair and the queue moves.

One Tuesday a courier turns up with a trolley of two hundred files. He joins the queue like anyone else, because rules are rules, and when his turn comes he occupies the desk for forty minutes. Behind him are twelve people with one piece of paper each. Nobody has broken any rule. The courier has the same right as everyone else and uses the same procedure. What fails is that the office was designed assuming a certain size of transaction and another one three orders of magnitude bigger has turned up.

The real solutions for an office are three, and all three have an exact equivalent in the platform. Open a desk for bulk deliveries, which is separating pools. Put up a capacity sign and send people back later, which is admission control with its 503. And say that files are handed in during the afternoon, which is the part that on an inference platform can almost never be applied, because the agent does not negotiate schedules.

What does not work in the office is giving the citizen a priority number when the courier is already at the desk. That number does not interrupt the transaction in progress. It is, literally, what priority does in the two layers of the platform, and it is worth keeping in mind while reading the next section.

The two kinds of traffic, measured

Before configuring anything, you have to be able to state the difference in numbers, because every later adjustment comes out of it.

Trait	Human chat	Agentic loop
Input per request	hundreds of tokens	tens of thousands, and it grows with each turn
Output per request	hundreds of tokens	from two tokens (a tool call) to several thousand
Concurrency per user	1	3 to 20 requests in flight
The metric that hurts	time to first token	total turn throughput
Tolerance for waiting	seconds	minutes
Cancelled requests	rare	routine, by the agent’s own decision
Retries	the browser’s	the SDK’s, plus the loop’s
Witness when it fails	the user	nobody

Two rows deserve comment because they contradict intuition.

The first is the metric row. An agent does not perceive time to first token, so on a shared engine it is the natural candidate to give up that metric. The operational conclusion is that the agent pool can be configured for aggressive throughput, with large batches and deep queues, while the interactive pool is configured for the opposite.

The second is the cancellations row. An agent that launches three speculative continuations and discards two generates two aborted streams per turn. At the gateway, that touches spend accounting, the release of concurrency counters and the open connection to the engine. LiteLLM 1.100 handles it reasonably well, and it did not six months ago.

What does not work: `priority` in LiteLLM

The first reaction of any operator is to look for a priority field. LiteLLM has one, documents it on its Request Prioritization page, and that page carries the beta label and the sentence saying it is for testing. The label is correct and has to be taken literally, because behind it there are four things.

The queue is only consulted when the whole group is in cooldown. The scheduler’s poll() method returns true if there is any healthy deployment, without touching the queue. With available capacity, which is the normal situation, the request goes straight through and priority orders nothing. The only thing that changes is that the response carries the x-litellm-request-prioritization-used header.

The queue does not empty. In that success branch there is no heappop and no call to the cleanup, which only happens in the expiry branch. The list under the key scheduler:queue:{model} grows without bound, is serialised whole into Redis on every insertion, and there is no maximum size and no discard policy. When the group does enter cooldown, the head of the heap is occupied by fossil entries from requests that finished hours ago, and new ones compare their identifier against that head, never match, and wait until the timeout runs out.

On /chat/completions, the highest priority disables the feature. The code reads the value with a kwargs.get("priority") or self.default_priority, and in Python 0 or None is None. Since in this scheduler the lowest number is the highest priority, priority: 0, which is the value in the documentation’s own examples, goes down the path without queuing. On /completions the check is done with is not None and it works; the two routes do not behave the same.

And default_priority in router_settings breaks the request. If the client does not send a priority but the global setting is in place, the value is truthy, the scheduling branch is entered and a function is invoked that requires the positional argument nobody has injected. The setting has no associated test and does not appear in the router_settings reference.

There is also a /queue/chat/completions, marked as experimental and hidden from the OpenAPI schema, which reads data["priority"] with no default value: a request without that field raises an exception that comes out as a 400 authentication error, with a message that has no relation to the cause.

There is a priority feature that is solid, and it is a different thing. The v3 dynamic limiter shares out token and request per minute quota between classes with weights, through litellm.priority_reservation with values like {"premium": 0.75, "standard": 0.25}. It shares out capacity, it does not order a queue: below the saturation threshold it lends free capacity and above it applies the weights. It never makes anyone wait, either it passes or it returns a 429. It requires Postgres and an enterprise licence, and it is still marked as beta.

What does work: two pools

Effective separation is done earlier, by deciding which set of replicas each class of traffic goes to. LiteLLM offers two mechanisms with very different security properties.

By credential: `model_group_alias` at key or team level

Router settings are resolved in the order key, team, global. Among those that accept that override is model_group_alias, and there lies the piece:

model_list:
 - model_name: qwen-30b-interactivo
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-chat.inferencia.svc:8000/v1
 model_info:
 id: chat-01
 max_input_tokens: 32768
 - model_name: qwen-30b-agentes
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-agentes.inferencia.svc:8000/v1
 model_info:
 id: agentes-01
 max_input_tokens: 131072

And on the agent team’s key:

curl -X POST "$PROXY/key/generate" \
 -H "Authorization: Bearer $MASTER_KEY" \
 -d '{
 "team_id": "plataforma-agentes",
 "models": ["qwen-30b"],
 "router_settings": {
 "model_group_alias": {"qwen-30b": "qwen-30b-agentes"}
 }
 }'

The client asks for qwen-30b and knows nothing about any of it. The resolution is tied to the credential, so the caller cannot get around it by changing a header. The same list allows routing_strategy, fallbacks, context_window_fallbacks, retry_policy, cooldown_time and allowed_fails to be set per key or team, which gives different reliability policies for each class of traffic on the same proxy instance.

It should not be confused with access groups (model_info.access_groups), which are permissions and not routing: they decide whether a key can invoke a model name, not which replica the request goes to.

By `User-Agent`: `tag_regex`

When separate credentials cannot be issued, because the agentic client uses the same personal key as that person’s chat, LiteLLM classifies by regular expression against the request headers:

model_list:
 - model_name: qwen-30b
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-agentes.inferencia.svc:8000/v1
 tag_regex: ["^User-Agent: claude-code\\/", "^User-Agent: .*codex"]
 model_info: {id: agentes-01}
 - model_name: qwen-30b
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-chat.inferencia.svc:8000/v1
 tags: ["default"]
 model_info: {id: chat-01}

router_settings:
 enable_tag_filtering: true
 tag_filtering_match_any: true

The proxy composes strings of the form User-Agent: <value> and tests them with re.search. Exact tag matching takes precedence over the regex. The documentation itself carries a warning that has to be respected: the User-Agent is written by the client, so this is traffic classification and not a security boundary. As load sharing it is enough, because an agent that lies about its identity to sneak into the interactive pool is a governance problem, not a routing one.

The behaviour when no deployment carries the requested tag has a nuance that comes as a surprise in production. If the tag is unknown to the group, the request falls through to the pool marked with tags: ["default"]. If the tag exists in the group but no healthy deployment carries it at that moment, the request fails with the tag configuration error, even though a default pool is available. To get deterministic fallthrough to the default pool you have to set model_info.allow_fail_open: true.

A different strategy per pool with `routing_groups`

router_settings:
 routing_strategy: simple-shuffle
 routing_groups:
 - group_name: interactivo
 models: [qwen-30b-interactivo]
 routing_strategy: simple-shuffle
 - group_name: agentes
 models: [qwen-30b-agentes]
 routing_strategy: simple-shuffle

The recommendation to leave simple-shuffle on both is not laziness, and sustaining it requires looking at what the alternatives do on a homogeneous vLLM fleet.

least-busy keeps a complete dictionary under a single key and updates it with non-atomic reads and writes, without using INCR, so with several proxy replicas increments are lost. It has no TTL, so a request that never completes leaves the counter raised forever and that replica is permanently excluded. Counters can go negative. And when the minimum corresponds to an identifier that is no longer in the healthy list, it falls back to random.choice without warning. The pattern of requests that do not complete is exactly that of an agent that cancels.

latency-based-routing measures seconds per output token, not wall-clock latency. A chat turn of twenty tokens looks extremely slow by that metric and an agent generation of two thousand tokens looks extremely fast, so mixing both kinds of traffic in the same group poisons the signal. It also seeds new deployments with zero latency, and with the buffer defaulting to zero, a freshly started replica takes all the traffic until its first success callback.

usage-based-routing-v2 uses minute windows that reset all at once, and it accounts for tokens when the request completes, so long generations in flight are invisible to the router. With long outputs, it systematically underestimates the real load.

cost-based-routing sorts by the sum of the input and output unit prices and takes the first, with no random tie-break. With replicas of the same model at the same price, all the traffic always goes to the same place. And a model that is not in the price map gets a default cost of 5.0 per token, a figure designed to deprioritise it.

Admission control: the new piece in the 1.101 branch

When the agent pool saturates, what you want is to reject fast and with a code the client understands, not to accumulate requests inside the proxy until memory runs out. That arrives with per-worker admission control, added in the 1.101 branch, in release candidate as this is written.

general_settings:
 max_in_flight_requests_per_worker: 64
 max_queued_requests_per_worker: 64
 admission_queue_timeout_seconds: 1.0

It is a semaphore with a per-uvicorn-process queue, without Redis. When the queue fills or the wait expires it returns 503 with retry-after: 1 and a body with "type": "overloaded_error". It is observed through /health/backlog, which exposes requests in flight, admitted, queued and rejected, and through the Prometheus litellm_admission_* metrics, whose rejection counter distinguishes queue_full from queue_timeout.

Two properties have to go into the runbook. The middleware rejects before authenticating, so what is discarded is not attributed to any key in the spend records, and the real number of requests rejected per team cannot be reconstructed from the spend table. And the limits are read on the first request: changing them requires a restart.

Concurrency per key, and the three settings that do nothing

max_parallel_requests is the right brake for a specific agent, and its configuration surface is full of dead ends.

The only level where it is applied is the virtual key. The v3 limiter, active by default since 1.94, has descriptors for user, team, team member, end customer, organisation, model per key, tag and agent, and all of them carry only request and token counters per window. The only reference to max_parallel_requests is the one in the key descriptor. The old limiter says so in a comment: supporting it for model, user and team is pending.

At the same time, /team/new, /user/new, /customer/new and the organisation endpoints accept the field, validate it and write it to Postgres, and the documentation shows it in their examples. Nobody reads it afterwards. A team with max_parallel_requests: 20 in its row has no concurrency limit at all.

global_max_parallel_requests is injected into the request metadata and is only consulted by the old limiter, which is disabled unless LEGACY_MULTI_INSTANCE_RATE_LIMITING=true is set. The server settings documentation still describes it as a global limit coordinated through Redis. And general_settings.max_parallel_requests does not appear in any execution path: its only two appearances are the copy from the database and the type schema of the interface.

What does work, in detail:

curl -X POST "$PROXY/key/generate" -H "Authorization: Bearer $MASTER_KEY" \
 -d '{"team_id":"plataforma-agentes","max_parallel_requests":8,"rpm_limit":600}'

Counting is done with a sorted set in Redis scored by Redis’s own clock, with Lua scripts to acquire and release. Each slot expires after 3,600 seconds, which is at once the maximum request duration the meter can track and the time a leaked slot takes to heal by itself. Release is hooked to success, to failure and to client disconnection during streaming, which was the classic leak with clients that cancel.

One live leak remains and it affects exactly the case in this article: every tool call through the MCP gateway acquires a slot and does not release it. The MCP route invokes the pre-call hook, which acquires, and none of the release functions is called from that module. With an agent that makes twenty tool calls per turn and a key with eight slots, the 429 arrives on the first turn and lasts an hour.

The order of evaluation matters for interpreting 429s. First the window counters, requests and tokens per minute, are checked, and only if they pass is the concurrency slot acquired. The code comment explains why: the other way round, every rate limit rejection would leave an orphan slot. In practice, with a low rpm you always see the requests type 429, and the max_parallel_requests one only appears when the relationship between concurrency and duration makes it binding first. The message distinguishes the two cases with the Limit type: field and the rate_limit_type header.

A minor detail with consequences on the client: the response to a concurrency rejection carries retry-after and reset_at computed from the window size, sixty seconds by default, when a concurrency meter has no window. A well-behaved client that respects that header will wait a minute when it might have had a free slot in two seconds.

Session affinity: the KV cache that gets thrown away

An agent turn resends the whole conversation context, system prompt and tools included. If that request lands on a different replica from the previous turn’s, vLLM’s prefix cache does not have the prefix and the prefill is recomputed in full. With twenty turns and four replicas balanced blindly, the long prefill is paid for fifteen times over.

LiteLLM has two mechanisms for this and they are not equally good.

The prompt_caching check computes a SHA-256 of the cacheable prefix, defined as everything up to the last block with cache_control, and pins the deployment with that hash. It has two problems. The TTL is hardcoded to 300 seconds in two places in the code, with no way to configure it, something which is reported and open; with one-hour caches, affinity is lost after five minutes of inactivity. And the key is an exact hash, so a client that moves the cut point forward on every turn, which is exactly what agents do to extend the cached prefix, generates a new key every time.

The session_affinity check is the one to use:

router_settings:
 optional_pre_call_checks: ["session_affinity"]
 deployment_affinity_ttl_seconds: 3600

It pins the deployment with an atomic Lua script in Redis, with a configurable TTL that defaults to one hour and that is refreshed on every request, so it bounds the idle time between turns and not the duration of the conversation. If the pinned deployment enters cooldown, the request goes out through the normal strategy and keeps the pin so it can come back when it heals.

The session identifier comes from a precedence chain that starts with x-litellm-trace-id, continues with x-litellm-session-id and then accepts any header of the form x-<something>-session-id whose value looks like an identifier. That includes the session header Claude Code already sends, with nothing configured on the client. Whatever matches populates both the session identifier and the trace identifier, which links this to the correlation from the first article in the track.

One dangerous interaction when combining it with tags: affinity is evaluated before tag-based routing. If the pin reduces the list to a deployment that does not satisfy the request’s tag, the request fails instead of falling through to the default pool, unless allow_fail_open has been enabled. Retagging a pool with live sessions breaks them.

And a clarification that saves a pointless search: LiteLLM does not do prefix routing. There is no prefix tree and nothing that directs the request to the vLLM replica that already has that prefix warm. What there is, is affinity by session or by key. For the other thing you need a specific router underneath the gateway, and that architecture has its own article pending.

The engine: what vLLM can and cannot do

Everything above shares out requests between engines. Inside one engine, the question is what happens when a twenty-token request and a sixty-thousand-token one live side by side.

Priority exists and does less than it looks like. --scheduling-policy priority turns the waiting queue into a heap ordered by priority, arrival time and identifier, with the lowest number first. The request can carry the priority field in the body, and there is also the X-Vllm-Priority header, which takes precedence over the body and is not documented anywhere.

What priority does not do is push ahead of whoever is already generating. In the running queue loop, when KV cache runs short, the policy only changes who is chosen as the victim of eviction: with priority the one with the worst priority and arrival time is evicted, and with fcfs the last in the queue. In the waiting queue loop, if there are no blocks available the iteration is cut short, without evicting anyone. And the max_num_seqs cap is a hard cut. If the agents have filled that cap, a chat with maximum priority waits just the same. There has been an open issue since April and an unmerged pull request about this.

That nuance about the victim is not negligible, because with fcfs the victims are the most recently admitted, that is, the chats that have just come in. Switching the priority policy on already improves the situation even if it pushes nobody ahead.

Eviction is always by recomputation. The swap-to-host-memory mode and the enumeration that controlled it disappeared from the code; all that is left is freeing blocks and returning the request to the queue with the computed token counter at zero. An evicted request loses all its prefill, and the prefix cache recovers part of it, but the perceived time to first token restarts.

Two admission settings help and do not appear in the narrative documentation. watermark reserves a fraction of free KV blocks on admission, with the declared purpose of avoiding repeated eviction, and it ships disabled. And scheduler_reserve_full_isl, active by default, checks that the full input length fits in the cache before admitting, instead of looking only at the first chunk.

Chunked prefill is active by default and prioritises decode. The scheduler walks the whole running request loop before touching the waiting queue, so a long agent prefill does not freeze a chat’s generation in progress. That protects the inter-token latency of what has already been admitted, and it does not protect the time to first token of what is waiting.

Here there is a breaking change that forces a review of the manifests. max_num_partial_prefills and max_long_partial_prefills were removed in 0.27. They were precisely the mechanism that allowed short prompts to get ahead of long ones. A deployment with those flags in its args does not start. long_prefill_token_threshold still exists, and its meaning changed: today it is a per-request chunk size cap, not a classification of long requests.

The partial replacement arrives in 0.29 with max_num_queued_tokens and max_num_queued_reqs, admission valves that return 503 when the pending prefill work exceeds the target. The code itself frames it as a quality of service mechanism for time to first token, with the formula of setting it to the latency target multiplied by prefill throughput. They are blind to the class of traffic: they reject a chat just the same as an agent.

And the default value of --max-num-batched-tokens depends on the hardware and on the start-up mode. With vllm serve on H100 or H200 it is 8,192, on cards of 160 GiB or more it is 16,384, and on the rest 2,048. The 2,048 that appears in the scheduler configuration only applies to tests.

The metrics for diagnosing coexistence, with the exact names from 0.29:

Metric	What for
`vllm:num_requests_waiting`	queue depth, the autoscaling signal
`vllm:num_requests_waiting_by_reason`	separates waiting for capacity from waiting in a blocked state
`vllm:num_preemptions_total`	evictions, the signal that the two classes are treading on each other
`vllm:kv_cache_usage_perc`	cache pressure, the cause of the evictions
`vllm:request_queue_time_seconds`	what a chat waits with the agents inside
`vllm:prefix_cache_hits_total` and `_queries_total`	measured in tokens, not in requests

A warning for anyone with old alerts. The log warning about evicted sequences that appears in the optimisation documentation belongs to the previous version of the engine and no longer exists in the code. Preemptions are only visible as a counter. An alert based on searching for that string in the logs has not fired for months, and not because there are no evictions.

On isolation between classes inside one engine, the short answer is that it does not exist. There are two queues and neither is per class of traffic. The conclusion, which is the one that holds up this article’s architecture: effective isolation means separate fleets, and on top of them the gateway deciding who goes to which.

Retries: the product nobody calculates

A human who gets an error sees it and decides. An agentic loop retries, and beneath it there are two more layers that also retry.

Layer	Default
Client SDK	2 retries, 3 attempts
LiteLLM router	2 retries, 3 attempts
Fallbacks	up to 5 groups

The product is about 45 calls to the engine for one agent turn, before counting the loop’s own retry. LiteLLM does what it can: when the call comes from the router, it forces max_retries=0 on the SDK it itself uses against the engine. What it cannot touch is the client’s SDK. The only layer that breaks the product is that one, and it is broken by setting max_retries=0 on the client and delegating the retry to the gateway.

The retryable errors are 408, 409, 429 and anything 500 or above. And there lies the problem with long reasoning: litellm.Timeout is a subclass of OpenAI’s timeout exception and comes out with code 408, which is retryable, so the most expensive request of all, the one that expired after ten minutes of generating, is launched again in full. Twice by the router, and as many times again by the client.

On timeouts you have to be precise because the code comment is: the configured timeout is per attempt, not per request. A request_timeout: 600 with two retries and two fallback groups gives a worst case of almost two hours before the client receives anything. The default value is still 6,000 seconds, as was already covered in the day 2 article.

The retry storm has two specific causes and both are default settings.

The first: a single 429 takes the deployment out of the pool, without going through the allowed_fails threshold. The cooldown logic has an explicit branch which, on a 429 in a group with more than one deployment, returns immediate cooldown. With an agentic burst against four replicas, all four leave the pool at once and the gateway answers everybody with its own 429 of “no deployments available”. The default cooldown is five seconds, which in a sustained burst gets renewed.

The second: backoff is zero while the pool looks healthy. The function that computes the wait returns 0 if the healthy deployment list is not empty. Exponential backoff with jitter, capped at eight seconds, only kicks in when there is nothing healthy left. Under partial degradation, the proxy hammers away without waiting and only slows down once everything has already fallen over.

On top of that, the engine’s Retry-After header is only respected if its value is between 1 and 60 seconds. A longer limit window is ignored and replaced by the proxy’s own backoff.

The settings that cut the problem off:

router_settings:
 num_retries: 1
 retry_after: 2 # real floor; the default 0 is hammering
 cooldown_time: 30
 allowed_fails_policy:
 RateLimitErrorAllowedFails: 3 # stops a 429 taking the deployment out
 retry_policy:
 TimeoutErrorRetries: 0 # do not relaunch the expensive request
 AuthenticationErrorRetries: 0
litellm_settings:
 request_timeout: 900 # per attempt
 drop_params: false

That last setting deserves an explanation, because it is a silent way of breaking an agent. With drop_params: true, LiteLLM removes from the body any parameter the destination provider does not support, and in that branch it writes no warning at all. The candidates for disappearing are tools, tool_choice, response_format and parallel_tool_calls. An agent that has tools taken away receives free text where it expected a function call, and interprets it as the model having decided not to use tools.

Worse still is prompt-based emulation. For a non-OpenAI-compatible provider that does not support tools, LiteLLM does not fail: it turns the tools into text inside the prompt, forces JSON format, discards tool_choice, and on the way switches on a process-global variable that makes any other tool-less provider in the same process go to emulation instead of erroring.

With that in mind, the warning about fallbacks is clear: LiteLLM does not check capabilities before falling back. The function that knows whether a model supports function calling exists and is only used in an informational endpoint. The pre-call checks filter by context window, request limit and region, and nothing else. A fallback list that goes from a model with tools to one without them degrades the agent silently. There are guards for provider-scoped resources, files and batches, so the absence of the capability guard is an omission and not a design oversight.

A month’s budget in twenty minutes

A runaway agent consumes in minutes what a team consumes in weeks, and the operational question is how long the gateway takes to find out.

The answer has changed, and the old mental model, the one where spend consolidates every minute and until then there is no brake, is no longer correct. Since 1.84 there are two new pieces. An optimistic reservation before the call, which estimates the maximum cost of the request and reserves it atomically in the counter before heading out to the engine, reconciling it with the real cost on completion. And a counter increment after the call that is explicitly awaited, with the comment that this way the counter is up to date before the next request goes through authentication. Budget checks read that counter, with Redis first and seeding from the database when cold.

Batch writing to Postgres still exists, every 10 to 15 seconds, and it affects persistence and what is shown in the interface, not the cut-off. The production documentation’s recommendation to raise that interval to 60 does not relax the control, and it does delay by a minute what /key/info shows, something which frequently generates operational panic. And the budget rescheduler’s ten minutes do not consolidate spend: they fire the job that resets expired budgets.

The overspend window then comes down to the cost of the requests in flight, provided there is Redis. Without Redis, the counters are per pod, and with N replicas you can spend up to N times the budget. That is the configuration failure that matters and it is the same Redis that the day 2 article declared non-optional for other reasons.

Two settings that do not appear in the public documentation, only in the data model descriptions, and that decide behaviour at the worst moment:

general_settings:
 fail_closed_budget_enforcement: true # false by default: if the counter fails, it lets things through
 # disable_budget_reservation: do not set it to true

With the default value, if the counter backend does not respond, the proxy fails open and carries on serving. For an agent in a loop that is exactly the opposite of what you want.

Time for the honest warning: this route is still unstable. There are open issues in both directions, one where the key and user budget limit stops being applied while spend is recorded, and another where a 429 is returned for stale spend in the reservation counters while the key query shows spend below the limit. Project budgets are not on the atomic reservation route and concurrency overshoots them. The practical conclusion is not to trust the hard cut-off to the budget alone.

On the manual cut-off, there is an asymmetry to know about before you need it. /key/block invalidates the local cache, the Redis one, and notifies all workers by publish, so with Redis the effect is almost immediate. /customer/block does the same. /team/block only writes to the database, without invalidating or notifying, so it takes however long the management object cache TTL takes, sixty seconds by default. To cut a team off dead you have to block its keys one by one, or lower user_api_key_cache_ttl. And /user/block no longer exists, even though the docstring of the function that replaced it still shows that route in its example.

On alerts, two numbers. The default thresholds warn at 85 % and 95 % of the budget. And deduplication of those alerts is 24 hours per event and identifier, so crossing 85 % generates one warning and then silence for a day. For agentic traffic, where 85 % and 100 % can be twenty minutes apart, the second warning arrives when it is no longer any use.

Attribution: who the user is when the caller is an agent

With agents, the question of who the spend is charged to has two answers and both are needed: the agent that executes and the human on whose behalf it acts.

The chain that extracts the end customer identifier tries the standard headers first, then the configurable header mapping, and only then the user field of the body, the litellm_metadata.user of the Anthropic format, the metadata.user_id and the safety_identifier. The last four are body fields and are therefore written by the client. The code itself warns about it: for untrusted callers the identifier has to be set from headers or from a middleware on the server, to avoid impersonation.

Out of that comes the correct pattern, which is the inverse of the one people put together out of inertia: the virtual key identifies the agent and is the non-forgeable part, and the header or the user field identify the human and serve for attribution, not for control. An agent that lies about the human behind it is still bounded by the budget and the limits of its key.

The spend table stores user, end_user, team_id, organization_id, request_tags, session_id and agent_id, with daily aggregation per agent, so the detail is available once it is sent. The x-litellm-agent-id header is what populates that last field.

A note about the MCP gateway, which on a platform with agents ends up being the other entry door. Tool calls do appear in the spend records, with call types of their own and a metadata block that stores tool name, arguments, result and server. What is missing is token counting: the cost is 0.0 by default and is only populated if the administrator configures a price per query, per server or per tool. It is the same pattern of silent zero cost that already came up with unpriced models in the first article in the track.

And the permission setting to review on day one: access control for MCP servers is open by default. If no level of the hierarchy defines a list, the request reaches every configured server. general_settings.require_key_mcp_access_defined: true inverts that inheritance, and it is not the default value.

A reference `config.yaml`

All of the above, together, on a fleet split into two pools:

model_list:
 - model_name: qwen-30b-interactivo
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-chat.inferencia.svc:8000/v1
 input_cost_per_token: 0.00000018 # without this, silent zero cost
 output_cost_per_token: 0.00000072
 model_info:
 id: chat-01
 max_input_tokens: 32768 # without this there is no context filter

 - model_name: qwen-30b-agentes
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-agentes.inferencia.svc:8000/v1
 input_cost_per_token: 0.00000018
 output_cost_per_token: 0.00000072
 tag_regex: ["^User-Agent: claude-code\\/"]
 model_info:
 id: agentes-01
 max_input_tokens: 131072
 allow_fail_open: true

router_settings:
 routing_strategy: simple-shuffle
 enable_pre_call_checks: true
 enable_tag_filtering: true
 optional_pre_call_checks: ["session_affinity"]
 deployment_affinity_ttl_seconds: 3600
 num_retries: 1
 retry_after: 2
 cooldown_time: 30
 allowed_fails_policy:
 RateLimitErrorAllowedFails: 3
 retry_policy:
 TimeoutErrorRetries: 0
 AuthenticationErrorRetries: 0

general_settings:
 fail_closed_budget_enforcement: true
 require_key_mcp_access_defined: true
 user_api_key_cache_ttl: 10
 proxy_batch_write_at: 10
 # 1.101 branch onwards:
 max_in_flight_requests_per_worker: 64
 max_queued_requests_per_worker: 64
 admission_queue_timeout_seconds: 1.0

litellm_settings:
 request_timeout: 900 # per attempt
 drop_params: false
 cache: true # Redis, not optional

And the start-up of the two engines, with the deliberate asymmetry:

# interactive pool: protect the first token
vllm serve Qwen/Qwen3-30B \
 --scheduling-policy priority \
 --max-num-seqs 64 \
 --max-model-len 32768 \
 --max-num-batched-tokens 4096

# agent pool: throughput and deep queues
vllm serve Qwen/Qwen3-30B \
 --max-num-seqs 256 \
 --max-model-len 131072 \
 --max-num-batched-tokens 8192

Checklist

Issue separate keys for agents, with model_group_alias at key or team level. It is the only separation the client cannot evade.
Add tag_regex over the User-Agent as a safety net for agents that use personal credentials, with a default pool and allow_fail_open: true.
Leave simple-shuffle on both pools. The other four strategies have verifiable defects on homogeneous replicas.
Enable session_affinity with a one-hour TTL. Do not use prompt_caching for this while its TTL stays hardcoded.
Declare max_input_tokens on every deployment. Without that value, the context window filter is not applied and it fails silently for models that are not in the price map.
Declare per-token prices on every self-hosted model, or the agents’ spend will be zero and their budget will never run out.
max_parallel_requests on the agent’s key, never on its team. On the team it is stored and not applied.
Lower num_retries to 1, put a floor on retry_after, raise cooldown_time and disable retry on timeout. On the client, max_retries=0.
fail_closed_budget_enforcement: true and mandatory Redis. Without Redis, N replicas are N times the budget.
Start the interactive engine with --scheduling-policy priority even though it pushes nobody ahead: it changes who it evicts.
Alert on vllm:num_preemptions_total and vllm:num_requests_waiting_by_reason, not on log strings that are no longer emitted.
Prepare the cut-off: block by key, never by team, and verify the real propagation time in your own deployment.

Traps and things that are not what they seem

priority: 0 on /chat/completions disables prioritisation. It is the value in the documentation examples and it is falsy in Python.
max_parallel_requests on a team is stored and not applied. The field exists in the API, in the database and in the documentation; the limiter only reads it from the key descriptor.
global_max_parallel_requests is a no-op with the default limiter, and the server settings documentation still describes it as active.
Every MCP call leaks a concurrency slot, which only heals when its hour expires. With agents calling tools in a loop, the 429 appears early and lasts a long time.
The TTL of prompt cache affinity is 300 hardcoded seconds. For agent conversations, session affinity is the right option.
Affinity is evaluated before tag-based routing. A pin to a deployment that stops satisfying the tag makes the request fail instead of falling through to the default pool.
LiteLLM’s timeout is per attempt. The worst case is the timeout multiplied by attempts and by fallback groups.
A 429 takes the deployment out of the pool without going through allowed_fails, and backoff is zero while anything healthy remains.
The engine’s Retry-After is ignored if it asks for more than 60 seconds.
drop_params: true removes tools and response_format without leaving a warning, and the fallback does not check whether the destination model supports function calling.
/team/block does not invalidate the cache. It takes the full TTL; /key/block does propagate by publish in Redis.
Access to MCP servers is open by default if no level defines a list.
max_num_partial_prefills and max_long_partial_prefills no longer exist in vLLM. A manifest with those flags does not start from 0.27 onwards.
vLLM’s priority does not push ahead of running requests, and with max_num_seqs full it pushes nobody ahead.
The eviction log warning that appears in the vLLM documentation is no longer emitted. Only the Prometheus counter is left.

Closing

The coexistence of humans and agents on the same platform is not solved with a priority field, and that is the conclusion that takes the most effort to accept because the field exists and looks made for this. Neither the gateway nor the engine has a usable notion of class of service today: one orders a queue that is almost never consulted, the other picks who to evict. What they do have is the ability to direct each request to a different place, and that is where the win is.

The architecture that comes out of all of the above fits in one sentence. Two vLLM fleets with opposite parameters, a credential that decides which one each request goes to, session affinity so as not to throw away the prefill work, concurrency limits on the agent’s key and not on its team, retries trimmed back on the three layers where they multiply, and budgets that fail closed.

The rest is measurement. If vllm:num_preemptions_total rises in the interactive pool, the two classes are treading on each other despite the separation. If chat queue time grows while total throughput holds, the partition is badly sized. And if the agents’ spend comes out as zero, the problem is not coexistence but undeclared prices, which is where this track started.

Sources

LiteLLM, [BETA] Request Prioritization: https://docs.litellm.ai/docs/scheduler.
LiteLLM, Router - Load Balancing (estrategias, afinidad de sesión, comprobaciones previas): https://docs.litellm.ai/docs/routing.
LiteLLM, Tag Based Routing (incluye tag_regex y el ejemplo con clientes agénticos): https://docs.litellm.ai/docs/proxy/tag_routing.
LiteLLM, Router Settings for Keys and Teams (resolución clave, equipo, global): https://docs.litellm.ai/docs/proxy/keys_teams_router_settings.
LiteLLM, Server Tuning (control de admisión por worker): https://docs.litellm.ai/docs/proxy/server_tuning.
LiteLLM, Dynamic TPM/RPM Allocation (reserva de prioridad, beta y enterprise): https://docs.litellm.ai/docs/proxy/dynamic_rate_limit.
LiteLLM, Budgets, Rate Limits: https://docs.litellm.ai/docs/proxy/users.
LiteLLM, Production Best Practices: https://docs.litellm.ai/docs/proxy/prod.
LiteLLM, Reliability (fallbacks, ventana de contexto, enfriamientos): https://docs.litellm.ai/docs/proxy/reliability.
LiteLLM, código: litellm/scheduler.py, litellm/router_strategy/tag_based_routing.py, litellm/router_utils/pre_call_checks/deployment_affinity_check.py, litellm/proxy/hooks/parallel_request_limiter_v3.py, litellm/proxy/middleware/admission_control_middleware.py, litellm/router_utils/cooldown_handlers.py: https://github.com/BerriAI/litellm.
LiteLLM, incidencia 28427, TTL de la afinidad por caché de prompts fijado a cinco minutos: https://github.com/BerriAI/litellm/issues/28427.
LiteLLM, incidencia 34534, fuga de slot de concurrencia en cada llamada MCP: https://github.com/BerriAI/litellm/issues/34534.
LiteLLM, incidencia 26672, presupuesto de clave y usuario no aplicado: https://github.com/BerriAI/litellm/issues/26672.
LiteLLM, incidencia 872, comprobación previa de soporte de llamada a funciones, cerrada sin implementar: https://github.com/BerriAI/litellm/issues/872.
vLLM, Optimization and Tuning (prefill troceado, prioridad de decodificación): https://docs.vllm.ai/en/v0.29.0/configuration/optimization.html.
vLLM, Metrics design (nombres, convención _total, autoescalado como problema abierto): https://docs.vllm.ai/en/v0.29.0/design/metrics.html.
vLLM, Data Parallel Deployment (caché KV independiente por réplica): https://docs.vllm.ai/en/v0.29.0/serving/data_parallel_deployment.html.
vLLM, código: vllm/v1/core/sched/scheduler.py, vllm/v1/core/sched/request_queue.py, vllm/config/scheduler.py, vllm/engine/arg_utils.py, vllm/v1/metrics/loggers.py: https://github.com/vllm-project/vllm.
vLLM, incidencia 40004, la prioridad no desaloja peticiones en ejecución: https://github.com/vllm-project/vllm/issues/40004.

Virtual keys, budgets and limits in LiteLLM: the layer that decides who gets the GPU, and the four things the documentation gets wrong

Tue, 08 Sep 2026 07:30:00 +0200

Third article in the operational track of the control layer. The pair with Langfuse covered observability, and day 2 of the proxy covered availability. Here it is governance: who can call, which model, how much and on what budget. The economic model behind those figures is in FinOps and multi-tenancy with LiteLLM.

TL;DR

On a platform running on your own GPUs there is no provider invoice to put consumption in order. Capacity is fixed, known and shared, so someone has to impose the allocation, and that someone is the gateway.

The hierarchy has six levels: organisation, team, team member, internal user, key and end customer, plus the project and tag scopes. Each level’s budgets are evaluated independently and all of them have to pass. They neither add up nor cancel each other out, with one exception.

The exception is in the code and it is the most important governance trap in the system. A key that belongs to a team suppresses the check of its owner’s personal budget, unless apply_user_budget_to_team_keys is enabled. A user with a hundred-euro cap who works with their team’s key spends whatever the team allows.

Rotating a key preserves its spend. /key/regenerate does an UPDATE on the same row changing the hash, so spend, max_budget and the budget window remain intact. It accepts a grace period, stored in a table of its own, and if the period’s format is wrong it is ignored without blocking anything.

The v3 limiter is already the default in 1.100.0, and you go back to the previous one with an environment variable. Its window is sliding, sixty seconds, anchored to the first request, and the counters are incremented with Lua scripts inside Redis using Redis’s own timestamp to close the race between replicas.

And four points where the official documentation does not match the code, verified against tag v1.100.0: an exhausted budget returns 429, not 400; the audited actions are six, not three; /key/info and /key/list are GET, not POST; and the v3 limiter carries a docstring saying it is not production-ready while it is the one loaded by default.

You are here: the gateway as an administrative boundary

Without this layer, an in-house inference platform has a single mode of operation: whoever knows the URL consumes. It works while there is one team. It stops working the day there are three, or the day someone connects an agent that fires twenty calls per interaction and wrecks everyone else’s time to first token.

The two questions this layer answers are different and have to be kept apart. The capacity one is instantaneous: how many requests and how many tokens per minute each consumer can push, so the fleet does not saturate and latency stays within the service agreement. The cost one is cumulative: how much each consumer has spent this month against its budget. They are configured in the same place, applied at different moments and fail in different ways.

The analogy: the building’s key ring

An office building hands out keys at three levels. The master opens everything and the building manager has it. The floor keys open a whole floor and each tenant company has one. And the office keys open one office.

Three properties of that system are the ones that matter here. A key is replaced without changing the lock of the whole building, and the tenant keeps their parking space and their electricity meter: that is rotation preserving the spend. A lost key is cancelled from the central panel without collecting anything physical: that is blocking from the interface. And the record of who took out each key and when lives in a separate book, which is what you show an inspector: that is the audit table.

The analogy breaks at one point, and that point is where this system’s governance flaw lives. In the building, having a floor key does not cancel the office’s consumption limit. In LiteLLM, it does.

The hierarchy, and the line that breaks it

The levels and their management endpoints:

Level	Endpoints	Budget
Organisation	`/organization/new`, `/organization/member_add`, …	`max_budget`
Team	`/team/new`, `/team/member_add`, `/team/block`, …	`max_budget`
Team member	`/team/member_add` with `max_budget_in_team`	Its own row in the budgets table
Internal user	`/user/new`, `/user/update`, …	`max_budget`
Key	`/key/generate`, `/key/update`, …	`max_budget`, `soft_budget`
End customer	`/customer/new`, and the parallel `/end_user/*` set	`max_budget`

The default behaviour is the right one for a multi-tenant platform: each scope carries its own counter and the check of each one is independent of the rest. A request passes if all of them pass.

And then there is this condition, which lives in the authorisation checks:

is_team_key = team_object is not None and team_object.team_id is not None
if is_team_key and general_settings.get("apply_user_budget_to_team_keys") is not True:
 return

Translated: when the key belongs to a team, the check of the budget of the user who owns that key returns without evaluating anything. The personal cap exists in the database, is visible in the interface, and is not applied.

For a deployment where personal budgets are decorative, it makes no difference. For one where the per-person cap is the control that was shown to someone as a guarantee that a user cannot overrun the platform, it is a hole. The line that closes it is one:

general_settings:
 apply_user_budget_to_team_keys: true

Virtual keys: the complete life cycle

The endpoints all exist and some of them are not where you expect:

Route	Method	What for
`/key/generate`	POST	Create
`/key/update`	POST	Modify
`/key/info`	GET	Query one
`/key/list`	GET	List
`/key/delete`	POST	Delete
`/key/regenerate`	POST	Rotate
`/key/block`, `/key/unblock`	POST	Suspend and reactivate
`/key/health`	POST	Check that key’s callbacks

The two marked in bold are GET, and several examples in the documentation suggest otherwise. /key/health is not what its name suggests either: it does not validate the key or its budget, it checks the logging callbacks associated with it.

There are many creation parameters, and these are the ones that govern:

curl -X POST 'https://gateway.interno/key/generate' \
 -H 'Authorization: Bearer sk-...' \
 -d '{
 "key_alias": "equipo-datos-notebooks",
 "team_id": "t-datos",
 "models": ["llama-70b", "qwen-30b"],
 "max_budget": 250,
 "budget_duration": "30d",
 "tpm_limit": 400000,
 "rpm_limit": 600,
 "max_parallel_requests": 8,
 "duration": "90d",
 "tags": ["produccion", "notebooks"]
 }'

Alongside those there are controls that are rarely used and solve specific problems: model_max_budget to set a per-model cap within the same key, model_tpm_limit and model_rpm_limit for the same thing with capacity, enforced_params to require certain keys to always arrive with given fields, allowed_routes so an application key cannot touch the management endpoints, and blocked to be born suspended.

How they are stored

A key’s hash is a single-pass hexadecimal SHA-256, with no salt and no derivation function:

hashed_token = hashlib.sha256(token.encode()).hexdigest()

The row lives in LiteLLM_VerificationToken, with the hash as primary key, and it contains the accumulated spend, the limits, the expiry, the allowed models and the rest of the configuration.

Two consequences to be clear about before an audit. The first is that this hash is not designed to resist a dictionary attack against the database, and the code itself acknowledges it in a comment about encryption key derivation, where it admits that a single-pass unsalted SHA-256 is not a derivation function and that moving to HKDF would be more defensible in an audit. The real mitigation is that virtual keys are generated with enough entropy, not the strength of the hash. The second is that LITELLM_SALT_KEY plays no part here: that variable encrypts the stored provider credentials, not the virtual keys.

There is a trap around that variable that takes down a whole deployment. If it is not defined, it falls back to the master key. Rotating LITELLM_MASTER_KEY without having first set your own LITELLM_SALT_KEY leaves every stored provider credential unreadable, and the failure does not block: it logs an error and returns null, so the symptom shows up as models that stop authenticating with no explanation. Setting LITELLM_SALT_KEY on day one, and never touching it, is one of the things that saves the most grief.

Rotation

/key/regenerate does not create a new key and delete the old one. It updates the same row changing the hash, so the accumulated spend, the maximum budget and the reset window are preserved. For a monthly chargeback this is exactly what you want: rotating mid-month does not zero the counter.

It also accepts a grace period, so the old key remains valid while consumers update:

curl -X POST 'https://gateway.interno/key/sk-vieja/regenerate' \
 -H 'Authorization: Bearer sk-...' \
 -d '{"grace_period": "24h"}'

The old hash is stored in a separate table with its revocation date. Without that parameter, revocation is immediate, and the default value can be set with LITELLM_KEY_ROTATION_GRACE_PERIOD.

A detail with teeth: if the period’s format is wrong, the system logs a warning and continues with no grace period. The rotation completes, the old key dies on the spot, and all that is left is a log line. In an automated rotation procedure, that combination of “carry on” and “warn in the log” is what produces the outage at three in the morning.

What exactly happens when a budget runs out

The code raises BudgetExceededError with status 429. The documentation says 400 in one place, and there the code wins. Its being a 429 has a practical implication: OpenAI clients retry automatically on a 429, so an exhausted budget turns into a retry storm against a gateway that is going to keep rejecting. Internal clients should distinguish the reason, which travels in the message.

The messages are templates and state the scope, the spend and the cap:

Budget has been exceeded! Key=<clave> Current cost: <gasto>, Max budget: <tope>
Budget has been exceeded! Team=<equipo> Current cost: <gasto>, Max budget: <tope>
ExceededBudget: User=<usuario> over budget. Spend=<gasto>, Budget=<tope>
ExceededBudget: End User=<cliente> over budget. Spend=<gasto>, Budget=<tope>
Budget has been exceeded! Organization=<org> ...
Budget has been exceeded! Tag=<etiqueta> ...

Two clarifications about the comparison. It is spend >= cap, with the equals sign included, so landing exactly on the cap already blocks. And the soft budget, soft_budget, blocks nothing: it only writes a log line when it is crossed. It serves to warn, not to contain, and building an alert on top of it is the right way to use it.

The rest of the rejections, with their codes:

Situation	Code	Behaviour
Budget exhausted	429	Message with scope, spend and cap
Expired key	401	It also deletes the cache entry, so the expiry takes effect across the whole fleet without waiting for the TTL
Model not allowed	403	Different error type depending on the scope: key, team, user, organisation or project
Blocked key	Generic exception	`Key is blocked. Update via /key/unblock if you're an admin.`

The last row is an inconsistency in the project worth knowing about: blocking raises an exception with no explicit type or code, while expiry and model access do carry them. Any alert that discriminates by error type will not see blocks the way it sees the rest.

A useful note about allowed models: the check accepts both the alias and the underlying model, so a key authorised for llama-70b can call it by its alias or by its real name with no additional configuration.

Capacity limits: the v3 limiter

This is the most significant silent change of the year in this layer. In 1.100.0, the limiter loaded by default is the third version. Going back to the previous one has to be asked for explicitly:

LEGACY_MULTI_INSTANCE_RATE_LIMITING=true

There is no flag to turn it on, only to turn it off. And it carries a contradiction worth knowing before trusting it: the file itself still opens with a comment saying it is under development and not production-ready, while it is the one running. The design is avowedly inspired by Envoy’s.

The window is sliding and lasts sixty seconds, configurable with LITELLM_RATE_LIMIT_WINDOW_SIZE. It is not an exact rolling log nor a clock-aligned window: it anchors to the first request and resets once the window size has elapsed from that moment. For one-minute limits, the difference from a calendar window shows up at the edges and rarely matters.

What does matter is Redis. The counters are incremented with Lua scripts executed inside Redis, using Redis’s own timestamp and not the pod’s, precisely so window resets are deterministic across replicas and to close the race between checking and counting. Without Redis, each pod counts on its own and the effective limit is multiplied by the number of pods. A limit of 600 requests per minute with five replicas is a limit of 3,000.

There is an inherent imprecision in the token limits that Redis does not solve. The number of output tokens is not known until the response finishes, so the limiter counts with an estimate before the call and reconciles afterwards. The possible excess is, per in-flight request, the difference between the estimate and the real figure. With high concurrency and long responses, that allows the cap to be exceeded within a window. To protect the fleet, the concurrent request limit (max_parallel_requests) is a more direct instrument than tokens per minute.

What the client sees

When limited, a 429 with the message giving the scope, the current limit, what is left and when it resets. And three headers: retry-after with the window size, that is 60 by default, plus rate_limit_type and reset_at.

On the success path, the informational headers use a nested format of their own:

x-ratelimit-api_key-remaining-requests
x-ratelimit-api_key-limit-tokens
x-ratelimit-team-remaining-requests

They are not the standard names x-ratelimit-limit-requests that OpenAI and compatibles emit, so a client expecting the usual format will find nothing. There is also an open issue about these headers being lost on streaming responses, which is most of an assistant’s traffic.

Identity: beyond the static key

Handing out keys by hand works with three teams and stops working with thirty. The two routes for connecting the gateway to corporate identity:

JWT. It is enabled with enable_jwt_auth and configured with the litellm_jwtauth block, which maps token fields to LiteLLM entities: the team comes from team_id_jwt_field, the user from user_id_jwt_field, the organisation from org_id_jwt_field. It supports dot notation for nested claims, automatic creation of users and teams with user_id_upsert and team_id_upsert, role-based access control with enforce_rbac, per-team model restriction with enforce_team_based_model_access, and several issuers at once. The issuer’s public key is cached with a TTL of 600 seconds by default.

general_settings:
 enable_jwt_auth: true
 litellm_jwtauth:
 team_id_jwt_field: "groups"
 user_id_jwt_field: "sub"
 user_email_jwt_field: "email"
 enforce_rbac: true
 enforce_team_based_model_access: true
 team_id_upsert: true

OIDC for the admin interface. With Keycloak you use the generic provider, and the variables are PROXY_BASE_URL, GENERIC_CLIENT_ID, GENERIC_CLIENT_SECRET, GENERIC_AUTHORIZATION_ENDPOINT, GENERIC_TOKEN_ENDPOINT, GENERIC_USERINFO_ENDPOINT, plus a set of attributes to map the identifier, the email, the name and the role, and GENERIC_ROLE_MAPPINGS_* to translate Keycloak groups into LiteLLM roles. Login is rate-limited and sessions have a TTL of their own.

For automatic provisioning there is SCIM in the enterprise version, with base /scim/v2, and deprovisioning a user blocks their keys and takes them out of the authentication cache. That part is not in the open tree, so it cannot be verified against the code.

This fits with the identity work already covered in hardening and secrets of the sovereign stack and in the article on MCP authentication with Keycloak, which uses the same issuer.

Audit: what gets recorded

The table is LiteLLM_AuditLog and its columns are what you would expect from a change log:

id, updated_at, changed_by, changed_by_api_key,
action, table_name, object_id, before_value, updated_values

It stores the previous value and the new one, and attributes the change to an actor and to the key it was made with.

Three clarifications the documentation does not capture well.

The actions are six, in the past tense: created, updated, deleted, blocked, unblocked and rotated. The documentation mentions three. That blocking, unblocking and rotation are actions in their own right is exactly what is needed to reconstruct a credential’s life cycle for an auditor.

The scope is wider than advertised. Besides teams, users, keys and models, changes to the proxy configuration and to the SSO configuration are audited. A change of identity issuer is recorded.

The default depends on the licence. The resolution order is litellm_settings.store_audit_logs, then the variable LITELLM_STORE_AUDIT_LOGS, and if neither is set, it stays enabled in enterprise and disabled in the open version. An OSS deployment that assumes there is an audit trail does not have one:

litellm_settings:
 store_audit_logs: true

Two more pieces for a serious chain of evidence. The actor attribution can be passed with the LiteLLM-Changed-By header, and its use is restricted by the key’s or the team’s configuration, so any caller cannot forge it. And the events can be shipped out with audit_log_callbacks and its S3 parameters, which is what allows depositing them in an immutable store with object lock.

That last part is what turns the record into evidence. The Postgres table is mutable by anyone with database access, and as of September 2026 it has no indexes, so querying it by date range over a long history will not be comfortable either.

What this demonstrates to an auditor, and what it does not

This is the part that decides whether the setup is good enough for a regulated customer. With everything above configured, the platform sustains without difficulty:

Access control via corporate identity, with onboarding and offboarding in the identity provider and not in the gateway, and with the role determining which models are reachable. Segregation between tenants, with independent counters per organisation, team and person. Traceability of administrative changes, with actor, time, previous value and new value, exportable to an immutable store. And demonstrable consumption limits, with the evidence of the rejection in the spend log.

What it does not cover, and has to be solved elsewhere:

The record of the interactions is not this. The audit table records configuration changes, not model calls. The calls are in the spend log and in the traces, and traces are best-effort, which is the discussion in the first article in the series.

The integrity of the record is not something Postgres provides. Without export to immutable storage, the evidence is only as strong as the database permissions.

The content of the prompts is a separate problem, with its own personal-data treatment and its own decision about whether it is stored, masked or not logged at all.

And a warning about the control that tends to be shown first: while apply_user_budget_to_team_keys is not enabled, the per-person cap is not being applied to team keys. Showing that field in the interface as proof of a control that is not executed is the kind of finding a competent auditor discovers. The complete control mapping is in ENS, ISO 42001 and the EU AI Act.

Checklist

apply_user_budget_to_team_keys: true if personal budgets have to be enforced.
LITELLM_SALT_KEY set on day one, different from the master key and never rotated lightly.
store_audit_logs: true explicitly, without trusting the default.
audit_log_callbacks towards an immutable store if an audit has to be sustained.
Redis mandatory if there is more than one replica, or the limits get multiplied by the number of pods.
max_parallel_requests per key, on top of the per-minute limits, to protect the fleet from a runaway agent.
allowed_routes on application keys, so they cannot reach the management endpoints.
soft_budget with an alert on top, understanding that it does not block.
Rotation with a verified grace_period, because an invalid format revokes on the spot.
JWT against Keycloak as soon as there are more than a handful of teams, with enforce_rbac and enforce_team_based_model_access.

Traps and things that are not what they look like

A user’s cap is not enforced. They are using a team key and apply_user_budget_to_team_keys is missing.

Clients retry in a loop against an exhausted budget. The code is 429 and the OpenAI libraries retry 429s automatically.

The soft budget has stopped nothing. It does not block, it only logs.

The per-minute limits are enforced at double or triple. There is no Redis, and each replica keeps its own count.

The client does not see the limit headers. The names are nested and proprietary, and in streaming there is an open issue where they get lost.

There is no audit log. It is an OSS deployment and the default there is disabled.

The models stop authenticating after rotating the master key. There was no fixed LITELLM_SALT_KEY, and the encrypted provider credentials can no longer be decrypted.

The rotation cut the service on the spot. The grace_period carried a format the system did not understand and it carried on without it.

Blocking a key does not fire the alert. It is raised as a generic exception, without the type that expiry and model access do carry.

Closing

The keys and budgets layer is the one that turns a shared URL into a platform with tenants. It is well solved in LiteLLM and has more depth than the documentation suggests, with per-model budgets inside a key, multiple budget windows, per-tag limits and rotation that preserves the history.

Out of the whole article, two things take the attention. The apply_user_budget_to_team_keys line, because it is the difference between having a control and believing you have one. And setting LITELLM_SALT_KEY, because its default turns a routine rotation of the master key into an outage that is hard to diagnose. Both are one line of configuration and both are discovered late.

Sources

LiteLLM, Virtual Keys (parámetros de creación, ciclo de vida, rotación): https://docs.litellm.ai/docs/proxy/virtual_keys.
LiteLLM, Budgets, Rate Limits (jerarquía, presupuestos por nivel, clientes finales): https://docs.litellm.ai/docs/proxy/users.
LiteLLM, JWT Token Auth (bloque litellm_jwtauth y todos sus campos): https://docs.litellm.ai/docs/proxy/token_auth.
LiteLLM, Audit Logs y administración múltiple (la cabecera LiteLLM-Changed-By): https://docs.litellm.ai/docs/proxy/multiple_admins.
LiteLLM, SCIM (aprovisionamiento automático, versión enterprise): https://docs.litellm.ai/docs/tutorials/scim_litellm.
LiteLLM, litellm/proxy/auth/auth_checks.py en la etiqueta v1.100.0 (la condición de apply_user_budget_to_team_keys y los mensajes de presupuesto agotado): https://github.com/BerriAI/litellm/blob/v1.100.0/litellm/proxy/auth/auth_checks.py.
LiteLLM, litellm/proxy/management_endpoints/key_management_endpoints.py en v1.100.0 (rutas, rotación sobre la misma fila, periodo de gracia): https://github.com/BerriAI/litellm/blob/v1.100.0/litellm/proxy/management_endpoints/key_management_endpoints.py.
LiteLLM, litellm/proxy/hooks/parallel_request_limiter_v3.py y litellm/proxy/hooks/__init__.py en v1.100.0 (el limitador v3 como valor por defecto, la ventana, los scripts Lua): https://github.com/BerriAI/litellm/blob/v1.100.0/litellm/proxy/hooks/parallel_request_limiter_v3.py.
LiteLLM, litellm/exceptions.py en v1.100.0 (BudgetExceededError con código 429): https://github.com/BerriAI/litellm/blob/v1.100.0/litellm/exceptions.py.
LiteLLM, litellm/proxy/common_utils/encrypt_decrypt_utils.py en v1.100.0 (LITELLM_SALT_KEY cayendo en la clave maestra, y el comentario sobre la derivación de claves): https://github.com/BerriAI/litellm/blob/v1.100.0/litellm/proxy/common_utils/encrypt_decrypt_utils.py.
LiteLLM, schema.prisma en v1.100.0 (LiteLLM_VerificationToken, LiteLLM_AuditLog, la tabla de claves deprecadas): https://github.com/BerriAI/litellm/blob/v1.100.0/schema.prisma.
LiteLLM, issue #27748 — cabeceras x-ratelimit-* perdidas en respuestas en streaming con el limitador v3: https://github.com/BerriAI/litellm/issues/27748.

LiteLLM on day 2: one worker per pod, six thousand seconds of timeout and the other five things to change before opening the traffic

Tue, 08 Sep 2026 07:00:00 +0200

Second article in the operational track of the control layer. The operational pair with Langfuse covered the seam between the gateway and observability; here everything stays inside the gateway. The prior decisions, which gateway and why, are in choosing the OSS gateway and in the L7 inference router.

TL;DR

LiteLLM Proxy starts from a twenty-line config.yaml and serves traffic that same afternoon. The operational work consists of correcting seven defaults designed for a scenario other than an in-house inference factory.

One uvicorn worker per pod. The project’s production guide asks for --num_workers 1 on Kubernetes and horizontal scaling, with 1 vCPU and 4 GiB per worker set at the same time as requests and as limits. The 4 GiB are a floor, not a target: the Prisma query engine marks a resident memory high-water mark that grows up to the largest statement it has executed and that glibc does not return to the system.

Autoscale on CPU, never on memory. A direct consequence of the above. Memory goes up and never down, so an HPA on memory scales and never scales back. The recommended target is targetCPUUtilizationPercentage: 60.

The Postgres connection arithmetic. database_connection_pool_limit is 10, and the number of connections is instances times workers times the pool. The Helm charts ship maxReplicas: 100, on the order of a thousand connections, far above what a stock Postgres accepts.

Redis stops being optional from around a thousand requests per second or ten instances. Without it, each instance applies its limits on its own, caches are not shared, and spend updates against the same rows produce locks and exhaust the connections with FATAL: sorry, too many clients already.

Three health endpoints and only two are usable as probes. /health/liveliness and /health/readiness require no authentication and answer fast. Bare /health demands a key and fires a real request against every model in the catalogue, with its cost in tokens.

The resilience numbers are 2, 3 and 5: two retries, three allowed failures and five seconds of cooldown, the last one despite the function’s own documentation saying the default is 1.

request_timeout is 6,000 seconds. A hundred minutes holding a connection against an engine that does not answer.

And a warning about the performance figures: the project publishes 8 ms of p95 overhead on one benchmark and 257.7 ms of p99 on another. Both are theirs, and the difference is in the shape of the load.

You are here: the gateway layer, the day after

In the seven-layer stack, this article lives entirely in the gateway layer. The difference with the selection article is the one that separates buying from maintaining: there the question was which piece to put in front of the fleet, and here it is what happens when that piece has been serving for six months, the team has tripled the traffic and someone has added an agent that fires twenty calls per interaction.

The gateway has an uncomfortable property worth keeping in mind: it sits on the critical path of every request and contributes not a single token. All it does is coordination, and any coordination that fails turns into an outage of the whole inference service, even with the GPUs perfectly healthy.

The analogy: the switchboard and its notebook

A hospital switchboard does three jobs at once. It puts calls through, which is the urgent part. It writes down in a notebook who called, to which extension and for how long, which is what sustains the billing. And it consults a list of extensions to know where to put each call through.

The three jobs compete. If the operator stops to write each line in the notebook before putting the next call through, the waiting queue grows. If there are three operators and each keeps her own notebook, the totals do not add up. And if the list of extensions is consulted in a filing cabinet on another floor, every call costs the trip.

The three tensions have an exact equivalent in LiteLLM. The notebook is the SpendLogs in Postgres, which is why the transaction buffer in Redis exists. The three operators with three notebooks are the pod replicas applying limits independently. And the filing cabinet on another floor is usage-based routing, which adds a Redis lookup inside the request path.

The process model: one worker per pod

The production documentation is explicit: on Kubernetes, --num_workers 1 and scale with replicas. On a virtual machine with no orchestrator, NUM_WORKERS equal to the number of vCPUs.

The reason is not ideological. An additional worker inside the same pod shares the container’s memory limit with the others, multiplies the database connections by the same factor, and registers a copy of every periodic job. With separate replicas, each one has its own resource budget, its own connection quota and a place where the scheduler can put it.

The recommended sizing is 1 vCPU and 4 GiB per worker, and the part that gets overlooked is that these go as requests and as limits at the same time. Setting less than 4 GiB produces an OOM restart loop that shows up late, once some large request has grown the query engine’s footprint.

resources:
 requests: { cpu: "1", memory: "4Gi" }
 limits: { cpu: "1", memory: "4Gi" }

That footprint needs to be described precisely, because it determines the autoscaling policy. The Prisma query engine keeps a resident memory that behaves as a high-water mark: it grows up to the largest statement the process has executed and does not come back down, because glibc does not return that freed memory to the system. A pod that served a spike yesterday still shows the footprint of that spike today.

Two rules come out of this. The first: autoscale on CPU and never on memory. An HPA on memory sees a metric that only goes up, scales at the peak and never scales back. The recommended target is targetCPUUtilizationPercentage: 60, lower than the 80 the charts ship by default, because the startup probe allows up to 300 seconds before passing the first readiness check and there needs to be headroom for the new pod to be ready before the saturated one goes down.

The second: bound the process lifetime. --max_requests_before_restart 10000 recycles the worker before the high-water mark matters.

Postgres: the arithmetic that blows up when you scale

database_connection_pool_limit is 10 by default. The total number of connections the deployment opens is replicas times workers times pool, and it has to be compared against the Postgres max_connections.

The formula the documentation gives is the inverse, and it is the one to apply when sizing:

database_connection_pool_limit = MAX_DB_CONNECTIONS / (instances × workers)

The detail that hurts in production is in the Helm charts, which ship autoscaling.maxReplicas and keda.maxReplicas at 100. With the default pool, that is on the order of a thousand connections at full scale-out, far above what an untouched Postgres accepts. And the moment the HPA reaches those replicas is exactly the moment of highest load, so connection exhaustion arrives when it can least be afforded.

There are two more valves on the same path. Provider errors are written to the database by default, and under a sustained engine failure that inflates the spend table without contributing anything that is not already in the metrics. They are switched off like this:

general_settings:
 disable_error_logs: true # stops writing provider errors
 proxy_batch_write_at: 60 # batches the spend writes

disable_spend_logs: true is the radical version, which removes the per-request detail from the interface and leaves the cost in Prometheus and in the tracing backend. Before going that far you have to decide whether that table is the system’s audit record, which is the discussion in the previous article: if it is, it cannot be switched off, and the way out is the Redis buffer.

Redis: when it stops being optional

Without Redis, LiteLLM works. Each instance keeps its own in-memory cache and applies its own counters. The consequences are three, and all of them show up as you grow.

Requests-per-minute limits are applied per instance, so a limit of 100 with five replicas is in practice a limit of 500. Cache hits are local, so the same repeated question hits once in five. And there is no leader election, so periodic jobs run in every process at once.

The threshold the documentation gives for enabling the transaction buffer is a thousand requests per second or ten instances. Below that, each instance updates the key, user and team rows directly; above it, they all write to the same rows, locks appear and Postgres starts rejecting with FATAL: sorry, too many clients already.

general_settings:
 use_redis_transaction_buffer: true

The metrics to watch when that is enabled are litellm_in_memory_spend_update_queue_size, litellm_redis_spend_update_queue_size and litellm_pod_lock_manager_size. If the first grows without the second coming down, the flush to Postgres is not keeping up.

A note about the routing strategy, which interacts with this. simple-shuffle consults nothing external. Usage-based routing does, and it adds a round trip to Redis inside the request path. On a homogeneous vLLM fleet behind the same model, the improvement it brings rarely pays for that latency; on a heterogeneous fleet, the conversation is different and the router article covered it.

Background jobs register per worker

This detail appears in no getting-started guide and produces considerable bewilderment when it is discovered. The proxy’s periodic jobs register per uvicorn worker, not per pod. With --num_workers 4 and ten replicas that is forty copies of every job, all of them running with no coordination if there is no Redis for leader election.

The variable that separates them is LITELLM_JOB_ROLE: value serving on the pods that serve traffic and a separate replica with value worker for the jobs. With the one-worker-per-pod recommendation the problem shrinks, but it does not go away while there are several replicas.

Probes: three endpoints and only two are usable

The three endpoints exist and do different things:

Endpoint	Authentication	What it does	What it is for
`/health/liveliness`	No	Returns `I'm alive!`, or 503 during shutdown	`livenessProbe`
`/health/readiness`	No	State of the process and the database. 503 if the configured Postgres does not answer	`readinessProbe`
`/health`	Yes	Fires a real request against every model in the catalogue	Manual diagnosis, never a probe

The last row deserves insisting on. /health spends tokens on every call, one per configured model. Set as a probe, with the Kubernetes default interval and a fleet of eight models, that is thousands of inference requests a day that serve nobody.

There are more endpoints useful for diagnosis: /health/readiness/details, /health/services?service=langfuse to check one specific callback, /health/history and /health/latest for the history, and /health/backlog, which returns the in-flight requests and shows up in the latency section. /health/drain exists but returns 404 unless it is enabled with enable_drain_endpoint: true, and it is protected with X-Drain-Token.

The periodic model checks are configured separately and not in the probe:

general_settings:
 background_health_checks: true
 health_check_interval: 300 # seconds
 background_health_check_model_groups: ["llama-70b", "qwen-30b"]

And one setting with consequences in deployments with no Internet egress: allow_requests_on_db_unavailable: true allows serving traffic with the database down, but leaves /health/readiness always returning 200. With that set, the readiness probe stops detecting the very failure it was designed to detect, and budget and model errors keep blocking just the same.

Retries, cooldowns and fallbacks: the real numbers

The defaults, read from the project’s code and not from the documentation:

Constant	Value	What it governs
`DEFAULT_MAX_RETRIES`	2	Retries before moving on to fallbacks
`DEFAULT_ALLOWED_FAILS`	3	Failures before cooling down a deployment
`DEFAULT_COOLDOWN_TIME_SECONDS`	5	Seconds a deployment stays out
`SINGLE_DEPLOYMENT_TRAFFIC_FAILURE_THRESHOLD`	1000	Minimum requests before applying the cooldown logic with a single deployment
`DEFAULT_REQUEST_TIMEOUT_SECONDS`	6000.0	Request timeout

Besides the failure counter, there is a proportional criterion: a deployment enters cooldown if 50% of its requests fail in any given minute.

On the cooldown there is a discrepancy worth knowing about before debugging blind. The docstring of the router function says cooldown_time defaults to 1, and the code uses the constant of 5. The code wins.

The fallbacks are three separate lists and are applied after the retries are exhausted:

router_settings:
 num_retries: 2
 timeout: 120
 fallbacks: [{"llama-70b": ["qwen-30b"]}]
 context_window_fallbacks: [{"llama-70b": ["llama-70b-128k"]}]
 content_policy_fallbacks: [{"llama-70b": ["llama-70b-sin-guardrail"]}]

A note on the behaviour when everything is in cooldown: if no deployment is left available in the group, LiteLLM falls back onto a specific model_info.id, skipping the cooldown check. It is a deliberate degradation, serve something rather than return 503, and it is worth knowing it exists because it masks the real state of the fleet.

And a capability lost along the way: since version 1.85.0, the mock_testing_fallbacks parameter and its two siblings are stripped from requests arriving through the proxy and have no effect. Fallbacks can no longer be tested against the proxy with a doctored request; it has to be done against litellm.Router directly, in a test. Anyone who had a smoke test built on that has it broken without warning.

The six thousand second timeout

It deserves its own section because it is the setting with the worst damage-to-effort ratio in the whole system. request_timeout is 6,000 seconds. A hundred minutes. An inference engine that stops answering without closing the connection holds a worker for that long, and with one worker per pod that is an entire pod out of service per trapped request.

general_settings:
 request_timeout: 600

Ten minutes is still generous for a long generation, and it bounds the damage.

Overhead: the project’s two figures

LiteLLM publishes two measurements of its own overhead that differ by a factor of thirty, and both are legitimate because they measure different loads.

The benchmarks page, with Locust, a thousand users with think time and a fake endpoint, gives for four instances an x-litellm-overhead-duration-ms of median 2 ms, p95 8 ms and p99 13 ms at 1,170 requests per second. With two instances, median 12 ms, p95 29 ms and p99 43 ms. The page itself warns that those figures correspond to around 130 in-flight requests, and that a closed-loop client with no think time keeps a thousand in flight and sees on the order of eight times that latency at the same rate, by Little’s law.

The AIGatewayBench benchmark, from July 2026, measures the Python proxy at 257.7 ms of added p99 and 329.5 MB of peak memory, with no callbacks, no spend tracking and no persistence. The Rust variant, in beta, gives 0.7 ms.

The reading for an in-house platform: the first figure is the best case with human clients, and the second is the order of magnitude to plan for agentic load, which arrives in a closed loop against fast responses. There is one reported case that resembles this scenario closely: an in-house OpenAI-compatible engine served around 16 requests per second directly and around 9 through LiteLLM, with the degradation growing with concurrency, on a 4 vCPU, 8 GB pod.

The latency gap the metrics cannot see

LiteLLM’s timers start when its handler starts. The time the request spends queued in the uvicorn event loop, before getting there, appears in none of its metrics. The example in its own troubleshooting guide is a case in which LiteLLM records 10 seconds and the user experiences 20.

It is detected by comparing two sources: GET /health/backlog or the litellm_in_flight_requests gauge against the response time measured by the load balancer in front. If they diverge, the gap is in the input queue and the answer is more replicas, not more tuning.

Two headers are always on and serve to watch it without instrumenting anything: x-litellm-overhead-duration-ms and x-litellm-callback-duration-ms. For the second, the documentation sets the suspicion threshold at 100 ms, above which the diagnosis is that the payloads are too large.

And the first cause of latency that guide mentions is none of the above: LITELLM_LOG=DEBUG serialises the payload with json.dumps(indent=4) synchronously, and with payloads over 2 MB that alone can cost between 2 and 5 seconds per request. A log level set to debug a problem and forgotten there produces exactly the problem it was meant to debug.

The failure mode to put in the runbook

There is an open issue describing a cascade with a characteristic shape. Under sustained 429s from the engine upstream, the pods stopped answering readiness probes during startup, Kubernetes killed them on failed probes, and the restart put more pressure on an upstream that was already saturated. Peaks of 500 requests per second, average of 60, between two and five replicas with 1.3 CPU and 4 GB.

The shape of the failure is what matters more than the specific case: the engine’s saturation turns into a restart loop of the gateway, and from the outside it looks like the problem is the gateway. The defences are the ones already in this article, applied together: startupProbe with a wide margin so readiness does not compete with startup, request_timeout bounded so trapped requests get released, cooldowns that take the deployment returning 429 out of rotation, and CPU autoscaling at 60% so the capacity exists before it is needed. It fits with what was covered in incident response runbooks.

Upgrading without cutting traffic

The deployment is a normal Deployment with RollingUpdate, maxUnavailable: 0 and maxSurge: 1, and there is no further mystery to it except on one point: schema migrations. LiteLLM uses Prisma and applies migrations at startup. With several replicas coming up at once after a version change, the migration must be run by a single process, which is another reason for the dedicated replica with LITELLM_JOB_ROLE: worker.

Before a version bump, two checks that pay for themselves: read the release notes looking for changes in the metadata emitted to the callbacks, because a change like that breaks saved dashboards without touching the service, and verify that the new configuration starts with --detailed_debug on a separate pod before applying it to the fleet.

Go-live checklist

--num_workers 1 and scaling by replicas.
requests equal to limits, 1 vCPU and 4 GiB per worker.
HPA on CPU at 60%. Never on memory.
database_connection_pool_limit computed from max_connections, and the chart’s maxReplicas lowered to something the database can take.
Redis as soon as there is more than one replica, and use_redis_transaction_buffer from ten instances or a thousand requests per second.
livenessProbe on /health/liveliness, readinessProbe on /health/readiness, and /health out of the probes.
startupProbe with a margin of up to 300 seconds.
request_timeout: 600.
LITELLM_LOG out of DEBUG, and disable_error_logs: true.
A dashboard with x-litellm-overhead-duration-ms, x-litellm-callback-duration-ms, litellm_in_flight_requests and the response time of the load balancer in front.

Traps and things that are not what they look like

The HPA scales up and never back down. It is measuring memory, which in this process only goes up.

The pods die in a loop right after deploying. Less than 4 GiB of limit. The footprint appears with the first large request, not at startup.

Postgres starts rejecting connections at the peak. The chart has maxReplicas: 100 and the pool defaults to 10.

The per-minute limits are enforced badly. There is no Redis, and each replica keeps its own count.

The health check consumes tokens. Someone set /health as the readiness probe.

The readiness probe returns 200 with the database down. allow_requests_on_db_unavailable is set.

A deployment keeps receiving traffic after failing. The cooldown is 5 seconds, not minutes, and with a single deployment in the group you have to pass a thousand requests for the logic to apply.

The fallback test no longer tests anything. The simulation parameters are stripped from requests arriving through the proxy since 1.85.0.

The p99 is ten times worse than the published benchmark. The benchmark has think time and the real load is agentic.

Closing

None of these settings is complicated and all of them are one line. What unites them is that their defaults describe a small deployment, with one instance, a human client on the other side and a commercial provider taking care of capacity. An in-house inference factory is the opposite on all four axes: several replicas, automated clients, and an engine whose capacity is finite and known.

Of the ten points in the checklist, three decide almost everything in an on-premise deployment: the process model with its sizing, the database connection arithmetic, and the request timeout. The other seven are the ones that avoid the Saturday phone call.

Sources

LiteLLM, Production best practices (topología, workers, memoria, pool de conexiones, LITELLM_JOB_ROLE, buffer de transacciones en Redis, request_timeout): https://docs.litellm.ai/docs/proxy/prod.
LiteLLM, Health checks (los tres endpoints, comprobaciones de fondo, enable_drain_endpoint): https://docs.litellm.ai/docs/proxy/health.
LiteLLM, Reliability (fallbacks, reintentos, deprecación de los parámetros de simulación en la 1.85.0): https://docs.litellm.ai/docs/proxy/reliability.
LiteLLM, Benchmarks y Load test advanced (las cifras de sobrecarga por número de instancias y la advertencia de la ley de Little): https://docs.litellm.ai/docs/benchmarks · https://docs.litellm.ai/docs/load_test_advanced.
LiteLLM, Rust AI Gateway benchmarks (los 257,7 ms de p99 del proxy en Python): https://docs.litellm.ai/blog/rust-ai-gateway-benchmarks.
LiteLLM, Latency overhead (el hueco de la cola de uvicorn, las cabeceras de diagnóstico, el coste de LITELLM_LOG=DEBUG): https://docs.litellm.ai/docs/troubleshoot/latency_overhead.
LiteLLM, DB info (disable_spend_logs, disable_error_logs y qué se pierde con cada uno): https://docs.litellm.ai/docs/proxy/db_info.
LiteLLM, litellm/constants.py (los valores por defecto de reintentos, fallos permitidos, enfriamiento y timeout): https://github.com/BerriAI/litellm/blob/main/litellm/constants.py.
LiteLLM, litellm/router.py (la discrepancia entre el docstring de cooldown_time y la constante que usa el código): https://github.com/BerriAI/litellm/blob/main/litellm/router.py.
LiteLLM, issue #15526 — cascada de 429 sostenidos, sondas de readiness fallidas y bucle de reinicios: https://github.com/BerriAI/litellm/issues/15526.
LiteLLM, issue #21046 — degradación de throughput contra un motor propio compatible con OpenAI: https://github.com/BerriAI/litellm/issues/21046.

LiteLLM and Langfuse: the operational pair, and the four places where the trace is lost between the gateway and the dashboard

Mon, 07 Sep 2026 18:00:00 +0200

This post opens the operational track of the control layer. The previous articles left the pieces chosen and assembled: choosing the OSS gateway and the L7 inference router explain why LiteLLM ends up in front of the fleet, and Langfuse from the inside takes apart the backend that receives the traces. Here we deal with the seam between the two, which is where the work of the next three years lives.

TL;DR

The LiteLLM plus Langfuse pair is configured with a list of callbacks and two API keys, and from there four matters decide whether the dashboard is good for operating or only for showing in a demo.

There are three integration routes and they are not equivalent. The native langfuse callback is tied to the Python SDK v2 (langfuse>=2.59.7,<3.0 in LiteLLM’s pyproject.toml), which writes against the legacy ingestion that Langfuse Cloud retires on 16 November 2026. The langfuse_otel preset exports over OTLP and is the path recommended in the current documentation of both projects. The OTel v2 route, behind the LITELLM_OTEL_V2=true flag, changes span names, leaves the success status at UNSET and, above all, its Langfuse mapper does not read the request metadata: session_id, trace_user_id, tags and trace_id stop arriving.

The cost of a self-hosted model comes out as zero unless a price is registered. get_model_info() raises an exception for a model that is not in the pricing map, completion_cost() propagates it, the logger catches it, stores it in a debugging field and returns None, and the payload turns it into 0.0. There is no warning at the default log level. A team’s budget is never exhausted because its spend is always zero.

Client-gateway-engine correlation does not come out of the box. LiteLLM does continue the client’s trace if a traceparent arrives, and it also extracts from it the 32-hex trace-id to use as litellm_trace_id, which makes the Langfuse identifier equal to the W3C one. Upstream the story changes: forward_traceparent_to_llm_provider is False by default, the function that applies it returns early if no otel logger is loaded, and when it does act it forwards the client’s original header, so the vLLM span ends up a sibling of LiteLLM’s rather than a child. Add to that the fact that vLLM exports OTLP over gRPC by default and Langfuse does not accept gRPC.

Between the request and the dashboard there are four bounded queues that drop events. LiteLLM’s logging thread pool (100 threads, 10,000 pending tasks, dropping once full), the asynchronous LoggingWorker (50,000, and once full it aggressively flushes half), OTel’s BatchSpanProcessor (2,048 spans by default) and Langfuse’s ClickhouseWriter, which drops rows after exhausting the retries with no dead-letter queue. None of the four applies back pressure on the request, which is exactly what you want, and none of them guarantees delivery, which is exactly what an auditor asks about.

The rest of the article develops the four points with configuration applicable to a generic 4×H100 cluster with vLLM, the overhead figures the project itself publishes, and the honest limit of what this data sustains before ENS or ISO/IEC 42001.

You are here: the seam between DEPLOY and OBSERVE

In the seven-layer stack, LiteLLM lives in the gateway layer and Langfuse in the observability one. The separation is clean on the diagram and false in operation: the gateway is the only point in the system that sees the complete request before chopping it up and the complete response after reassembling it, so it is also the only reasonable place from which to emit the trace. The inference engine sees tokens and latencies; it does not see who is asking or how much that team has spent this month.

That turns LiteLLM into a telemetry producer as well as a traffic distributor, and moves onto the gateway a responsibility that was not in the original design: if the callback fails, the request is served all the same and nobody notices that the dashboard has been incomplete for two days.

The analogy: the toll booth that issues the ticket

A motorway toll booth does two independent jobs. It raises the barrier, which is its visible function, and it issues a ticket with number plate, time, section and amount, which is the function the concession lives off. The two share a booth and do not share a destination: the barrier responds in a hundred milliseconds because there is a car waiting, and the ticket travels to a processing centre over a slower channel that batches thousands of records before sending them.

The design is well thought out as long as its consequence is understood. If the processing centre goes down, cars keep going through. If the ticket queue fills up, tickets are thrown away, traffic is not stopped. The concession accepts losing records rather than causing a jam, and charges on a statistical sample it knows to be imperfect.

LiteLLM and Langfuse work the same way. The four queues in the corresponding section are the slow channel, and all of them prefer to drop rather than block. Whoever builds the pair inheriting the complete analogy gets the operation right. Whoever thinks the ticket is an accounting entry ends up explaining to an auditor why traces from a Tuesday afternoon are missing.

Three routes, and one with an expiry date

As of September 2026 three paths coexist for getting a request from LiteLLM to Langfuse. LiteLLM’s documentation already labels the first as legacy.

Route	How it is enabled	Transport	Status
Native `langfuse` callback	`success_callback: ["langfuse"]`	Langfuse Python SDK v2, legacy ingestion	Maintained for compatibility. Tied to `langfuse<3.0`
`langfuse_otel` preset (v1)	`callbacks: ["langfuse_otel"]`	OTLP/HTTP against `/api/public/otel`	Recommended by both projects
OTel v2 route	`callbacks: ["langfuse_otel"]` + `LITELLM_OTEL_V2=true`	OTLP/HTTP, new mapper	Opt-in. Changes semantics and loses metadata

The date that orders the decision is 16 November 2026. Langfuse’s compatibility matrix marks the legacy batch ingestion of the Python SDK v2 as retired on that date for Cloud, at which point trace ingestion through that route stops working. On self-hosted the endpoint survives, but a v4 server in events_only mode returns 400 for every event type except score-create and sdk-log.

Since LiteLLM’s native callback is pinned to langfuse>=2.59.7,<3.0 in its own pyproject.toml, and its code calls APIs that only exist in SDK v2 (self.Langfuse.trace(...), trace.generation(...)), there is no way to move up an SDK without changing route. Anyone who today has success_callback: ["langfuse"] against Langfuse Cloud has a migration with a date on it. Anyone who has it against a self-hosted Langfuse has one with some slack, because v3 receives security patches until the end of January 2027, but they have one all the same.

One detail of the OTLP route worth checking in older deployments: the x-langfuse-ingestion-version: 4 header is the one that enables real-time ingestion. Without it the traces arrive, but with a delay that Langfuse’s documentation puts at around ten minutes (the compatibility page says fifteen, and that discrepancy remains unresolved in their own documents). LiteLLM started sending it in version 1.95.0, of August 2026. A proxy older than that version against a Langfuse v4 produces the classic symptom: the dashboard works, but it is always behind and nobody knows why.

The configuration that works today

For a generic 4×H100 cluster with vLLM behind it and Langfuse self-hosted in the same Kubernetes, the OTLP route in its v1 variant is the one that gives the most functionality with the fewest surprises.

model_list:
 - model_name: llama-70b
 litellm_params:
 model: hosted_vllm/meta-llama/Llama-3.3-70B-Instruct
 api_base: http://vllm-llama70b.inference.svc.cluster.local:8000/v1
 api_key: os.environ/VLLM_API_KEY
 # Price derived from the cluster's €/GPU-hour. Without this, spend = 0.0
 input_cost_per_token: 0.00000031
 output_cost_per_token: 0.00000089

litellm_settings:
 callbacks: ["langfuse_otel"]
 turn_off_message_logging: false # a conscious decision, see the PII section
 langfuse_default_tags: ["model_group", "user_api_key_team_alias", "cache_hit"]

general_settings:
 disable_error_logs: true # provider errors inflate SpendLogs
 proxy_batch_write_at: 60

The credentials travel through the environment. LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY and LANGFUSE_HOST are enough: the preset builds the Authorization: Basic base64(public:secret) header and adds the ingestion version one, so there is no need to touch OTEL_EXPORTER_OTLP_HEADERS by hand. The preset’s default host points to the US region of Langfuse Cloud, a relevant detail for anyone building a platform with data sovereignty requirements: if the variable is not defined, the telemetry leaves Spain without anyone having decided it.

For real multi-tenancy, LiteLLM supports per-team Langfuse credentials (POST /team/<id>/callback with callback_vars) and per-key ones (metadata.logging[] when generating the key), and the key’s wins over the team’s without mixing. There is a limitation that breaks the architecture of anyone who planned to separate tenants across different Langfuse instances: under OTel v2, a per-key or per-team langfuse_host does not move that tenant to another host, because the exporter fixes the destination at startup and the only thing that varies per tenant are the headers.

And one operational warning with years of history behind it: MAX_LANGFUSE_INITIALIZED_CLIENTS is 50 by default because each Langfuse client starts a thread. The comment in LiteLLM’s code is explicit about the incident that motivated it, with the CPU at 100% from repeated initialisations. A deployment with per-team credentials and many teams hits that ceiling.

What is lost when moving to OTel v2

The LITELLM_OTEL_V2=true flag brings a step up in quality in the proxy’s general instrumentation, and at the same time a concrete regression on the path to Langfuse that the documentation acknowledges without underlining it.

The Langfuse mapper of the v2 route emits langfuse.observation.type, langfuse.observation.model.name, the observation identifier, the model parameters, input and output, usage_details and cost_details, plus langfuse.trace.metadata.team_id and team_alias. It does not emit session.id, nor user.id, nor langfuse.trace.tags, nor langfuse.trace.id, nor the generation name, nor parent_observation_id. LiteLLM’s documents put it this way: those attributes are set by the preset from the request and the response, not from metadata supplied by the client.

The practical consequence is that an application that today groups conversations by sending metadata: {"session_id": "...", "trace_user_id": "..."} in the request body stops grouping them when the flag is enabled, with no error and no warning. The traces keep arriving, loose.

The other v2 changes are semantic and affect alerts already written. The root span goes from being called Received Proxy Server Request to being named after the route (POST /v1/chat/completions), the inference span goes from litellm_request to {operation} {model}, the guardrail spans stop hanging off the inference span and hang off the root instead, and the success status stays at UNSET instead of OK. Any rule that fires on status == OK stops firing. The way back is to remove the flag and restart.

Cost: why the trace says zero

This is the part that takes the most time to diagnose and the least to fix.

LiteLLM computes the cost of a request from a per-model pricing map. For commercial provider models the map comes filled in. For hosted_vllm/whatever, which is all there is in an in-house inference factory, the map is empty, and the failure path is as follows:

litellm.get_model_info() raises an exception with the message This model isn't mapped yet.
completion_cost() and response_cost_calculator() propagate the exception.
Logging._response_cost_calculator() catches it, writes a StandardLoggingModelCostFailureDebugInformation into response_cost_failure_debug_information, logs it at debug level and returns None.
The standard logging payload does llm_response_cost = raw_response_cost or 0.0.

The system delivers a spend of zero, with no visible error at the default log level, and writes it into the spend column of LiteLLM_SpendLogs. The per-key, per-team and per-organisation budgets are evaluated against that zero and are never exhausted. The Langfuse dashboard shows a total cost of zero for everything self-hosted, which is precisely the part you wanted to measure.

Registering the price

The pricing keys are accepted both in litellm_params and in model_info, and those in litellm_params win when they are in both places. The complete set has 91 fields, but for a deployment with vLLM two are enough, and there are three things to know about them.

The first is that the price is per individual token, in USD. There are no per-thousand variants: searching for per_1k or per_1000 in the pricing model returns nothing. A three-orders-of-magnitude error here is easy to make and hard to see, because it produces plausible figures.

The second is that input_cost_per_request exists in the schema and is not applied in chat calls: a model registered with that field returns a cost of 0.0. To charge per request it has to be converted into a cost per token.

The third is that input_cost_per_second needs the response time and is computed as price times response_time_ms / 1000, so outside the proxy’s request path it silently returns zero.

The number that goes into those two keys comes from the article on FinOps and multi-tenancy with LiteLLM and the one on cost per token and per request: amortisation plus energy divided by tokens served. The example arithmetic, to set the order of magnitude: a cluster whose fully loaded cost is 9.60 USD per hour sustaining 3,000 output tokens per second produces 10.8 million tokens an hour, that is 0.00000089 USD per output token. That is the value that goes into output_cost_per_token. If the cluster’s average utilisation is 30 % and not 100 %, the divisor changes and the price triples, which is the same lever that article develops.

Before sending traffic, the /spend/calculate endpoint accepts the same parameters as completion_cost and returns the computed cost. It is the cheap way to check that the price resolves.

How that number reaches Langfuse, and who wins

The two integration routes write the cost into different fields. The native callback sends it as usage.total_cost of the v2 schema, and duplicates it in the trace metadata as litellm_response_cost. The OTLP preset emits langfuse.observation.cost_details as a JSON string of the form {"total": cost}, and langfuse.observation.usage_details with input, output and total.

The rule on the Langfuse side is documented and is the right one for this case: when there is ingested cost and inferred cost, the ingested one takes priority. Langfuse only computes from its own pricing table when the emitter sends no cost, and changes to a model definition apply to new generations, with no retroactive recalculation. A self-hosted model therefore needs no pricing definition in Langfuse: it is enough for LiteLLM to send cost_details.

One detail remains that explains a puzzling symptom. Langfuse stores the ingested cost and the computed one in separate columns, totalCost and calculatedTotalCost, and the session header totals add up only the first. A mixed deployment where part of the traffic brings cost and part lets Langfuse compute it shows sessions at zero while the individual generations show cost. There is an open issue about it.

The tokens of a streaming response

Almost all of an assistant’s traffic is streamed, and there the OpenAI usage object only appears if the client asks for stream_options.include_usage. LiteLLM solves this by injecting the option on its own: on the asynchronous completion routes with stream: true, if the caller did not set it and all the candidate deployments support it, it adds it and marks _litellm_strip_stream_usage so the usage block is used in the cost calculation and not returned to the client. The general_settings.always_include_stream_usage flag forces or disables that behaviour, and is worth setting to True when capability detection does not recognise the vLLM deployment.

When the usage is missing anyway, LiteLLM reassembles the response with stream_chunk_builder() and counts tokens with the tokeniser. It is an approximation, and it produces the second path towards a cost of zero. The two paths to zero are told apart by looking at total_tokens in LiteLLM_SpendLogs: zero tokens with a correct price is a usage problem, correct tokens with zero cost is a price problem.

Correlation: from the client to the engine

The operational question that justifies the whole setup is being able to take a user complaint, find their request and see where the time went. That requires the trace identifier to survive three hops.

What does work

If the client sends traceparent, LiteLLM’s spans hang off the existing trace. In addition, and independently of the OTel subsystem, litellm_pre_call_utils.py extracts by regular expression the 32-character hexadecimal trace-id and assigns it to data["litellm_trace_id"] as a last resort, and takes the session_id from the W3C baggage header. That makes the Langfuse trace identifier equal to the W3C trace identifier, which is the most robust correlation key available: with it, an identifier copied from the application log opens the trace in Langfuse with no translation.

For the routes that are not OTel v2, moreover, headers prefixed with langfuse_ inject per-request metadata. langfuse_trace_id, langfuse_trace_user_id and langfuse_trace_metadata are documented, but the code strips the prefix and writes the rest into the metadata, so any body key works as a header. Two warnings about this: header values arrive as strings and are not interpreted as JSON, and Nginx drops headers with underscores by default (underscores_in_headers off), which produces the symptom that the metadata works locally and disappears behind the ingress.

What does not work without effort

Towards the engine, propagation is off. The flag is litellm.forward_traceparent_to_llm_provider, it is False, and its implementation has three behaviours to know about before enabling it:

The function returns immediately if open_telemetry_logger is None. A deployment with callbacks: ["langfuse_otel"] and without the generic otel callback propagates nothing, whatever the flag is set to.
When it does propagate, it copies the client’s traceparent header as is, instead of injecting the context of LiteLLM’s active span. The vLLM span ends up hanging off the client’s span-id, a sibling of the gateway call’s span. The trace is one, the hierarchy is wrong.
If the client sends no traceparent, none is created.

Add to that the fact that LiteLLM’s general policy is not to forward client headers to the provider, and that the forward_client_headers_to_llm_api allowlist covers headers prefixed with x-, among which traceparent is not. This is the only route.

The vLLM side, and why the Collector stops being optional

vLLM enables traces with --otlp-traces-endpoint, and --collect-detailed-traces with model, worker or all adds forward and execute times, with a performance impact its own documentation warns about. The attributes it emits are in vllm/tracing/utils.py and are the most useful part of the setup: gen_ai.latency.time_in_queue, time_to_first_token, time_in_scheduler, time_in_model_prefill, time_in_model_decode, time_in_model_forward, time_in_model_execute, time_in_model_inference and e2e, plus the token counters. That breakdown is what turns a “it’s slow” into “the time goes in the queue, not in decode”, which is the conversation you want to have during an incident. It fits with what was already covered in instrumenting vLLM with OTel and with the metrics in DCGM and vLLM.

There are three constraints that decide the architecture:

vLLM exports over gRPC unless OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf is set, and its gRPC exporter is built with insecure=True hard-coded. Langfuse, for its part, does not support gRPC. And vLLM emits the old convention names (gen_ai.usage.prompt_tokens, gen_ai.usage.completion_tokens), renamed in the specification to input_tokens and output_tokens, plus a gen_ai.latency.* family that was never standard and that the file itself documents as its own attributes until they are normalised.

An OpenTelemetry Collector in between solves all three at once: it receives gRPC, exports HTTP with the Basic header towards /api/public/otel/v1/traces, and renames attributes with a transform. It is also the only place where tail sampling can be done, which is the next point. And add the warning from the llm-d community, which holds for any deployment with Envoy or sidecars in between: OTEL_SERVICE_NAME has to be set per engine, because vLLM sends no service name and without it every span arrives as unknown_service.

The four serial queues

A request’s telemetry crosses four bounded buffers before it is queryable. All four drop once full and none blocks the request.

Queue	Capacity	What it does once full
LiteLLM logging thread pool	100 threads, 10,000 pending tasks	Drops the new ones with a rate-limited warning
Asynchronous `LoggingWorker`	50,000 events, concurrency 100	Aggressively flushes 50 % of the queue; `enqueue()` documents that it never blocks
OTel’s `BatchSpanProcessor`	2,048 spans (`OTEL_BSP_MAX_QUEUE_SIZE`), batch 512, sent every 5 s	Drops spans silently
Langfuse’s `ClickhouseWriter`	Bounded retries	Drops rows after exhausting retries, with no dead-letter queue

The first is documented in LiteLLM’s own code with a sentence that saves half an hour of discussion: logging is best-effort and, once the cap is reached, new entries are dropped with a rate-limited warning instead of being queued forever. The third is a barely visible consequence: LiteLLM builds the BatchSpanProcessor without adjusting max_queue_size, so the OTel SDK defaults govern, and no LiteLLM documentation mentions the OTEL_BSP_* variables that change them. The Langfuse SDK does the same on its own account: it sets batch size and interval, and leaves the queue size at the default.

On top of this comes a fifth loss point that is not a queue: LiteLLM does not call flush() per request on the native route, so a SIGKILL on the pod takes the last batch with it. On Kubernetes, this is an argument for a generous terminationGracePeriodSeconds and a preStop that gives some slack.

None of this is a defect. It is the right design decision for a piece that sits on the critical path of inference. What changes is what can be asserted with the resulting data.

What cannot be promised to the auditor

A dashboard built on this pipeline answers statistical questions well: what is the p95 of time to first token for the model group, which team consumes the most, what proportion of requests fails, how cost per request evolves. That is what it is designed for, and that is what it is good for.

It does not answer completeness questions well, which are the ones an ENS auditor or an ISO/IEC 42001 management system auditor asks. Faced with “show me that I have a record of every interaction with the AI system during March”, the honest answer is that the traces are a best-effort record with four documented points of silent dropping, with no dead-letter queue on the last hop and no durability guarantee on any of them.

The clean separation is the one already proposed in technical controls ENS, 42001 and the AI Act: the regulatory audit record is a different artefact, with a different write path, durability guarantees and retention of its own. In the pair this article deals with, the LiteLLM_SpendLogs table in Postgres is a much better candidate than Langfuse for that function, because it is a transactional write against a relational database, and because it already contains request_id, api_key in hashed form, team_id, organization_id, end_user, model, tokens, startTime, endTime, status and requester_ip_address. Writing the audit event from the same place and in the same transaction as the spend is the sensible route; delegating it to the tracing pipeline is not.

Here an operational tension appears that has to be resolved deliberately. LiteLLM’s performance recommendation for production includes disable_error_logs: true and, in very loaded deployments, disable_spend_logs: true, because provider errors inflate the table and concurrent writes against the same rows produce locks and exhaust the Postgres connections. Disabling spend_logs to gain performance and at the same time leaning on that table for the audit record are incompatible decisions. Above a thousand requests per second or more than ten instances, the way out is use_redis_transaction_buffer: true, which batches the spend updates in Redis before pushing them down to Postgres, watching the litellm_in_memory_spend_update_queue_size and litellm_redis_spend_update_queue_size metrics.

PII: switching off the content before it leaves

An LLM trace contains, by definition, the user’s prompt. In an on-premise deployment for a regulated customer, that content may be exactly what cannot leave the application’s domain, not even towards the observability backend in the same cluster.

The controls on the proxy side, from most global to most fine-grained:

litellm_settings.turn_off_message_logging: true cuts messages and responses in every callback, and keeps the metadata and the spend. The redaction replaces the content with the REDACTED_BY_LITELLM sentinel in messages, choices, reasoning_content, streaming deltas, tool call arguments and in the standard logging object.
The x-litellm-enable-message-redaction: true header enables redaction per request, and LiteLLM-Disable-Message-Redaction: true disables it. The second is checked first and wins over everything else, including the global configuration, which makes it a control to block at the ingress if the policy is that content is never logged.
redact_user_api_key_info: true removes the hashed token, the user_id and the team_id from the log.
Per request and per callback, metadata: {"mask_input": true, "mask_output": true}.
The body parameter "no-log": true disables logging of that call. It is a body parameter, not a header, and it can be disabled globally with global_disable_no_log_param: true so a client cannot make itself invisible.

On the OTel v2 route the policy is inverted towards the safe side: content is off by default and is opened with OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT, with values no_content, span_only, event_only and span_and_event. The documentation underlines that the gate is applied centrally and that a user’s request cannot force their prompt towards the backend while capture is disabled. It is also the stance the OpenTelemetry specification recommends, whose preferred pattern for production is to store the content in external storage and leave only the reference in the span.

One specific version to check in the inventory: below LiteLLM 1.98.0, with per-team Langfuse credentials, the team’s own credentials ended up inside its traces. That version changed the origin of the metadata emitted to the StandardLoggingPayload to cut it off. The same change removed a couple of dozen metadata fields (model_group, deployment, queue_time_seconds, attempted_retries and others), so it breaks saved dashboards and alerts that filter on them. It is an upgrade with work attached, and it has to be done anyway. The same applies to the general hardening of the stack, covered in hardening and secrets of the sovereign stack.

The price in latency

LiteLLM publishes two overhead figures that differ by a factor of thirty, and both are theirs.

The benchmarks page, with Locust, a thousand users with think time and a fake OpenAI endpoint, gives for four instances an x-litellm-overhead-duration-ms of median 2 ms, p95 8 ms and p99 13 ms, at 1,170 requests per second. For two instances, median 12 ms, p95 29 ms and p99 43 ms. The page itself warns that those figures hold with around 130 in-flight requests, and that a closed-loop client with no think time keeps a thousand in flight and sees on the order of eight times that latency at the same rate, by Little’s law.

The AIGatewayBench benchmark, of July 2026, measures the Python proxy at 257.7 ms of added p99 with 329.5 MB of peak memory, against 0.7 ms for the Rust variant in beta. That measurement has no callbacks, no spend tracking and no persistence.

The operational reading is that the first figure is the best case and the second the order of magnitude to plan for agentic load, which is the one that arrives in a closed loop against fast responses. To watch it there are two always-on headers: x-litellm-overhead-duration-ms and x-litellm-callback-duration-ms, and the documentation sets the useful threshold of the second at 100 ms, above which the diagnosis is that the payloads are too large.

Three production settings that appear in LiteLLM’s documentation and that are wrong by default:

request_timeout is 6,000 seconds. A hundred minutes holding a connection against a hung upstream. The recommendation is to lower it to 600.

LITELLM_LOG=DEBUG serialises the payload with json.dumps(indent=4) synchronously, and the documentation puts the cost of that serialisation at 2 to 5 seconds with payloads over 2 MB. It is the first cause of latency their troubleshooting guide mentions.

Background jobs register per uvicorn worker, not per pod. With --num_workers 4 and ten replicas that is forty copies of every periodic job. The LITELLM_JOB_ROLE variable separates the pods that serve traffic from the replica that runs jobs.

And one failure mode with an open issue that deserves to be in the runbook: under sustained 429s from the upstream, pods have been reported unable to answer readiness probes during startup, with the ensuing restart loop. It fits with what was covered in incident response runbooks.

Start-up checklist

Ten checks before signing off the installation of the pair:

callbacks: ["langfuse_otel"] instead of success_callback: ["langfuse"], unless you depend on prompt management, which the OTLP preset does not support.
LANGFUSE_HOST defined explicitly. The preset’s default value points to the US region.
LiteLLM at 1.98.0 or higher, because of the team credential leak into the traces, and at 1.95.0 or higher because of the v4 ingestion header.
A price registered for every self-hosted model, per token and in USD, verified with /spend/calculate before opening traffic.
A test request with a known traceparent, checking that the same identifier appears in Langfuse.
An OpenTelemetry Collector between vLLM and Langfuse, with OTEL_SERVICE_NAME per engine and renaming of the old token attributes.
request_timeout: 600 and LITELLM_LOG out of DEBUG.
Tail sampling in the Collector, with a latency and status code policy, to keep 100 % of what is slow and what failed.
A written decision about prompt content, and blocking of the LiteLLM-Disable-Message-Redaction header at the ingress if the policy is not to log.
A dashboard with x-litellm-overhead-duration-ms, x-litellm-callback-duration-ms and Langfuse’s ingestion queue depth (langfuse.queue.ingestion.depth), which is the autoscaling signal for its workers.

Traps and things that are not what they look like

The trace arrives, but ten minutes late. The x-langfuse-ingestion-version: 4 header is missing, which LiteLLM sends from 1.95.0. It is not a ClickHouse capacity problem.

The metadata works in development and disappears in production. Nginx drops headers with underscores by default, and all of this integration’s have one.

Session cost comes out as zero and generation cost does not. Langfuse adds up only the ingested cost in the session totals, not the one it computes itself.

Enabling OTel v2 breaks nothing visible and ungroups the conversations. The v2 mapper does not read session_id or trace_user_id from the request.

A team’s budget is never exhausted. Its spend is zero because the model has no price, and the calculation failure is only logged at debug level.

The alerts stop firing after the migration to v2. The success status becomes UNSET; no rule on OK is satisfied.

Forwarding the traceparent is enabled and it does not reach the engine. The generic otel callback is missing, without which the function that applies it returns before looking at the flag.

All the engines’ spans appear under unknown_service. vLLM emits no service name; OTEL_SERVICE_NAME has to be set per deployment.

Closing

The gateway and the tracing backend are the two pieces of the inference stack that take the least time to install and the longest to operate well. The installation fits in a ConfigMap. The operation consists of four decisions no getting-started guide raises: which integration route to take knowing that one expires in November, where the price per token of a model nobody invoices comes from, what to do with the traceparent at each hop, and what to tell an auditor about data the system drops by design when things get tight.

The answer to the fourth is the one that orders the other three. This pipeline is operational instrumentation, with statistical precision and best-effort guarantees, and separating it from day one from the regulatory audit record avoids the awkward conversation two years from now. With that separation made, the LiteLLM and Langfuse pair is the pair of pieces that turns a fleet of GPUs into a platform with an owner, a known cost and explainable latency.

Sources

LiteLLM, Langfuse integration (rutas de integración, metadata aceptada, cabeceras langfuse_*): https://docs.litellm.ai/docs/observability/langfuse_integration.
LiteLLM, OpenTelemetry v2 y Migración a OTel v2 (preset, captura de contenido, cambios de semántica): https://docs.litellm.ai/docs/observability/opentelemetry_v2 · https://docs.litellm.ai/docs/observability/opentelemetry_v2_migration.
LiteLLM, Logging y Config settings (redacción, turn_off_message_logging, variables de entorno): https://docs.litellm.ai/docs/proxy/logging · https://docs.litellm.ai/docs/proxy/config_settings.
LiteLLM, Custom pricing y Spend tracking (claves de precio, SpendLogs, endpoints de gasto): https://docs.litellm.ai/docs/proxy/custom_pricing · https://docs.litellm.ai/docs/proxy/cost_tracking.
LiteLLM, Production best practices (topología, request_timeout, buffer de transacciones en Redis, LITELLM_JOB_ROLE): https://docs.litellm.ai/docs/proxy/prod.
LiteLLM, Benchmarks y Latency overhead (cifras de sobrecarga y cabeceras de diagnóstico): https://docs.litellm.ai/docs/benchmarks · https://docs.litellm.ai/docs/troubleshoot/latency_overhead.
LiteLLM, Rust AI Gateway benchmarks (la cifra de 257,7 ms de p99 del proxy Python): https://docs.litellm.ai/blog/rust-ai-gateway-benchmarks.
LiteLLM, notas de versión v1.95.0, v1.98.0 y v1.100.0: https://docs.litellm.ai/release_notes/v1.95.0/v1-95-0 · https://docs.litellm.ai/release_notes/v1.98.0/v1-98-0 · https://docs.litellm.ai/release_notes/v1.100.0/v1-100-0.
Langfuse, OpenTelemetry (endpoint, autenticación, tabla de mapeo de atributos, ausencia de gRPC): https://langfuse.com/integrations/native/opentelemetry.
Langfuse, Compatibility y Migrate v3 to v4 (la fecha del 16 de noviembre de 2026 y los modos de escritura): https://langfuse.com/docs/compatibility · https://langfuse.com/self-hosting/upgrade/upgrade-guides/upgrade-v3-to-v4.
Langfuse, Token and cost tracking (prioridad del coste ingerido sobre el inferido): https://langfuse.com/docs/observability/features/token-and-cost-tracking.
Langfuse, Scaling y Sampling (dimensionado, señal de autoescalado, muestreo de cabecera): https://langfuse.com/self-hosting/configuration/scaling · https://langfuse.com/docs/observability/features/sampling.
Langfuse, issue #13468 — descarte de filas en ClickhouseWriter sin cola de mensajes muertos: https://github.com/langfuse/langfuse/issues/13468. Issue #15109 — coste de sesión a cero: https://github.com/langfuse/langfuse/issues/15109. Issue #5173 — Nginx y cabeceras con guión bajo: https://github.com/langfuse/langfuse/issues/5173.
vLLM, vllm/tracing/utils.py y vllm/tracing/otel.py en la etiqueta v0.28.0 (atributos emitidos, protocolo por defecto, insecure=True): https://github.com/vllm-project/vllm/blob/v0.28.0/vllm/tracing/utils.py.
Langfuse, vLLM integration (la advertencia de que vLLM solo exporta contadores y latencias): https://langfuse.com/integrations/model-providers/vllm.
LiteLLM, litellm/constants.py y litellm/litellm_core_utils/logging_worker.py (capacidades de las colas y política de descarte): https://github.com/BerriAI/litellm/blob/main/litellm/constants.py.
OpenTelemetry, Semantic Conventions for Generative AI y el repositorio semantic-conventions-genai (estado de desarrollo, renombrados, captura de contenido): https://github.com/open-telemetry/semantic-conventions-genai.
OpenTelemetry Collector, tailsamplingprocessor (políticas de muestreo por cola y la restricción de instancia única por traza): https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/tailsamplingprocessor/README.md.
llm-d, End-to-end and fine-grained tracing in llm-d (OTEL_SERVICE_NAME y los spans huérfanos tras Envoy): https://llm-d.ai/blog/end-to-end-and-fine-grained-tracing-in-llm-d.

The blog map: from the utility feed to the token served

Sat, 29 Aug 2026 10:00:00 +0200

This blog started in March 2026 with a handful of articles about Kubernetes and networking, and by now it has piled up more than 150 pieces that, read one at a time, look like unconnected topics: PagedAttention, diesel generators, OpenCost, SPIFFE, immersion cooling, RAGAS. They are not. All of them describe the same object from different altitudes: a sovereign inference factory, that is, the complete machine that turns megawatts from the grid into tokens served to a user, with ownership and control of the data kept in house.

This article is the index of that machine. It orders what has been published into ten layers, bottom up, and links what is written in each one. It is good for three things: finding a specific topic, seeing which layers are covered in depth and which are half done, and choosing a reading route according to the problem in front of you.

The map

The map reads bottom up, and the criterion for ordering it is physical dependency: there is no inference engine without a powered and cooled GPU, no RAG without an engine serving embeddings, no demonstrable regulatory compliance without observability recording what happened. Layer 10 is the exception, and that is why it appears cutting across everything: cost, performance and energy are not a floor of the building, they are the measure of the whole building.

Layer 0 · Physical infrastructure

The most recent vertical, and the one that closes the circle: the path of the watt from the utility feed to the chip, and the path of the heat from the chip to the atmosphere.

The power chain: redundancy, Tier, kW against kVA, PUE, AI density and the limit of the grid.
Generators: NFPA 110, transients, sizing, diesel against gas, fuel and paralleling.
Switching: ATS against STS, the ITIC window, open and closed transition, and the A and B paths.
The UPS and ride-through: IEC 62040-3 topologies, eco-mode, flywheel, lithium against lead and the pulsing loads of AI.
The thermal challenge: why AI density broke air, and the spectrum of solutions.
Air and its limits: CRAC and CRAH, the geometry of the flow, the cube law, containment, ASHRAE and rear doors.
Direct liquid to the chip: cold plate, CDU, TCS and FWS loops, coolant chemistry and leaks.
Immersion and the megawatt horizon: single-phase and two-phase, the PFAS wall and microfluidics in the silicon.

Layer 1 · The node

Inside the machine, before any process exists: the buses, the memory and the topology that decide how much performance actually reaches the model.

PCIe topology, GPUDirect and ACS: the corridors the tensors travel through.
NUMA, hugepages and CPU isolation: why the neighbour on your socket costs you latency.
NVLink, NVSwitch and NCCL: the cable a large model is split across.
Network NUMA with Cilium and DraNet: the kitchen door nobody watches.
CPU, Memory and Topology Manager in RKE2: how to ask for a GPU with affinity and no surprises.
SM, CUDA streams and CUDA graphs: where time is lost inside the accelerator.
Mixed NVIDIA and Intel environments: from the homogeneous cluster to the real estate.
Measuring the power of a GPU: sensor metrology, which belongs to this layer and to layer 10 at the same time.

Layer 2 · Storage

State of the art, performance, security and availability: the set of four on the storage that holds weights, corpus and checkpoints.
From disk to HBM: why starting a model takes as long as it takes.
Speeding up the cold start: from minutes to seconds with serialisation and cache.

Layer 3 · The cluster

vLLM on Kubernetes: the inference piece that does scale.
LLM inference operators: OME, vLLM Production Stack and company.
The GPU cluster as a multi-tenant platform: from shared machine to service.
Sharing a GPU: time-slicing, MPS and MIG: the three ways to split an accelerator.
Volcano and Kueue: gang scheduling, queues and quotas.
Autoscaling with HPA and KEDA: scaling on metrics that mean something.
Serving several models on one GPU: co-residency, swap and sleep.
On-premise capacity planning: from the SLO to the number of nodes.
GitOps for the stack with Flux: operating the platform as code.
Cilium BGP on RKE2 and eBPF from zero to Cilium: the network underneath everything above.

Layer 4 · The inference engine

The densest block of the blog: vLLM cut open, piece by piece.

KV cache and PagedAttention: the working memory and its block manager.
Continuous batching and the scheduler step: how each round of work is assembled.
Optimising prefill and optimising decode: the two phases and their controls.
The attention backend and FlashAttention: the core of the computation.
Prefix cache: hit rate engineering and batch sizing with grid search: the two settings that give away the most performance.
Disaggregated serving and one big replica or many small ones: the topology decisions.
Long context and KV offloading: the KV memory hierarchy when the window does not fit in HBM.
Structured output and function calling: holding the model’s output in place.
Serving reasoning models: the invisible draft you pay for all the same.
Multimodal on-premise with vLLM and serving embeddings and rerankers with TEI: the other models in the stack.
The inverted roofline: why a small model is not optimised the same way as a large one.

Layer 5 · The model

Everything done to the model before serving it, or while it is being served.

Quantization for inference, FP8 end-to-end, aggressive sub-4-bit quantisation and quantising on the fly.
Pruning and distillation: reducing without amputating, and teaching a small model.
MoE in inference and device-native architectures.
Multi-LoRA serving, QLoRA and multi-LoRA at the limit and the complete QLoRA runbook.
Speculative decoding and self-speculative with early exit.
Modern alignment: DPO, KTO, ORPO and SimPO.
Continuous fine-tuning and retrain: closing the loop.

Layer 6 · Data and RAG

Document ingestion from the PDF to the indexed chunk and corpus curation.
Embeddings in 2026 and reranker with hybrid retrieval.
PostgreSQL and Qdrant in ingestion, Debezium and CDC and RAG over Kafka.
Semantic cache: the receptionist with a memory.
RAG on CPU and aggressive RAG on small models.
Evaluating a RAG without fooling yourself: RAGAS and the golden dataset.
Ontologies and knowledge graphs and data versioning with DVC and lakeFS.

Layer 7 · Control plane and delivery

Choosing the OSS gateway and the L7 inference router: the switchboard in front of the engine.
KServe and the Open Inference Protocol: the standard control plane.
Model registry and distribution with OCI and ORAS: where the bytes live.
Canary, blue-green and shadow: deploying a model without breaking the service.
Prompt versioning: the contract that avoids the scare.
MCP from the inside and adding authentication with Keycloak.
Operating the control plane (September 2026 run): the LiteLLM and Langfuse pair, day 2 of the proxy and virtual keys, budgets and limits and humans and agents on the same gateway, the prefix routing LiteLLM does not do and the MCP gateway as a second front door, sizing for agents and the seams with identity and traces: the gateway no longer as a piece you choose, but as a piece you operate.
Identity: Keycloak in an AI platform, with its position in the architecture and the MCP standards it does not implement, and how to complete it for MCP by building the protected resource side.
Operating Langfuse (a run of eight, September 2026): what goes into a trace, the first article about day 2 of version 4.
Building an end-to-end sovereign assistant, with all of the above pieces working together, is the exercise that closes this layer and is pending publication.

Delivery to the teams that use it is its own family of pieces:

Backstage as a self-service portal: the catalogue, the templates and the golden path, with the real cost of operating it.
Kubeflow in depth: which pieces of the ML toolbox are worth it and which ones you already have.
Knative and scale-to-zero: when switching off the GPU saves money and when the cold start violates your SLO.

Rebuilding the cloud experience on your own iron is a run of three:

From public cloud to private: what you lose when you leave the cloud and which projects fill each gap, with the business case and the sovereignty drivers.
Crossplane as a control plane: the declarative provisioning API that on-premise does not give you, and the cost of operating your own control plane.
KubeVela and Score: application-centric self-service by CLI, the friendly face Crossplane deliberately does not offer.

Layer 8 · Observability and evaluation

Tracing with OpenTelemetry GenAI and instrumenting vLLM with OTel.
Langfuse from the inside: the sorting centre for traces.
GPU observability with DCGM and the anatomy of DCGM and vLLM metrics.
Evals: the layer after tracing and LLM-as-judge.
Incident response runbooks: from the alert to the action.
Hubble, AgentSight and drift detection with eBPF: observing without instrumenting.

Layer 9 · Security, trust and compliance

Guardrails and safety and LLM Guard: the lines of defence around the model.
Hardening and secrets of the sovereign stack.
Isolating AI agents from the workspace and the runbook with bubblewrap and Tetragon, with Tetragon as the base.
Signing, provenance and AIBOM and identity and isolation with SPIFFE and Confidential Containers.
ISO/IEC 42001, the EU AI Act article by article and the cross-mapping of ENS, 42001 and the AI Act.
Compliance in healthcare and in defence: the same infrastructure facing two sector frameworks.

Layer 10 · Cost, performance and energy

The data track, written in a different format from the rest of the blog: tables, figures and a verifiable source, no analogies. These are the articles that hold up an investment decision.

FinOps: cost model and state of the art, OpenCost in depth, Kubecost against OpenCost, from the GPU-hour to the cost per token, chargeback and showback, the cost of the idle GPU, cloud GPU and neoclouds, full on-premise TCO and multi-tenancy FinOps with LiteLLM.

Benchmarking: frameworks and metrics, the tool catalogue, GuideLLM and the SLO under load, GenAI-Perf in depth, how to read MLPerf Inference, measurement bias and reproducibility, quality benchmarks and the Pareto frontier between engines.

Energy: measuring GPU power, energy per token in Spain, energy benchmarking frameworks, measuring in production with Kepler and DCGM, MLPerf Power, Wh per token leaderboards, from the watt to carbon, efficiency levers and energy in the TCO and in regulation.

Synthesis: the three axes and the identity that binds them, sovereign on-premise against hyperscalers, with data, from the SLO to the number of GPUs and the reproducible harness.

To that list you can add two cost pieces that are not about GPUs: the cost of agents with durable execution and the gateway’s cost per request.

The master maps

Five earlier articles work as the general plan and are still the best way in for anyone arriving new:

The six-stage LLMOps pipeline: the master map of the life cycle.
The seven layers of the on-premise stack: the architecture version of this same map.
Anatomy of an LLM request in production: a real request crossing every layer.
Five maturity levels of the platform and seven greenfield deployment phases: where you are and what comes next.
The OSS catalogue entry by entry and the parallel catalogue against the hyperscalers: which tool for each stage.
MLOps for LLMs in 2026: the starting landscape.

Four routes

I am about to build the first platform. Landscape and the six-stage pipeline, the seven layers of the stack, the seven deployment phases, capacity planning, vLLM on Kubernetes, gateway, observability with OTel and Langfuse, and guardrails before opening the door.

I have the cluster and the performance is not there. Anatomy of a request, KV cache and PagedAttention, prefill and decode, prefix cache, batch sizing, GuideLLM to find the saturation point, measurement bias so you do not fool yourself with the numbers, and the Pareto frontier between engines.

I have to justify the investment to management. The three axes, on-premise TCO, cost per token, idle GPU, on-premise against hyperscalers, sizing and justifying, and the reproducible harness so the figures survive an audit.

My job is the room, not the software. The power chain, the generators, the thermal challenge, air and its limits, direct liquid to the chip, immersion, and from there to storage and node topology.

What is missing

The map also serves to show the gaps, and three are identified, plus a line that is half open:

A comparison of vector databases: Milvus, Qdrant, pgvector and OpenSearch with data, beyond the scattered mentions.
Lakehouse for RAG: Iceberg and Trino as the substrate of the corpus.
Out-of-band management: OpenBMC, Redfish and DCIM, which is exactly the bridge between layer 0 and layer 3.
Sovereign substrate: the CLI self-service part is already covered by the layer 7 run (Crossplane, KubeVela and Score), and what remains is the IaaS-type substrate underneath, the sovereign alternatives to the hyperscaler (Spinifex, Harvester, OpenStack) on which everything else runs.

Langfuse from the inside: the sorting centre that must not become the bottleneck it came to observe

Sat, 06 Jun 2026 06:00:00 +0200

This post closes a trilogy of the Observe layer: in LLM tracing with OpenTelemetry GenAI the SDK → Collector → backend pipeline was set up and Langfuse was treated as a black box that receives spans; in Prompt versioning with Langfuse and MLflow its prompt management layer was used. Here we open the box: what is inside Langfuse, why v3 stopped being a monolith on Postgres, and how it is operated so it holds the traffic of an inference cluster without becoming the problem.

TL;DR

Langfuse v3 (stable since December 2024) is not an application, it is six services: two containers of its own (Web and Worker) and four stateful dependencies (Postgres, ClickHouse, Redis/Valkey and an S3-compatible blob store). The key architectural change compared with v2, which was a Next.js monolith on Postgres, is the asynchronous ingestion pipeline: traces are received in batches, written immediately to S3, only a reference is queued in Redis, and a Worker ingests them into ClickHouse in the background. This decouples reception speed (limited only by Redis write latency, ~1-5 ms) from the cost of persisting and merging into the analytical database. The result: the Web container sustains hundreds of events per second without a spike blocking the client that serves the inference. But that design only performs with the right settings. This post covers the architecture, its interaction with the rest of the on-premise stack, and ten backend knobs, from batching to ClickHouse to queue sharding, from the FINAL modifier to system log table hygiene, that decide the real throughput and the storage cost. And it marks where the async design hides data loss windows worth knowing about before promising “full traceability”.

You are here: OBSERVE (and the layer that holds up the rest)

The analogy: the postal sorting centre

Picture the central mail sorting office of a large city at rush hour. Lorries loaded with sacks (batches of letters) arrive at a rate that does not stop. If the counter clerk had to open every sack, read every letter, decide its destination and file it before accepting the next lorry, the queue of lorries would go round the block in ten minutes. No serious sorting centre works like that.

What they do is decouple reception from processing:

The reception counter accepts the sack, stamps a receipt on it, leaves it in a pigeonhole in the warehouse and drops a ticket on a conveyor belt. Time per sack: seconds. The counter never blocks.
Further back, in the sorting room, a team of operators picks tickets off the belt, retrieves the sack from its pigeonhole, opens it, sorts the letters and files them in the permanent archive, ordered, indexed, searchable.

Langfuse v3 is exactly this sorting centre:

Postal centre	Langfuse v3	Function
Reception counter	Web container (ingestion endpoint)	Accepts event batches, gives an immediate receipt (HTTP 207)
Pigeonhole warehouse	S3 / Blob store (MinIO on-prem)	Stores the raw sack (the full event)
Ticket on the belt	Redis / Valkey (BullMQ queue)	Only the reference to the object in S3, not the content
Sorting room	Worker container	Takes tickets, reads S3, transforms and files
Indexed permanent archive	ClickHouse (OLAP)	Traces, observations and scores, queryable by project+time
Administrative register	Postgres (OLTP)	Users, projects, API keys, prompts, datasets, config

The thesis of the whole post follows from this analogy: the value of Langfuse lies in the counter never blocking the client that serves the inference. An observability tool that adds latency or outages to the token-serving path is worse than having no observability, because it degrades precisely the system it set out to look after. The whole of the v3 design, and every knob in this post, exists to keep that promise under load.

The mechanism itself: six services, two planes

Langfuse v3 separates two planes that in v2 were fused together:

Ingestion and query plane (the two containers of its own, stateless, horizontally scalable): Web and Worker.
State plane (four dependencies, each with its own load profile): Postgres (transactional OLTP), ClickHouse (analytical OLAP), Redis/Valkey (queue + cache), Blob store (raw objects).

What to take away from this diagram:

Web and Worker are interchangeable and stateless. They store nothing locally. You can run 1 or 20 replicas of each; the state lives in the four dependencies. This is what allows scaling by load without choreography.
Redis never carries the event content, only the reference to the object in S3. That is why Redis holds the spike: a Redis write is ~1-5 ms and moves bytes, not kilobytes. The bottleneck of the Web container is, literally, Redis write speed.
Postgres and ClickHouse have opposite profiles. Postgres is OLTP: many small transactional reads and writes (is this API key valid? which version does the production label point to?). ClickHouse is OLAP: few enormous batch writes and analytical queries over billions of rows (give me the p95 TTFT of project X over the last 7 days). Putting traces in Postgres, which is what v2 did, works until it does not: at production volume, Postgres drowns in a workload it was not designed for. That was the reason for the redesign.

The ingestion flow step by step (and the maths of the decoupling)

The heart of the design is the ingestion route. Seen in detail, a POST /api/public/ingestion request with a batch of events does this:

The mathematical point is the early ACK. The latency the client perceives when sending traces is:

$$ t_{\text{client}} = t_{\text{S3 write}} + t_{\text{Redis enqueue}} \approx 10\text{–}40\,\text{ms} $$

whereas the real cost of persisting, reading S3, transforming, merging against the previous version, inserting into ClickHouse, letting the background merges compact, happens off that path, in the Worker, and can take hundreds of ms or seconds without the client caring. The decoupling turns a system whose throughput would be limited by ClickHouse speed into one limited by Redis speed. And Redis, on modest hardware, sustains on the order of 50,000 operations/second.

This has an important sizing consequence. If your inference load generates $E$ events/second (a chat with RAG + 2 tool calls easily produces 6-10 spans = events per request), the Web container absorbs them while $E \ll 50{,}000$. The Worker, by contrast, scales with the cost of processing: that is the component to watch and replicate, and the first knob of the post.

Honest scepticism. The early ACK has a flip side: between the HTTP 207 and persistence in ClickHouse there is a potential loss window. If the event is in S3 and the reference in Redis, and Redis goes down without persistence (AOF/RDB) before the Worker processes it, the reference is lost; the data is still in S3 but nobody claims it any more. More subtly: the Worker buffers writes to ClickHouse in memory and flushes them in batches; a Worker crash with a full buffer loses that batch. There is a reported bug where the ClickhouseWriter discards rows after exhausting flush retries with no dead-letter queue. For observability this is usually tolerable (losing 0.01 % of the traces breaks nothing). For regulatory auditing, where the trace is evidence, it is not, and Langfuse is better treated as “best-effort” and not as a ledger. We will come back to this at the close.

Interaction with the rest of the stack: Langfuse in the example 4×H100 cluster

Langfuse does not live in isolation. In the seven-layer stack it occupies the LLM-aware observability layer, and it relates to almost every other one. On the generic reference cluster we use throughout the blog, 4×H100 SXM 80 GB (320 GB aggregate VRAM), NVLink, 640 GB of system RAM, NVMe-oF, 25/100 GbE network, the telemetry flow looks like this:

Three ideas from this topology:

Langfuse receives from the OTel Collector, not from the application directly (in the recommended pattern). The app SDK or vLLM emit spans with the gen_ai.* semantic conventions; the Collector does batch, tail-sampling (preserving 100 % of errors and high latencies, sampling the rest) and enriches with its own attributes (tenant_id, priority_tier); and it distributes: LLM traces go to Langfuse, infrastructure spans to Tempo, metrics (GPU DCGM, vLLM metrics) to Prometheus. Langfuse is one more exporter, not the only destination. This is covered in detail in the OTel tracing post.
Langfuse runs off the GPUs. It is a consumer of CPU, RAM, disk and network (ClickHouse wants memory, MinIO wants disk, Redis wants CPU for networking) but it does not touch VRAM. On the 4×H100 cluster, Langfuse lives on a CPU node (or on the GPU nodes but with nodeSelector/taints keeping it away from the vLLM pods). Mixing ClickHouse with vLLM on the same node without resource limits is asking for an ingestion spike to steal memory bandwidth from inference. Isolation by design.
The telemetry route is “cold” and the data route is “hot”. The data plane (left) serves tokens with a millisecond latency budget; the telemetry plane (right) tolerates seconds. The early ingestion ACK is what keeps these two clocks apart: the app does not wait for Langfuse to file anything before returning the response to the user.

The 10 backend knobs that move the needle most

These are, in approximate order of impact/frequency, the settings that decide whether your self-hosted Langfuse ingests 50 events/s or 5,000, and whether your disk grows sustainably or explodes in three weeks. They are all environment variables or config injected into the Web and Worker containers (except the ClickHouse ones, which go in its server-side config). The canonical detail is in the Langfuse scaling doc.

Knob 1 — Scale the Worker by load (the first lever, always)

The Worker is the component that saturates first, because it does the expensive work: reading S3, transforming, merging, inserting into ClickHouse. The Langfuse operational rule is simple: a 2-CPU Worker container above 50 % CPU use is saturated; add replicas. Better than CPU, the Worker publishes via statsd the metric langfuse.queue.ingestion.length (ingestion queue length), which is the direct signal for autoscaling: if the queue grows without draining, there are not enough Workers.

# The ideal autoscaler watches queue depth, not just CPU.
# (KEDA ScaledObject over the statsd metric → Prometheus)
triggers:
 - type: prometheus
 metadata:
 query: langfuse_queue_ingestion_length
 threshold: "10000" # if the queue goes past 10k refs, scale

In AWS deployments there is ENABLE_AWS_CLOUDWATCH_METRIC_PUBLISHING=true to push these metrics to CloudWatch. On-premise, the path is statsd → Prometheus → KEDA, slotting into the autoscaling on Kubernetes with KEDA we already covered for vLLM. Always start here: most “Langfuse is slow” problems are simply insufficient Workers, not fine tuning.

Knob 2 — Separate the ingestion deployment from the UI one

When ingestion is heavily loaded, UI queries and the public API become slow because they share the same Web container. The solution is to split langfuse-web into two identical deployments and route by path: everything under /api/public/ingestion*, /api/public/media* and /api/public/otel* goes to the ingestion deployment; the rest (UI, read API) to the interface one.

# Ingress / gateway rule
location ~ ^/api/public/(ingestion|media|otel) {
proxy_pass http://langfuse-web-ingest; # replicas dedicated to writing
}
location / {
proxy_pass http://langfuse-web-ui; # replicas dedicated to reading
}

It is the same idea as the read/write separation of any system with mixed loads: a storm of writes should not starve whoever is trying to look at the dashboard right in the middle of the incident, which is precisely when they need it most.

Knob 3 — Batching writes to ClickHouse (interval + batch size)

ClickHouse hates small, frequent inserts: every INSERT creates a part on disk that then has to be merged, and thousands of tiny inserts generate thousands of parts and a storm of background merges that saturates the disk. The defence is to accumulate in an in-memory buffer in the Worker and flush in large batches:

# Worker: fewer flushes, larger batches → fewer parts, fewer merges
LANGFUSE_INGESTION_CLICKHOUSE_WRITE_INTERVAL_MS=1000 # raise e.g. to 2000-5000
LANGFUSE_INGESTION_CLICKHOUSE_WRITE_BATCH_SIZE=10000 # raise if there is throughput

Raising the interval and the batch size reduces the flush frequency and improves sustained throughput. The trade-off is direct and has to be understood: larger, less frequent batches mean more data in the Worker’s volatile buffer, that is, a bigger loss window if the Worker goes down (a knob coupled to the scepticism at the close). Langfuse also uses ClickHouse async_insert, which accumulates server-side before confirming; it adds another layer of buffering to keep in mind.

Knob 4 — Skip the ClickHouse read-before-write in ingestion

By default, when ingesting an event the Worker reads the existing event from ClickHouse and merges it with the incoming one (necessary when legacy SDKs send partial events: a start, then an end, then an update of the same observation). That per-event read loads ClickHouse on the write path and limits total throughput.

If your projects are not migrated from an old version, because the full history already lives in S3, you can disable that read:

# Date earlier than the creation of your first project
LANGFUSE_SKIP_INGESTION_CLICKHOUSE_READ_MIN_PROJECT_CREATE_DATE=2025-01-01

With modern Langfuse SDKs or with ingestion via OpenTelemetry, this does not affect you negatively and removes one read per event. A warning from the doc itself: if you combine this with aggressive deletion (lifecycle) rules in S3 plus late event updates, you can generate duplicates in the history. Know that before enabling it.

Knob 5 — Write concurrency to S3/Blob storage

In high-throughput scenarios, the S3 client can exhaust its sockets and start queueing and throttling writes. The symptom is unmistakable in the logs of the Web container that handles ingestion:

@smithy/node-http-handler:WARN - socket usage at capacity=150
and 387 additional requests are enqueued.

…accompanied by a rise in memory in that container (queued requests pile up in RAM). The cure is to raise the concurrent write limit from its default of 50:

LANGFUSE_S3_CONCURRENT_WRITES=100 # raise gradually from 50

Each additional socket has a small memory cost, so the official advice is to raise it gradually while observing the behaviour, not to jump to 1000 in one go.

Knob 6 — Redis queue sharding + concurrency per shard

If Redis goes past 90 % CPU, first the obvious: an instance with at least 4 CPUs (so Redis spreads networking and background tasks across different cores) and Redis Cluster mode enabled. If CPU still does not come down, you can shard the queues Langfuse uses:

# Advanced: only if Redis is genuinely drowning and you already did the above
LANGFUSE_INGESTION_QUEUE_SHARD_COUNT=6 # ~2-3× the number of Redis cluster shards
LANGFUSE_TRACE_UPSERT_QUEUE_SHARD_COUNT=6
# Concurrency counts PER SHARD; target ~20 per worker
LANGFUSE_INGESTION_QUEUE_PROCESSING_CONCURRENCY=3 # 6 shards × ~3 ≈ 18
LANGFUSE_TRACE_UPSERT_WORKER_CONCURRENCY=3

Two traps the doc underlines and that are worth tattooing: once you shard, do not reduce the number of shards (it breaks the distribution); and concurrency counts per shard, not globally, so if you have 10 shards and want concurrency 20 per worker, set 2, not 20. It is an advanced knob: most on-premise deployments never need it.

Knob 7 — The `FINAL` modifier for OTel-only projects

Langfuse stores observations in a ClickHouse ReplacingMergeTree and, by default, adds the FINAL modifier to API queries so the last version of each row wins at read time. FINAL is necessary when ingestion produces several versions of the same observation (legacy SDKs with their start/end/update events), but it adds merge work to every read and slows it down.

Projects that ingest exclusively over OpenTelemetry write each observation as a single immutable row, so FINAL is redundant for them:

# Recommended in mixed deployments: per-project, flagged in Redis with a 24h TTL
LANGFUSE_SKIP_FINAL_FOR_OTEL_PROJECTS=true

# Only if ALL projects are OTel-only: global, no Redis lookup
LANGFUSE_API_CLICKHOUSE_DISABLE_OBSERVATIONS_FINAL=true

Since on the example cluster the instrumentation is 100 % OTel (gen_ai.* via Collector), this knob is free money in dashboard read latency. Careful with the global version: do not enable it if some project is still using legacy ingestion, or reads may return duplicate or stale rows.

Knob 8 — Separate analytical reads from the write path (compute-compute)

Heavy dashboard queries (percentiles over millions of spans) compete with ingestion inserts and with background merges on the same ClickHouse. If your deployment supports compute-compute separation (ClickHouse Cloud or BYOC), you can route reads to a read-only compute group:

CLICKHOUSE_URL=http://clickhouse-primary:8123 # writes, migrations, ingestion
CLICKHOUSE_READ_ONLY_URL=http://clickhouse-reader:8123 # UI + public API reads

Critical nuance for on-premise, and here it is time to be sceptical about the usefulness of this knob in our context: on a single-node ClickHouse or on a self-managed cluster without compute separation, this variable adds nothing, because the read endpoint would be the same as the write one. It is a knob for cloud architectures with storage separated from compute. On an on-premise 4×H100 cluster with ClickHouse on one node, the real alternative is to scale ClickHouse vertically (the doc recommends ≥16 GiB of RAM for large deployments; ClickHouse scales vertically well) and to make sure every query filters by projectId and time, which is how the tables are indexed. Without a time filter, even the fattest ClickHouse suffers.

Knob 9 — Data retention: TTL in ClickHouse + lifecycle in S3

Disk is the cost that grows on its own. LLM traces carry whole inputs and outputs (sometimes prompts of tens of KB), and ClickHouse also accumulates its own system tables. The first-order lever is a retention policy that nightly deletes traces, observations, scores and media older than N days, coordinating ClickHouse and blob storage. Where the retention feature is not available, it is done by hand:

-- ClickHouse: TTL over the tracing tables
ALTER TABLE traces MODIFY TTL toDateTime(timestamp) + INTERVAL 90 DAY;
ALTER TABLE observations MODIFY TTL toDateTime(start_time) + INTERVAL 90 DAY;
ALTER TABLE scores MODIFY TTL toDateTime(timestamp) + INTERVAL 90 DAY;
ALTER TABLE event_log MODIFY TTL toDateTime(timestamp) + INTERVAL 30 DAY;

# S3/MinIO: lifecycle rule, e.g. 30 days for the raw event bucket
# WATCH OUT! Do NOT apply retention to the MEDIA bucket:
# - it breaks the files referenced in traces
# - it breaks future uploads (state is tracked by hash in Postgres)

Two operational parameters that avoid nasty surprises on large deletions:

LANGFUSE_CLICKHOUSE_DELETION_TIMEOUT_MS=600000 # default 10 min; raise it if deletions time out
# ClickHouse 25.7+: less mutation pressure on bulk deletions
CLICKHOUSE_LIGHTWEIGHT_DELETE_MODE=lightweight_update
CLICKHOUSE_USE_LIGHTWEIGHT_UPDATE=true

The mental rule: short retention for raw events (S3, 30 days is usually enough, they are recoverable/recomputable), retention by business value for the ClickHouse tables (90 days, 180, whatever compliance asks for), and never touch the media bucket with blind lifecycle rules.

Knob 10 — Hygiene of the ClickHouse system log tables (the silent disk killer)

This is the knob nobody configures and that fills the disk without showing up in any Langfuse metric, because it is not Langfuse data: they are the system tables of ClickHouse itself (trace_log, text_log, opentelemetry_span_log, asynchronous_metric_log, metric_log, latency_log). By default they have no TTL, and the query profiler writes to system.trace_log continuously. On a ClickHouse with traffic, these tables can dominate disk usage while you look for the problem in your traces. Langfuse does not read from them, so they can be trimmed without fear. Two options:

<!-- Option A — disable the ones Langfuse never reads
 (file in /etc/clickhouse-server/config.d/) -->
<clickhouse>
 <trace_log remove="1"/>
 <text_log remove="1"/>
 <opentelemetry_span_log remove="1"/>
 <asynchronous_metric_log remove="1"/>
 <metric_log remove="1"/>
 <latency_log remove="1"/>
</clickhouse>
<!-- Keep query_log, part_log and error_log: useful for debugging and small -->

-- Option B — aggressive TTL + turn off the profiler, if you want to keep them for debugging
-- (in config: query_profiler_real_time_period_ns = 0)
SET max_table_size_to_drop = 0;
TRUNCATE TABLE system.trace_log;
ALTER TABLE system.trace_log MODIFY TTL event_date + INTERVAL 7 DAY;
-- repeat for each log table to cap

To identify which table is eating the disk, the golden query:

SELECT table, formatReadableSize(sum(bytes)) AS size, sum(rows) AS rows
FROM system.parts WHERE active GROUP BY table ORDER BY sum(bytes) DESC;

If you take only one knob from this post to your first real deployment, make it this one: the difference between a ClickHouse that grows 2 GB/day of useful data and one that grows 20 GB/day of system logs nobody looks at.

Summary table of the 10 knobs

#	Knob	Variable / action	When
1	Scale Worker	replicas by CPU>50 % / `langfuse.queue.ingestion.length`	always, first
2	Separate ingestion/UI	route `/ingestion`,`/media`,`/otel*` to a dedicated replica	slow UI under load
3	Batching to ClickHouse	`LANGFUSE_INGESTION_CLICKHOUSE_WRITE_INTERVAL_MS` / `_BATCH_SIZE`	high throughput
4	Skip CH read-before-write	`LANGFUSE_SKIP_INGESTION_CLICKHOUSE_READ_MIN_PROJECT_CREATE_DATE`	non-migrated projects
5	S3 concurrency	`LANGFUSE_S3_CONCURRENT_WRITES` (def. 50)	“socket usage at capacity”
6	Redis queue sharding	`LANGFUSE__QUEUE_SHARD_COUNT` + `_CONCURRENCY` (per shard)	Redis CPU >90 %
7	Remove `FINAL` (OTel)	`LANGFUSE_SKIP_FINAL_FOR_OTEL_PROJECTS=true`	100 % OTel instrumentation
8	CH read/write split	`CLICKHOUSE_READ_ONLY_URL` (cloud/BYOC only)	compute-compute available
9	Retention + TTL	TTL in CH + S3 lifecycle + `LANGFUSE_CLICKHOUSE_DELETION_TIMEOUT_MS`	always (disk cost)
10	CH system log hygiene	`<trace_log remove="1"/>` or aggressive TTL	always (hidden disk)

How to maximise Langfuse on the example 4×H100 cluster

With the architecture and the knobs clear, this is a concrete sizing to get the most out of Langfuse on the generic reference cluster (4×H100 SXM, 320 GB VRAM, 640 GB RAM, NVMe-oF, 25/100 GbE), without stealing a single GB of VRAM from inference.

Component distribution

Langfuse is 100 % CPU/RAM/disk/network load, so its natural place is off the GPU nodes or, if it cohabits, with taints/nodeSelector confining it away from the vLLM pods. Suggested distribution:

nodo-cpu-01 (control + observability, no GPU)
├── langfuse-web-ingest ×3 (2 CPU / 4 GiB each) ← ingestion, scales with load
├── langfuse-web-ui ×2 (2 CPU / 4 GiB each) ← dashboard/read API
├── langfuse-worker ×4 (2 CPU / 4 GiB each) ← the one that scales most
├── redis/valkey ×1 (4 CPU / 4 GiB, cluster mode)
└── postgres ×1 (2 CPU / 8 GiB, replica for HA)
nodo-storage-01 (heavy state, local NVMe)
├── clickhouse ×1 (8 CPU / 32 GiB / NVMe) ← ≥16 GiB is the minimum; 32 is comfortable
└── minio (S3) ×1 (4 CPU / 8 GiB / HDD+NVMe cache)
nodo-gpu-01..02 (4×H100 SXM each) → inference ONLY
└── vLLM, embeddings, reranker, guardrails (emit spans, do not host Langfuse)

Sizing by real load

Let us put numbers on an example load. Suppose the cluster serving 300 requests/second of chat-with-RAG, where each request generates on the order of 8 spans (request, retrieval, rerank, 2× tool, guardrail in, llm, guardrail out):

$$ E = 300\,\tfrac{\text{req}}{\text{s}} \times 8\,\tfrac{\text{spans}}{\text{req}} = 2,400\ \text{events/s} $$

Against the Redis ceiling (~50,000 ops/s), $E = 2{,}400$ leaves the reception counter at ~5 % of its capacity: enormous headroom. The component to watch is the Worker. With a target of ~20 concurrency per Worker and batches of 10,000 events every ~1-2 s, 4 Workers drain 2,400 ev/s with margin; the langfuse.queue.ingestion.length metric should stay flat near zero. If it grows, knob 1 (more Workers) is the answer before any fine tuning.

Tail-sampling is the multiplier that changes the economics. If the Collector preserves 100 % of errors/high latencies but samples normal traffic at, say, 10 %, the 2,400 ev/s you store in ClickHouse drop to ~240-300 effective ev/s without losing the signal that matters. The rule: sample in the Collector, not in Langfuse. Langfuse should receive, already filtered, what deserves to be persisted. This is developed in the OTel tracing post; here it is enough to note that upstream sampling is, in effect, the knob 0 that multiplies all the others.

Storage estimate

An LLM observation with full input+output weighs, compressed in ClickHouse, on the order of 1-3 KB (ClickHouse compresses text very well, 5-10×). With sampling at 10 % over 2,400 ev/s:

$$ 240\,\tfrac{\text{ev}}{\text{s}} \times 2\,\text{KB} \times 86,400\,\tfrac{\text{s}}{\text{day}} \approx 41\ \text{GB/day (raw)} \;\xrightarrow{\text{compression}}\; \sim 5\text{–}8\ \text{GB/day in CH} $$

At 90 days of retention (knob 9), the permanent archive settles at around 500-700 GB in ClickHouse, comfortable on the NVMe of the storage node, plus the raw events in MinIO with a 30-day lifecycle. Without the system log hygiene (knob 10), add easily as much again in rubbish nobody queries. The two disk knobs together are the difference between planning storage once a year and fighting a full disk every month.

“Maximum use” checklist

Sampling in the Collector (tail: 100 % errors + N % normal), before touching anything in Langfuse.
Workers scaled by queue length via KEDA (knob 1), not fixed.
Ingestion separated from UI (knob 2) so the dashboard responds during incidents.
SKIP_FINAL_FOR_OTEL_PROJECTS enabled (knob 7) because the instrumentation is 100 % OTel.
Generous CH batching (knob 3) tuned to the throughput, accepting the loss window.
Retention + TTL + system log hygiene (knobs 9 and 10) configured on day 1, not when the disk screams.
ClickHouse with ≥16 GiB and every query filtering by projectId+time (knob 8 in its on-premise version: vertical scaling).
Langfuse isolated from the GPUs by taints/nodeSelector: not a MB of VRAM, no memory bandwidth contention with vLLM.

Traps and things that are not what they look like

“Langfuse guarantees me full traceability.” No: the design is high-performance best-effort, not a ledger. Between the HTTP 207 and the row in ClickHouse there are volatile buffers (Redis without hard persistence, the Worker’s in-memory buffer, ClickHouse’s server-side async_insert). There is a known bug where the writer discards rows with no dead-letter queue after exhausting retries. For operational observability, losing 0.01 % of spans is irrelevant. For ENS/EU AI Act audit evidence, where the trace is the proof, Langfuse should not be the only record; the regulatory audit log needs durability guarantees this pipeline does not promise. A distinction covered in the technical controls for ENS/42001/EU AI Act.

Raising ClickHouse batching “to go faster” and nothing else. Knob 3 improves throughput at the cost of enlarging the loss window and the latency of data appearing in the dashboard. Batches of 50,000 every 10 s perform wonderfully… until the Worker restarts with 50,000 events in the buffer. Tune with awareness of the trade-off, not by blind maximisation.

Putting ClickHouse on the same node as vLLM with no limits. ClickHouse is voracious with memory bandwidth during merges. Sharing a node with vLLM without resources.limits or NUMA isolation means an ingestion spike can degrade inference TTFT, exactly the original sin this whole architecture wanted to avoid. Isolate.

Forgetting the time filter in your own queries. The ClickHouse tables are indexed by projectId and time. A custom dashboard or an API query without a time filter scans the whole history and takes performance down for everyone. It is not Langfuse “being slow”: it is a badly written query.

Applying lifecycle to the media bucket. It breaks the files referenced in traces and blocks future uploads (state is tracked by hash in Postgres). The media bucket is managed only with the Langfuse retention feature, never with blind S3 rules.

Treating queue sharding as a routine optimisation. It is an advanced knob for a genuinely drowning Redis, irreversible (do not reduce shards) and with per-shard concurrency semantics that are easy to misread. In the vast majority of on-premise deployments it is not needed; if you enable it “just in case”, you complicate your life without gaining anything.

Conclusion

Langfuse v3 solved the structural problem of LLM observability, that the observer should not suffocate the observed, by moving from a monolith on Postgres to a six-service sorting centre with asynchronous ingestion. That design is what allows a cluster serving thousands of tokens per second to be instrumented end to end without the app ever waiting for a trace to be filed. But the design is a necessary condition, not a sufficient one: it performs if the right levers are tuned. Of the ten knobs, three decide almost everything in a typical on-premise deployment, scaling Workers by queue length (1), retention + TTL (9), and system log hygiene (10); the rest are refinements that show up when the load bites. And above them all lives knob 0, which is not a Langfuse one: sampling in the Collector, which decides how much reaches the pipeline before any internal setting matters. Maximising Langfuse on the 4×H100 cluster is not about squeezing its peak throughput: it is about putting it off the GPUs, feeding it already-sampled traffic, sizing the Worker by the queue, and configuring retention on day one, so that the tool that came to tell the story does not end up being the chapter about the incident.

References

Langfuse, Scaling Langfuse Deployments (doc oficial de sizing y todos los env vars de este post): https://langfuse.com/self-hosting/configuration/scaling.
Langfuse, Self-host Langfuse y Configuration via Environment Variables: https://langfuse.com/self-hosting · https://langfuse.com/self-hosting/configuration.
Langfuse, ClickHouse (self-hosted): https://langfuse.com/self-hosting/deployment/infrastructure/clickhouse.
Langfuse, From Zero to Scale: Langfuse’s Infrastructure Evolution (el porqué del rediseño v2→v3): https://langfuse.com/blog/2024-12-langfuse-v3-infrastructure-evolution.
Langfuse, Migrate v2 to v3 (self-hosted): https://langfuse.com/self-hosting/upgrade/upgrade-guides/upgrade-v2-to-v3.
ClickHouse, Langfuse and ClickHouse: A new data stack for modern LLM applications: https://clickhouse.com/blog/langfuse-and-clickhouse-a-new-data-stack-for-modern-llm-applications.
Langfuse, issue #13468 — ClickhouseWriter drops rows after max flush attempts with no DLQ (la ventana de pérdida documentada): https://github.com/langfuse/langfuse/issues/13468.
ClickHouse, TTL for tables and columns: https://clickhouse.com/docs/guides/developer/ttl.
OpenTelemetry, Semantic Conventions for Generative AI (gen_ai.*): https://opentelemetry.io/docs/specs/semconv/gen-ai/.

Debezium and CDC: the notary who hears changes before anyone asks

Thu, 04 Jun 2026 11:00:00 +0200

TL;DR

Change Data Capture (CDC) with Debezium listens to the PostgreSQL Write-Ahead Log and turns every INSERT, UPDATE and DELETE into a structured Kafka event. Unlike traditional polling (SELECT ... WHERE updated_at > ?), it detects deletes, has latency in the tens of milliseconds and adds no extra load to the database. In RAG pipelines, this means that when a document is deleted from Postgres, the Qdrant chunks disappear too, automatically and in real time. The supporting infrastructure is modest: the connector uses 2-4 cores and 4-8 GB RAM to process thousands of events per second.

The master analogy: the land registry notary

Picture the land registry. Every time a flat is sold, mortgaged, or a mortgage is cancelled, the registrar records the operation in the registry ledger, a chronological and immutable journal. If you want to know what has changed in the registry, you have two options.

Option A (polling): you send someone every 5 minutes with a list of properties to ask “has anything changed?”. Problems: if a title was cancelled (DELETE), the property no longer exists by the time your envoy arrives, and there is no trace of it. If 20 different departments are doing the same thing, that is 20 people pestering the registrar every 5 minutes. And the minimum latency is the interval: 5 minutes.

Option B (Debezium): you hire a notary who sits right at the registrar’s desk. Every time the registrar signs an operation into the ledger, the notary notes it down immediately and notifies whoever needs to know. Title cancellations included, because the notary sees them as clearly as any other operation: he was there when it was signed.

In this analogy:

The registry ledger is the PostgreSQL WAL (Write-Ahead Log).
The notary is the Debezium connector.
The notary’s bookmark, which guarantees he loses no page even if he steps out for a moment, is the logical replication slot.
The messenger who carries the notifications to interested parties is Kafka (or Redpanda, or NATS JetStream).

We will pick this thread up in every section. When a technical detail is unclear, go back to the image of the notary.

1. The problem CDC solves

The most common synchronisation pattern between services sharing PostgreSQL is periodic polling:

SELECT id, content, updated_at
FROM documents
WHERE updated_at > $1
ORDER BY updated_at
LIMIT 1000;

This pattern has three structural problems.

DELETEs are invisible. When you delete a row, updated_at is not updated: the row is gone. The next time the poller queries, the row does not exist and there is no way to know it ever did. In a RAG pipeline, this means orphan chunks in Qdrant: the document no longer exists in Postgres, but its vectors keep polluting search results.

The minimum latency is the interval. If the poller runs every 5 seconds, average latency is 2.5 seconds. For near-real-time synchronisation (dashboards, alerts, RAG with frequently changing documents), that is too much.

Load scales with the number of consumers. If 10 services poll the same table every 5 seconds, that is 10 × 12 = 120 queries/minute producing no useful work; they only check whether there is anything new. On large tables with complex indexes, that is real load on the database.

CDC inverts the model: the database notifies, the consumers listen. Zero polling, zero extra load, DELETEs included, latency in the tens of milliseconds.

2. What the PostgreSQL WAL is

The operations journal

The Write-Ahead Log (WAL) is the chronological, immutable record of every operation Postgres performs. Before modifying any data page on disk, Postgres writes the operation to the WAL. That sequence, log first then data, is what guarantees durability (the D in ACID) and enables crash recovery: if Postgres dies mid-transaction, on restart it replays the WAL to return the database to a consistent state.

The WAL is the registry ledger of our analogy: chronological, immutable, complete.

Physical vs logical replication

PostgreSQL supports two WAL-based replication modes:

Physical replication: it replicates disk blocks verbatim. The standby receives the same bytes as the primary. It serves high availability and failover, but the target must be an exact copy of Postgres; you cannot send the changes to an external application.
Logical replication: instead of disk blocks, it replicates semantic operations: “the row with id=42 was inserted into table documents with these values”. The target can be anything that understands the protocol: another Postgres, Debezium, or any custom consumer.

CDC uses logical replication. It is what lets Debezium understand “what changed and in which table” rather than “which disk block changed at which offset”.

The replication slot: the notary’s bookmark

A logical replication slot is a persistent cursor in the WAL. Postgres keeps a record of how far into the WAL each slot has consumed. While a slot exists, Postgres guarantees it will not discard the WAL segments the slot has not yet read.

This is exactly the notary’s bookmark: even if the notary goes out for lunch, the ledger stays open at the last page he read. When he comes back, he carries on from exactly where he left off, having lost nothing.

The risk is the inverse: if the notary never comes back, the bookmark stops the registrar from archiving the old pages. If the Debezium connector goes down and does not recover for hours, the WAL grows indefinitely on disk until the slot is deleted manually or the consumer starts consuming again. This is called WAL disk blowup and is the most important operational risk of Debezium.

Mandatory monitoring:

SELECT slot_name,
 confirmed_flush_lsn,
 pg_current_wal_lsn(),
 pg_wal_lsn_diff(pg_current_wal_lsn(), confirmed_flush_lsn) AS lag_bytes
FROM pg_replication_slots
WHERE slot_type = 'logical';

The pgoutput plugin

The WAL stores operations in an internal binary format. For Debezium to understand them, Postgres needs to decode them into a readable format. The pgoutput decoding plugin, included in the Postgres core since version 10, does exactly that: it translates the binary WAL events into messages with the before/after structure of each row.

Debezium uses pgoutput by default. It requires no external extensions (unlike the wal2json plugin that was popular before Postgres 10).

3. Debezium architecture

The connector as a Kafka Connect plugin

Debezium is not a standalone service; it is a plugin for the Kafka Connect framework. Kafka Connect manages the connector’s lifecycle (start, stop, reconnection, offset tracking) and provides the parallelism and fault tolerance infrastructure.

The connector talks to Postgres through the logical replication protocol (not over JDBC), using the credentials of a user with the REPLICATION role.

PostgreSQL (WAL + pgoutput)
│
│ logical replication protocol
▼
Debezium Connector (Kafka Connect worker)
│
│ Kafka Producer API
▼
Kafka topic: rag.public.documents
│
▼
Consumer (Qdrant sync, audit log, fine-tuning pipeline...)

Structure of a Debezium event

Each change in the table becomes a JSON message with this structure:

INSERT ("op": "c" — create):

{
 "before": null,
 "after": {
 "id": 42,
 "content": "Contrato de arrendamiento...",
 "tenant_id": "acme",
 "updated_at": 1748934000000
 },
 "op": "c",
 "source": {
 "version": "2.7.0.Final",
 "connector": "postgresql",
 "db": "rag_db",
 "schema": "public",
 "table": "documents",
 "lsn": 29823948,
 "txId": 1047,
 "ts_ms": 1748934000123
 }
}

DELETE ("op": "d"):

{
 "before": {
 "id": 42,
 "content": "Contrato de arrendamiento...",
 "tenant_id": "acme",
 "updated_at": 1748934000000
 },
 "after": null,
 "op": "d",
 "source": { "lsn": 29824102, "txId": 1051, "ts_ms": 1748934060200 }
}

The before field holds the row’s previous state, available because Postgres can be configured with REPLICA IDENTITY FULL to include the complete row in the WAL on delete or update. Without that setting, before contains only the primary key.

This is the key for the RAG pipeline: the DELETE event carries the document id. The consumer uses it to delete every associated chunk in Qdrant with a doc_id = 42 filter. Without CDC, those chunks would never have been deleted.

Initial snapshot

When the connector starts for the first time (or after a reset), it cannot begin consuming the WAL from “the beginning of time”, only from the moment the slot was created. How does it guarantee the consistency of the initial state?

Through a transactional snapshot: the connector opens a transaction in REPEATABLE READ mode, exports the snapshot ID (pg_export_snapshot()), and does a full SELECT of the configured tables inside that transaction. It then starts consuming the WAL from the snapshot’s LSN. That way there is no gap: the snapshot covers the state up to an instant, and the WAL covers from that instant onwards.

SMT transformations (Single Message Transforms)

Before emitting the event to the Kafka topic, the connector can apply inline transformations called SMT. Common use cases:

Filtering sensitive columns (ReplaceField with blacklist): strip password_hash, phone_number before they reach the topic.
Adding metadata (InsertField): enrich the event with a tenant_id extracted from the original HTTP header (if it is in the row).
Conditional routing (Filter): drop events from rows with status = 'draft' before emitting them.

SMTs are pure configuration, requiring no code, and they are applied inside the connector process with no perceptible extra latency.

4. Debezium vs the Outbox pattern

The Outbox pattern is the most common alternative to pure CDC. Instead of emitting events directly to Kafka, the application writes to an outbox table in Postgres within the same transaction that modifies the data. A separate worker reads that table and publishes the events.

Criterion	Debezium (pure CDC)	Outbox pattern
Event latency	~50-200 ms from commit	Depends on the worker interval (typically 1-5 s)
Consistency	At-least-once	At-least-once
DELETE detection	Native (the DELETE event includes `before`)	Only if the app writes to the outbox on delete
Setup complexity	High (Kafka Connect, replication slot, permissions)	Low (an extra table plus a simple worker)
Infrastructure dependency	Requires Kafka/Redpanda/NATS JetStream	Postgres plus a worker; Kafka optional
WAL disk blowup risk	Yes, if the slot stops consuming	No
Schema visibility	Reads the table’s real schema	The event schema is defined by the app
Schema migration	Needs care (events reflect DDL changes)	More flexible (the event is whatever the app puts there)
When to use it	When you need DELETEs, low latency, or you cannot modify the app	When the app controls the event domain and infrastructure is limited

Rule of thumb: if you control the application code and do not need native DELETEs, the Outbox is simpler. If you do not control the code (legacy database, third-party application) or DELETEs are critical (a RAG pipeline with document deletion), Debezium is the right choice.

5. The maths

Throughput

Debezium on a connector with 4 workers can process between 10,000 and 50,000 events/second on modest hardware (4 cores, 8 GB RAM). The real bottleneck is not the connector but the Kafka broker: with 3 brokers and suitable partitions, Kafka can comfortably sustain 500,000 messages/second with 1 KB messages (source: public Confluent benchmarks, 2023).

For a typical RAG pipeline with 100 documents modified per minute:

$$\text{events/s} = \frac{100}{60} \approx 1{.}7 \text{ events/s}$$

That is 0.0034% of the connector’s capacity. Debezium will not be the bottleneck in any realistic RAG scenario.

End-to-end latency

The path from a commit in Postgres to an upsert in Qdrant has these stages:

Stage	Typical latency
Commit in Postgres → WAL written	< 1 ms (synchronous with the commit)
WAL written → Debezium reads it (WAL lag)	10-50 ms
Debezium → Kafka produce (ack)	5-20 ms
Kafka → Consumer (poll interval)	0-100 ms (configurable)
Consumer → Qdrant upsert/delete	5-15 ms
Typical total	30-200 ms

With fetch.min.bytes=1 and fetch.max.wait.ms=10 on the consumer, the Kafka poll latency drops to ~10 ms. The realistic range for an optimised pipeline is 30-100 ms.

WAL disk blowup risk

If the connector stops consuming, Postgres retains the WAL from the slot’s confirmed_flush_lsn onwards. The retained volume grows linearly with time and the write rate:

$$\text{retained WAL} = \text{write rate} \times \text{average WAL event size} \times \text{time without consuming}$$

Example with a moderate load (50,000 writes/hour, 500 bytes average per WAL event):

$$50{,}000 \times 500 \text{ B} \times 1 \text{ h} = 25 \text{ MB/h}$$

With a high load (1,000,000 writes/hour):

$$1{,}000{,}000 \times 500 \text{ B} \times 1 \text{ h} = 500 \text{ MB/h}$$

If the connector is down for 48 hours under high load: 24 GB of retained WAL. That can fill the disk and block Postgres completely.

Recommended alert: set an alert when lag_bytes > 1 GB or when confirmed_flush_lsn has not advanced for more than 15 minutes. See the monitoring query in section 2.

6. Use cases in LLMOps / RAG

RAG synchronisation with real deletion

This is the use case that most clearly justifies Debezium over polling. The flow:

A user deletes document id=42 from the document management interface.
Postgres runs DELETE FROM documents WHERE id = 42.
Debezium detects the DELETE in the WAL and emits the event with "op": "d" and "before": {"id": 42, ...}.

The consumer receives the event and runs:

qdrant_client.delete(
 collection_name="documents",
 points_selector=Filter(must=[FieldCondition(key="doc_id", match=MatchValue(value=42))])
)

Every chunk with doc_id=42 disappears from Qdrant in ~100 ms.

Without Debezium, those chunks would stay indefinitely, polluting retrieval results with fragments of documents that no longer exist in the source of truth.

Event sourcing for fine-tuning datasets

Every time a human annotator updates a row in the annotations table (correcting an LLM output), Debezium emits the UPDATE with before and after. The consumer writes the pair (original_output, correction) into the dataset curation pipeline, with no need for the annotator to do anything beyond saving in the interface. The fine-tuning pipeline knows exactly what changed and when, with no polling and no risk of duplicates from overlapping time windows.

Immutable audit log

WAL events are, by definition, the most faithful record of what happened in the database: they are the same data Postgres uses for crash recovery. Kafka with long retention (90 days, or size-based retention) serves as an immutable audit log without modifying the application schema or adding triggers. This is particularly useful in regulated environments where traceability of data modifications is required.

7. Architecture diagram

8. Minimum configuration

PostgreSQL: enable logical replication

-- Requires restarting Postgres
ALTER SYSTEM SET wal_level = logical;
ALTER SYSTEM SET max_replication_slots = 10;
ALTER SYSTEM SET max_wal_senders = 10;

-- Reload configuration (wal_level requires a full restart)
SELECT pg_reload_conf();

-- Dedicated user for Debezium
CREATE USER debezium WITH REPLICATION LOGIN PASSWORD 'cambiar_esto';
GRANT SELECT ON TABLE public.documents TO debezium;

-- REPLICA IDENTITY FULL to get a complete 'before' on DELETEs and UPDATEs
ALTER TABLE public.documents REPLICA IDENTITY FULL;

Debezium connector (Kafka Connect REST API)

{
 "name": "postgres-debezium",
 "config": {
 "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
 "database.hostname": "postgres",
 "database.port": "5432",
 "database.user": "debezium",
 "database.password": "cambiar_esto",
 "database.dbname": "rag_db",
 "topic.prefix": "rag",
 "table.include.list": "public.documents",
 "plugin.name": "pgoutput",
 "slot.name": "debezium_rag",
 "publication.autocreate.mode": "filtered",
 "snapshot.mode": "initial",
 "tombstones.on.delete": "true",
 "transforms": "unwrap",
 "transforms.unwrap.type": "io.debezium.transforms.ExtractNewRecordState",
 "transforms.unwrap.drop.tombstones": "false",
 "transforms.unwrap.delete.handling.mode": "rewrite"
 }
}

curl -X POST http://kafka-connect:8083/connectors \
 -H 'Content-Type: application/json' \
 -d @connector-config.json

Check its status:

curl http://kafka-connect:8083/connectors/postgres-debezium/status

Minimal consumer in Python

from confluent_kafka import Consumer
from qdrant_client import QdrantClient
from qdrant_client.models import Filter, FieldCondition, MatchValue
import json

consumer = Consumer({
 "bootstrap.servers": "kafka:9092",
 "group.id": "qdrant-sync",
 "auto.offset.reset": "earliest",
 "enable.auto.commit": False,
})
consumer.subscribe(["rag.public.documents"])
qdrant = QdrantClient("qdrant", port=6333)

while True:
 msg = consumer.poll(timeout=0.1)
 if msg is None:
 continue
 event = json.loads(msg.value())
 op = event.get("op")
 if op in ("c", "u"): # INSERT or UPDATE
 doc = event["after"]
 # ... vectorise and upsert into Qdrant
 elif op == "d": # DELETE
 doc_id = event["before"]["id"]
 qdrant.delete(
 collection_name="documents",
 points_selector=Filter(
 must=[FieldCondition(key="doc_id", match=MatchValue(value=doc_id))]
 )
 )
 consumer.commit()

9. On-premise deployment

The Debezium stack does not compete for GPU. On a node with 4×H100 SXM (320 GB, NVLink) serving the inference LLM, the CDC pipeline runs entirely on general-purpose (CPU-only) nodes:

Component	Recommended resources	Role
Kafka Connect + Debezium	2-4 cores, 4-8 GB RAM	Read the WAL, emit events
Kafka brokers (×3)	4 cores, 32 GB RAM each	High availability, retention
Qdrant-sync consumer	2 cores, 4 GB RAM	Vectorise plus upsert/delete
Qdrant	8 cores, 64 GB RAM	Vector store

The Debezium connector is notably light: in production at 10,000 events/second, the connector typically uses less than 1 core and 2 GB of RAM. JVM memory (Kafka Connect runs on the JVM) must be capped explicitly with -Xmx4g to stop the GC from causing pauses.

For high availability, Kafka Connect supports distributed mode with multiple workers. If a worker dies, the connector is reassigned automatically to another worker within seconds, and the replication slot guarantees no events are lost during the switchover.

What we have not covered

Debezium with MySQL, MongoDB and Oracle: each connector uses the native log mechanism (binlog on MySQL, oplog on MongoDB, LogMiner on Oracle). The resulting event API is similar, but the configuration details and limitations differ.
Debezium Server: standalone mode without Kafka Connect, with direct sinks to HTTP, S3, Redis Streams or NATS. Useful when the Kafka infrastructure is too complex for the use case.
Schema Registry: how Avro with Confluent Schema Registry or Apicurio manages event schema evolution (adding columns, changing types) without breaking existing consumers.
Exactly-once semantics: why at-least-once is enough for most RAG cases (an idempotent upsert into Qdrant with the same vector does no harm) and when exactly-once is needed (financial counters, inventory deductions).
Outbox pattern plus Debezium combined: Debezium reading the outbox table instead of the business table’s WAL directly, the Transactional Outbox + CDC pattern that combines the best of both worlds.

References

Debezium Documentation — PostgreSQL Connector. debezium.io/documentation/reference/stable/connectors/postgresql.html
PostgreSQL Documentation — Logical Replication. postgresql.org/docs/current/logical-replication.html
PostgreSQL Documentation — Write-Ahead Logging. postgresql.org/docs/current/wal-intro.html
PostgreSQL Documentation — Replication Slots. postgresql.org/docs/current/logicaldecoding-explanation.html
Confluent — Kafka Performance Benchmarks (2023). confluent.io/blog/kafka-fastest-messaging-system
Gunnar Morling — Outbox Pattern. morling.dev/blog/sending-messages-as-part-of-database-transactions
Debezium — SMT documentation. debezium.io/documentation/reference/stable/transformations
Qdrant Documentation — Filtering. qdrant.tech/documentation/concepts/filtering

LLM tracing with OpenTelemetry GenAI: the aircraft black box the field stabilised in 2026

Wed, 27 May 2026 11:00:00 +0200

This post complements the one on Prompt versioning with Langfuse and MLflow, where prompt_id@version appeared as a span attribute without the whole pipeline being explained, and the one on Evals for LLMs, which distinguishes tracing from eval. Here we go inside tracing itself.

TL;DR

OpenTelemetry GenAI, the gen_ai.* and mcp.* semantic conventions, have been stable since late 2025, and as of May 2026 they are the substrate on which the whole ecosystem (Langfuse, Phoenix, LangSmith, Braintrust, Arize, Honeycomb, Datadog, New Relic) builds its LLM observability. That means your team instruments once, against OTel, and picks a backend afterwards; not the other way round. The canonical pipeline is application → OTel SDK (OpenLLMetry or openinference) → OTel Collector → backend(s), with two-layer sampling (head-based at 1-5 % to keep volume viable, tail-based at 100 % over errors and high latencies). This post takes apart the exact attributes you have to fill in on each span (gen_ai.system, request.model, usage.input_tokens, and so on), the operational difference between the three usual SDKs, and the anatomy of a real chat-with-RAG-and-tool trace with all its child spans.

You are here: OBSERVE

The analogy: the aircraft black box

An aircraft black box does three things at once:

It records everything without anyone having to ask for it explicitly. Every flight parameter, every cockpit conversation, every action by the pilots is logged with a precise timestamp and a standardised format.
It uses a format everybody knows. Any air accident investigator anywhere in the world can read the black box of any manufacturer, because the schema is agreed between all of them. There is no “Boeing black box” and “Airbus black box”: there is one single black box.
It survives the event: it is designed on the assumption that the aircraft may go down. It does not depend on the aircraft itself to be stored or to be read.

LLM tracing with OTel GenAI does exactly those three things:

It records everything automatically. Every model call, every agent step, every tool call, every RAG retrieval ends up as a span, a unit of work with a start, an end, attributes and a parent/child relationship with other spans.
It uses a common format. The attributes gen_ai.request.model, gen_ai.usage.input_tokens, gen_ai.response.finish_reasons and the rest are agreed by the OpenTelemetry GenAI working group and every backend respects them.
It survives the backend. Spans do not travel straight to Langfuse: they travel to an intermediate OTel Collector that handles buffering, sampling and fan-out to 1, 2 or N backends. If Langfuse goes down, the Collector holds the spans.

The analogy is not decorative: it is the only way to understand why OTel GenAI beat the ecosystem of proprietary SDKs that every vendor shipped in 2023-2024. The standardised black box beats the proprietary recorder, every time.

Why OTel GenAI exists (and why it won)

The state of things at the end of 2024 was broken: every vendor shipped its own SDK with its own attribute names. LangSmith called it model_name, OpenLLMetry called it llm.model, Helicone called it model. If you wanted to change observability provider, you re-instrumented the whole application. For serious teams, that was unacceptable.

OpenTelemetry already had a framework to solve this: the semantic conventions. A working group (the OpenTelemetry GenAI SIG) proposed a schema. In 2025 it reached “Stable” for the attributes of the basic modality (chat completions + tool calls), and in 2026 it also covers multimodal, embeddings and reasoning modalities.

What changes in practice:

Before (proprietary SDKs): you move from Langfuse to Phoenix → you re-instrument, re-deploy everything, lose the history.
Now (OTel): you move from Langfuse to Phoenix → you edit the exporter in otel-collector-config.yaml, reload the Collector, and the new spans go to the new destination without touching a line of application code.

The canonical `gen_ai.*` attributes

As of May 2026 the stable attributes, that is, the ones that are not going to change, are those in the table. There are more in “experimental” state (multimodal, reasoning) and others in “deprecated” (kept for compatibility).

Attribute	Meaning	Example
`gen_ai.system`	Provider family	`openai`, `anthropic`, `vllm`, `huggingface`
`gen_ai.request.model`	Model requested	`gpt-4o`, `meta-llama/Llama-3.1-8B-Instruct`
`gen_ai.request.temperature`	Sampling temp	`0.7`
`gen_ai.request.max_tokens`	Output cap	`1024`
`gen_ai.request.top_p`	Nucleus sampling	`0.95`
`gen_ai.response.model`	Model that answered (may differ)	`gpt-4o-2024-08-06`
`gen_ai.response.id`	ID of the provider’s response	`chatcmpl-9xY...`
`gen_ai.response.finish_reasons`	Finish reasons	`["stop"]`, `["length"]`, `["tool_calls"]`
`gen_ai.usage.input_tokens`	Input tokens	`1247`
`gen_ai.usage.output_tokens`	Output tokens	`412`
`gen_ai.operation.name`	Operation type	`chat`, `text_completion`, `embeddings`
`gen_ai.prompt.id`	ID of the versioned prompt	`customer_support_v3`
`gen_ai.prompt.version`	Specific version	`14`
`gen_ai.prompt.label`	Semantic label	`production`, `staging`, `canary`
`gen_ai.conversation.id`	Multiturn conversation ID	`session_abc123`

And for tool calls (a function invoked by the model), the gen_ai.tool.* attributes:

Attribute	Meaning
`gen_ai.tool.call.id`	ID of the specific tool call
`gen_ai.tool.name`	Name of the tool
`gen_ai.tool.type`	`function`, `mcp`, and so on
`gen_ai.tool.description`	Short description

And for tool calls over MCP (Model Context Protocol), the mcp.* attributes add a specific layer (mcp.server.name, mcp.method, mcp.session.id and others) that coexists with the gen_ai.* ones.

The practical rule: if your instrumentation fills in these attributes on every span, the Langfuse dashboard, the Tempo graph and the Prometheus metrics come out “for free”, because all three know how to read the convention.

The canonical pipeline: SDK → Collector → backends

Three layers, each with its own responsibility:

1 · The SDK

Three dominant options in 2026:

OpenLLMetry (Traceloop) — the most widespread. It automatically instruments the OpenAI Python SDK, the Anthropic SDK, LangChain, LlamaIndex, Haystack, Cohere, Mistral, vLLM and almost any library in the ecosystem. A single line (Traceloop.init()) instruments everything. Apache 2.0 licence. It maintains its own fork with extensions that are not yet in OTel core, but it exports in line with the convention.

openinference (Arize) — a direct competitor, also Apache 2.0. Closer to Phoenix (same vendor) but it exports standard OTel. Better instrumentation for LangChain/LlamaIndex/DSPy in some versions; worse for vLLM directly.

Langfuse SDK — proprietary in form (the API is Langfuse’s), but underneath it emits OTel spans. The natural choice if Langfuse is the main backend. It has the best support for “session” and “user” linking, concepts that are not yet in OTel core but that Langfuse maps to gen_ai.conversation.id for compatibility.

Practical recommendation for May 2026: OpenLLMetry if you want to stay backend-agnostic (you will change provider); the Langfuse SDK if Langfuse is already your bet (you save yourself a mapping). Both produce valid spans that any Collector consumes.

2 · The OTel Collector

The Collector is the most important piece of the pipeline and the least talked about. Three responsibilities:

Buffering: if Langfuse or Tempo have a problem, the Collector holds the spans in memory (or on disk with persistence enabled) and re-emits them when the backend returns.

Sampling: it applies the two sampling layers (next section) without the application knowing anything about it.

Routing: with the exporters config it points at one or several backends at once. The usual practice is to send to Langfuse + Tempo + Prometheus simultaneously:

Langfuse for the LLM-first UI (prompts, evals, user sessions).
Tempo (or Jaeger) for the full distributed context (an LLM span inside an HTTP request that has touched 12 microservices).
Prometheus for aggregated metrics (P95 latency per model, tokens per second, error rate).

A representative production fragment of otel-collector-config.yaml:

receivers:
 otlp:
 protocols:
 grpc:
 endpoint: 0.0.0.0:4317
 http:
 endpoint: 0.0.0.0:4318

processors:
 batch:
 timeout: 1s
 send_batch_size: 1024
 tail_sampling:
 decision_wait: 10s
 policies:
 - name: errors
 type: status_code
 status_code: { status_codes: [ERROR] }
 - name: slow_traces
 type: latency
 latency: { threshold_ms: 5000 }
 - name: head_5pct
 type: probabilistic
 probabilistic: { sampling_percentage: 5 }

exporters:
 otlphttp/langfuse:
 endpoint: https://langfuse.internal:3000/api/public/otel/v1
 headers: { Authorization: "Basic ${LANGFUSE_AUTH}" }
 otlp/tempo:
 endpoint: tempo:4317
 tls: { insecure: true }
 prometheus:
 endpoint: 0.0.0.0:8889

service:
 pipelines:
 traces:
 receivers: [otlp]
 processors: [tail_sampling, batch]
 exporters: [otlphttp/langfuse, otlp/tempo]
 metrics:
 receivers: [otlp]
 processors: [batch]
 exporters: [prometheus]

3 · The backends

Langfuse is the dominant option for OSS LLM observability (MIT licence, complete suite: tracing + evals + prompts + datasets). Its model assumes OTel trace_id and span_id and draws them in the UI with the LLM context expanded (formatted messages, counted tokens, costs calculated from a model→price mapping).

Phoenix (Arize, ELv2 licence) has a more eval-first focus. Useful if your team comes from classic NLP and wants UMAP embedding visualisation by default.

LangSmith (LangChain) is proprietary, not OSS. Excellent integration with LangChain/LangGraph; less relevant if you do not use that stack.

Tempo / Jaeger show the LLM traces inside the context of the distributed request. It is what you need when a customer reports that “the page takes 8 seconds” and you want to see whether those 8 seconds are the model, the retrieval or the downstream microservice.

Sampling: the two layers that coexist

Without sampling, a system at 10 req/s generates about 26 million spans a month. The backend drowns, the cost explodes and most of the spans are redundant (200 similar requests tell you no more than 20).

The canonical pattern in 2026 is two combined layers:

Head-based sampling (on the way in)

You decide before the span exists whether you are going to trace it or not, based on a probabilistic decision (1-5 % typically) or on a deterministic rule (always trace if the user is premium, if the model is the new one in canary, and so on).

Pros: cheap (the span is not generated at all), deterministic.

Cons: you can miss infrequent errors. If an error happens once in every 1,000 and you trace 1 %, you will see it once in every 100,000, which is no longer debuggable.

Tail-based sampling (on the way out)

The complete span is generated and at the end you decide whether to keep it or throw it away. The decision can be:

Keep 100 % of errors.
Keep 100 % of traces with latency > X ms.
Keep 100 % of traces that triggered a guardrail.
Keep a random sample of the rest (1-5 %).

Pros: it guarantees that you see the errors and the latency outliers. Without it, the most interesting problems are the ones most likely to be left out.

Cons: you need a buffer, because until the trace finishes you cannot decide. The Collector’s tail_sampling processor keeps the spans in memory for 5-30 seconds until it takes the decision.

The practical rule in 2026 is to combine both: head-based at 5 % + tail-based capturing 100 % of errors and latencies > 5s. That gives you around 5 % of baseline traffic plus 100 % of what is interesting.

Anatomy of a real trace

We are going to take apart the trace of a single user question to a typical chat-with-RAG-and-tool system. The question is “what is the balance of my main account?”.

Seven spans, one single trace. The information you can exploit:

Latency broken down: 38 ms guardrail + 218 ms retrieval + 2104 ms LLM + 60 ms tool + 28 ms guardrail = 2,448 ms (the rest is orchestration overhead). The bottleneck is the LLM (74 % of the time). If the user complains about slowness, you know where to look.
Tokens and cost: 1,247 input + 412 output. With Llama 3.1 70B on-prem at about 12 W/token of equivalent energy cost, roughly 3.5 c€ per response. Multiplied by volume, that gives you the inference bill.
prompt.id+version travelling along: if in a month’s time you notice that quality has dropped, you filter by prompt.version and see whether it coincides with a change to the prompt.
Guardrail decisions traced: if a guardrail blocked the response, it stays in the trace with gen_ai.guardrail.decision=block and a reason. ENS audit satisfied.
Correlated tool calls: tool.get_balance is a child span of gen_ai.chat. If the tool failed, you see the error directly in its span; it does not turn up only in separate logs.

Implications for on-premise hardware

Tracing is not free, but its overhead, properly sized, is negligible.

Cost per span

At the SDK level, an LLM span adds around 50-200 µs of overhead (creation, attributes, serialisation). For an LLM call lasting 2 seconds, that is < 0.01 % of the latency.

At the network level, a typical compressed OTLP span weighs around 1-3 KB. For a system at 100 req/s with 7 spans per request, that is 700 spans/s × 2 KB ≈ 1.4 MB/s to the Collector. Trivial.

Storage

This is where the money goes. Without sampling, a month at 100 req/s is about 18 GB of spans. With head-based at 5 % + tail-based on errors, about 1-2 GB a month. Manageable in any self-hosted ClickHouse or Loki.

On an RTX 4090 (24 GB) + Collector and Langfuse on-prem

For a demo service or a small single-tenant: Langfuse + Postgres + Clickhouse + Collector on the same host as the model. It uses around 8 GB of RAM and < 1 % of CPU continuously. The 4090 serves the model, the rest lives on the CPU. Minimum viable setup for a startup or a small team.

On a generic 4×H100 SXM cluster + central observability

A dedicated pod for the OTel Collector (a DaemonSet on each node + a central Gateway). Langfuse + ClickHouse + Tempo + Prometheus + Grafana in an observability namespace separate from the serving namespace. This is the canonical architecture of any serious ENS/NIS2 production deployment: observability does not share resources with the model.

What we have not covered (upcoming articles)

LLM metrics (Prometheus): the canonical counters and histograms — gen_ai_client_request_duration_seconds, gen_ai_client_input_tokens_total and the rest. How Grafana dashboards are built on that base.
Guardrails as spans: how to model Llama Guard 4, NeMo Guardrails and LLM Guard as child spans of the LLM span, and what attributes they carry. Covered in the post on guardrails — attributes gen_ai.guardrail.line, gen_ai.guardrail.detector, gen_ai.guardrail.category, gen_ai.guardrail.score, gen_ai.guardrail.action.
Distributed tracing with MCP — traceparent propagation from the LLM client to the MCP server, mcp.* attributes, known problems.
eBPF for automatic tracing without an SDK — Tetragon and Hubble extracting LLM traces without explicit instrumentation, for cases where the code cannot be modified.

References

OpenTelemetry GenAI Semantic Conventions: https://opentelemetry.io/docs/specs/semconv/gen-ai/.
OpenLLMetry (Traceloop): https://github.com/traceloop/openllmetry.
openinference (Arize): https://github.com/Arize-ai/openinference.
Langfuse OTel integration: https://langfuse.com/docs/opentelemetry/get-started.
OTel Collector tail-sampling processor: https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/processor/tailsamplingprocessor.
Phoenix (Arize) docs: https://docs.arize.com/phoenix.
Model Context Protocol observability (mcp.* semconv): https://modelcontextprotocol.io/docs/observability.

LLM-as-judge: the exam marker who grades other models without turning into an oracle

Wed, 27 May 2026 08:30:00 +0200

This post goes deeper into the judges section of Evals for LLMs. There it was one piece of the mixed panel; here we get into why it works, where it breaks and how its calibration is measured before accepting it in CI.

TL;DR

An LLM judge is not “a GPT-4 you ask whether the answer is any good”. It is a trained marker: it has a rubric written in advance, it is asked for explicit reasoning before the verdict, its score is computed as an expectation weighted by token probabilities (not as the first token it spits out), and before being accepted in production it is calibrated against a sample of ~50 human-annotated examples until it reaches κ ≥ 0.5 (Cohen’s kappa). The state of the art in May 2026 is three patterns, G-Eval, Prometheus 2 and panel of judges, each answering a different trade-off between cost, quality and reproducibility. All of them share four documented biases: position, verbosity, self-preference and narcissism. This post explains how a real judge is built, how to measure whether it lies, and when each pattern is appropriate.

You are here: EVAL

The analogy: the civil service exam marker

A public competitive exam has thousands of papers. There is no way the senior board can mark them all. The Spanish system’s answer has been the same for decades: trained markers. People who are not professors, but who are given:

A marking template written in advance: what is valued, how many points each section is worth, what gets deducted.
Prior training with a sample of papers already marked by the senior board, until their marks agree reasonably well.
Continuous auditing: a fraction of their marking is re-marked by the senior board to verify that the marker is not drifting.

A well-built LLM judge is exactly that. It is not an oracle, it is a trained marker:

The rubric is the explicit criteria in the prompt (what faithfulness is, what relevancy is, and so on).
The training is the calibration against ~50 human-annotated examples.
The continuous auditing is the weekly sampling where a human re-evaluates a fraction of the traffic the judge has judged.

And, as with the human marker, the judge has systematic biases that the public exam system has learned to watch for: markers prefer certain handwriting styles, certain lengths, certain structures. The same happens to the judge. The rest of this post takes apart exactly which ones and how they are measured.

Why LLM-as-judge exists

The direct reason: money and time. A professional human annotation costs on the order of 0.50 € to 5 € per example (depending on complexity and domain), and takes 30 seconds to several minutes. A GPT-4 judgement costs ~0.01-0.05 € and takes ~2 seconds. For a golden dataset of 500 examples evaluated continuously over 10 candidates a day, the difference is between 2,500 € a day and 50 € a day. And the wall-clock difference is between days and minutes.

The less direct but more relevant reason: methodological scalability. A golden dataset of 500 examples is relatively easy to annotate once. What happens afterwards is the hard part:

Every time a candidate adapter appears, all 500 have to be re-evaluated.
Every time the dataset is updated (because a new incident came in) it has to be re-evaluated.
Every time the system prompt changes it has to be re-evaluated.

Without an automatable, cheap judge, the eval battery stops running and the system goes blind between releases. That is what really justifies the pattern.

The three canonical patterns in 2026

1 · G-Eval

Published by Liu et al. (2023). The base idea is so simple it fits in one sentence: give the judge a detailed rubric, ask it to reason before scoring, and read the result as a weighted expectation instead of as the first token. The three levers:

Rubric: a prompt with an explicit criterion (e.g. “Faithfulness: the degree to which the answer relies only on the context, without inventing data. Score 1 = invents everything, 5 = everything supported by the context”), ideally with one or two examples per extreme value.

Chain-of-thought + form-filling: the model is first asked to “reason briefly about each criterion” and then to “fill in this JSON form”. That forces it not to spit out an arbitrary number.

Probability-weighted scoring: instead of reading the first token after the score: field, you look at the model’s probability distribution over the tokens 1, 2, 3, 4, 5 and compute:

$$\hat{s} = \sum_{i=1}^{5} i \cdot \frac{p(\text{token}=i)}{\sum_{j=1}^{5} p(\text{token}=j)}$$

This turns a discrete score into a continuous one. The justification: if the judge was “torn” between 4 and 5 (probabilities 0.4 and 0.5 on 4 and 5), the real score is 4.55, not 5. This sharply reduces variance across runs of the same prompt, and captures information that greedy decoding throws away.

Limitations of G-Eval: it needs access to the model’s logprobs. Closed-source models have been restricting that access (Claude does not expose it, GPT-4 does but only top-5). In 2026 G-Eval with strict probability weighting is only practical against open source models you serve yourself (vLLM exposes them) or against GPT-4 with logprobs=true.

2 · Prometheus 2

Published by Kim et al. (KAIST, 2024). The insight is complementary to G-Eval: what if, instead of asking a generalist judge to evaluate, we fine-tune a specific judge?

Prometheus 2 is a Mistral 8×7B (MoE, ~47 GB in BF16, ~24 GB in INT4) fine-tuned on 100k+ evaluation examples with varied rubrics. The metric published in the paper: 0.897 Pearson correlation with GPT-4-as-judge on the Vicuna Bench and similar. That matters because it means GPT-4 can be replaced as a judge at the cost of local inference without losing almost any quality.

Why it matters in on-premise production:

No data leaves the perimeter. For strict ENS/NIS2 customers this is not a preference, it is a requirement. A judge that travels over an external API is not an option there.
Zero marginal cost. When the judge runs on-prem, evaluating 50,000 cases a day adds no external bill.
Controlled latency. Continuous eval over real traffic can run in parallel without saturating an external provider’s rate limits.

The price: you have to maintain one more inference service (Prometheus 2 running in its own vLLM), and the judge does not “update” unless it is re-fine-tuned.

3 · Panel of Judges

Verga et al. (Cohere, 2024) formalised it: instead of a single judge, use 3-5 heterogeneous judges, different models, different prompts, different temperatures, and aggregate their judgements.

Common aggregation mechanisms:

Median for continuous scores. Robust to outliers.
Majority vote for pairwise judgements (chosen vs rejected). If 3 of 5 prefer A, the winner is A.
Calibration-weighted mean: weight each judge by its κ against humans in calibration. The more reliable judges vote more.

What a panel gives that a single judge does not:

Reduction of self-preference (a judge’s bias towards outputs stylistically similar to its own): if the judges come from different vendors, the sum cancels out.
A measure of case difficulty: if all 5 judges agree, the case is easy; if they are split, the case is ambiguous and should be escalated to a human. That turns the panel into an automatic triaging system for human annotation.
Lower variance: each judge’s noise is averaged out.

The cost: 3-5× the bill of a single judge. That is why the panel is usually reserved for critical eval gates (does this adapter get promoted or not?), not for continuous eval over traffic.

How to measure whether the judge lies: Cohen’s kappa

Accepting the judge in production without measuring its agreement with humans is the same as accepting an uncalibrated thermometer. The standard metric for inter-rater agreement with discrete or ordinal scales is Cohen’s kappa:

$$\kappa = \frac{p_o - p_e}{1 - p_e}$$

Where p_o is the observed agreement proportion (what percentage of examples judge and human agree on) and p_e is the agreement proportion expected by chance (what you would expect if both scored at random respecting each one’s marginals).

The intuition: if p_o = 0.9 but p_e = 0.85 (because both nearly always score “4 or 5”), agreement is 90 % in raw terms but κ = 0.33: most of the agreement comes from both scoring high, not from them understanding each other. κ corrects for that baseline.

The usual interpretive scale (Landis and Koch 1977, still the reference):

κ	Interpretation	Production threshold
< 0.20	Poor	Useless
0.21–0.40	Fair	Weak signal only
0.41–0.60	Moderate	Minimum acceptable in 2026
0.61–0.80	Substantial	State of the art for open source judges
0.81–1.00	Almost perfect	Human judges rarely reach it between themselves

An important point the field learned the hard way: humans between themselves rarely exceed κ = 0.70 on LLM tasks (faithfulness, relevancy). That is the realistic ceiling for an LLM judge. Chasing κ = 0.9 against humans is chasing a ghost: not even two human annotators get there.

Weighted kappa for ordinal scales

For 1-5 scores, the disagreement “judge says 4, human says 5” is not the same as “judge says 1, human says 5”. Standard kappa treats both as an identical failure. Linear or quadratic weighted kappa assigns weight to the magnitude of the disagreement:

$$\kappa_w = 1 - \frac{\sum_{i,j} w_{ij} \, o_{ij}}{\sum_{i,j} w_{ij} \, e_{ij}}, \quad w_{ij}^{(\text{lin})} = \frac{|i-j|}{k-1}, \quad w_{ij}^{(\text{quad})} = \frac{(i-j)^2}{(k-1)^2}.$$

In G-Eval with 1-5 scores, the usual practice is to publish κ_quad because it penalises large disagreements more and comes closer to human intuition.

Numerical calibration example

Imagine a golden set of 50 examples scored by a human and by the judge, both on a 1-5 scale. The confusion matrix:

	Hum 1	Hum 2	Hum 3	Hum 4	Hum 5	Judge total
Judge 1	3	1	0	0	0	4
Judge 2	1	4	2	0	0	7
Judge 3	0	1	6	2	0	9
Judge 4	0	0	1	12	3	16
Judge 5	0	0	0	2	12	14
Human total	4	6	9	16	15	50

Diagonal (exact agreements): 3+4+6+12+12 = 37 → p_o = 0.74.

p_e is computed as Σ_i (n_judge_i · n_hum_i) / n² = (4·4 + 7·6 + 9·9 + 16·16 + 14·15) / 50² = (16+42+81+256+210) / 2500 = 605/2500 ≈ 0.242.

$$\kappa = \frac{0.74 - 0.242}{1 - 0.242} = \frac{0.498}{0.758} \approx 0.66$$

Substantial. Acceptable. If we wanted quadratic weighted κ, near disagreements (Judge 4 vs Hum 5) weigh less than distant ones (Judge 2 vs Hum 4), and κ_quad typically comes out 0.05-0.10 above the linear one.

The four biases of the judge

Position bias

First documented by Wang et al. (2023). If you present two answers A and B to the judge in a pairwise setting, it prefers A more often than B, on the order of 55-65 % when A and B are objectively equivalent, with typical 2023-2024 judges. In 2026 frontier judges (GPT-5, Claude 4.5, Llama 4 Judge) have it fairly well mitigated, but it is still measurable.

How to measure it formally: run the dataset twice, once with (A, B) and once with (B, A). If the judge is consistent, the two passes should agree. The fraction of cases where the verdict flips is the position-bias rate. Accepted threshold in 2026: < 10 %.

Canonical mitigation: always run two passes with the order reversed and average. Frameworks such as Promptfoo and Inspect AI do it by default.

Verbosity bias

Documented by Saito et al. and Dubois et al. in 2024. For open-ended tasks (faithfulness, helpfulness), the judge tends to give a higher score to longer answers. The typical Pearson correlation between score and length on answers of equivalent human-rated quality can rise to 0.4-0.5 without mitigation.

How to measure it: Pearson correlation between the judge’s score and the length of the answer, computed over a subset where humans have confirmed equivalent quality. If the correlation is > 0.3 on that controlled subset, there is significant verbosity bias.

Mitigation: a rubric explicitly neutral on length (“the answer should be appropriately concise for the question; length is not a criterion”) and few-shot examples where a short answer beats a long one. AlpacaEval 2.0 builds a length correction directly into the metric.

Self-preference bias

Documented by Panickssery et al. (Anthropic, 2024). A GPT-4 judge prefers GPT-4 outputs. A Claude judge prefers Claude outputs. Not out of conspiracy, but because models share stylistic patterns with their close relatives (paragraph structure, use of bullet points, tone).

How to measure it: over a golden set, compare the scores of 3 different judges (e.g. GPT-4, Claude, Llama 4 Judge). If for the same candidate there is > 15 % systematic divergence attributable to the candidate’s identity, there is self-preference.

Mitigation: a panel of judges with heterogeneous vendors. If a single judge is going to be used, it must not come from the same vendor as the model being evaluated.

Narcissism

The extreme case of self-preference: the judge is exactly the same model as the candidate. This happens more than it seems: a team trains a Llama 3 8B with LoRA and evaluates it with Llama 3 8B as the judge because “it is what they have on-prem”. It is methodologically invalid. The delta between human score and judge score grows measurably.

Mitigation: a judge of a different architecture from the candidate. If your candidate is Llama 3, your judge should be Mistral, Qwen or a Prometheus 2 (which, although based on Mistral, was fine-tuned specifically for evaluation).

The judge as the pipeline’s hinge

The judge is not just “the eval piece”. It is the hinge that connects three stages of the pipeline:

Towards TUNE: the judge generates the (chosen, rejected) pairs for DPO without needing human labellers. That chain is what makes continuous fine-tuning work without a dedicated annotation team.

Towards the EVAL gate: the judge gives the score that is compared against the CI threshold. If the adapter does not clear 0.85 on faithfulness, no merge.

Towards continuous EVAL: over a sample of real traffic (1-5 %), the judge computes scores and persists them. That makes it possible to detect regressions that appear days after deployment and that CI did not see because its golden set did not cover them.

Towards RETRAIN: the cases where the judge gives a low score are automatic candidates for the next retraining dataset. The judge acts as the triage of the incident flow.

Implications for on-premise hardware

The numbers below are indicative for typical scenarios in May 2026.

On an RTX 4090 (24 GB)

Judge	Does it fit?	Approximate throughput	Notes
GPT-4o (API)	n/a	~50-100 judgements/min	External cost, not local
Prometheus 2 (8×7B INT4)	Yes, just about	~40-80 judgements/min	Q4_K_M GGUF, llama.cpp
Llama 3.1 8B fine-tuned judge	Yes, comfortably	~150-250 judgements/min	Reasonable on-prem default
Mistral Small Judge 22B	Not directly, requires offload	~10-20 judgements/min	Too much for 24 GB in BF16

Conclusion: on a single 4090, an open source 8B judge fine-tuned for evaluation (or a quantised Prometheus 2) is the sweet spot.

On a generic 4×H100 SXM cluster (320 GB, NVLink)

Judge	Configuration	Approximate throughput	Notes
Prometheus 2 BF16	TP=2	~400-700 judgements/min	Fits comfortably, low latency
Llama 3.3 70B Instruct	TP=4	~150-300 judgements/min	If used as a generalist judge
Panel of 3 judges in parallel	TP=1-2 each	~600-1200 judgements/min combined	Natural pattern in a cluster

On an NVLink cluster the natural approach is to run a panel of judges in parallel (each judge taking 1-2 GPUs) with a LiteLLM router in front. That removes the cognitive cost of “which judge do we use” because all three are used and the result is aggregated.

What we have not covered (upcoming articles)

Red teaming and safety eval: how robustness against adversarial prompts is evaluated. It is a different pattern from the ordinal judge.
Multistep agent eval: AgentBench, TauBench, evaluating trajectories instead of individual outputs.
Benchmark contamination: how to detect whether the evaluated model saw the golden set during pre-training, and why public benchmarks are half broken.
Cost-aware judging: when a cheap judge (Llama 8B) beats an expensive one (GPT-4o), and how to quantify the quality/cost trade-off with Pareto curves.

References

Liu, Y., Iter, D., Xu, Y., Wang, S., Xu, R., Zhu, C. G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment (EMNLP 2023).
Kim, S., Suk, J., Longpre, S., Lin, B. Y., Shin, J., Welleck, S., Neubig, G., Lee, M., Lee, K., Seo, M. Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models (EMNLP 2024).
Verga, P., Hofstatter, S., Althammer, S., Su, Y., Piktus, A., Arkhangorodsky, A., Xu, M., White, N., Lewis, P. Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Models (Cohere, 2024).
Panickssery, A., Bowman, S., Feng, S. LLM Evaluators Recognize and Favor Their Own Generations (Anthropic, NeurIPS 2024).
Wang, P., Li, L., Chen, L., Cai, Z., Zhu, D., Lin, B., Cao, Y., Liu, Q., Liu, T., Sui, Z. Large Language Models are Not Fair Evaluators (ACL 2024).
Dubois, Y., Galambosi, B., Liang, P., Hashimoto, T. Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators (Stanford, 2024).
Cohen, J. A Coefficient of Agreement for Nominal Scales (Educational and Psychological Measurement, 1960).
Landis, J. R., Koch, G. G. The Measurement of Observer Agreement for Categorical Data (Biometrics, 1977).

Reranker and hybrid retrieval: the committee that decides the 5 chunks the LLM will actually read

Mon, 25 May 2026 14:00:00 +0200

This post takes apart the retrieval layer inside the RAG piece of the six-stage LLMOps pipeline. It sits directly on top of the corpus curation post: if the librarian decided what enters the index, the committee in this post decides what leaves the index for the LLM’s face. It is the stage that moves the real quality of a RAG in production the most, and the one most teams solve with “dense top-k and off we go”.

TL;DR

The most widespread antipattern in real RAG is: curated corpus, dense embeddings with bge-m3, the user’s query straight through, top_k=5 of cosine similarity and off to the LLM. The system delivers answers that look good in the demo, mediocre with real traffic and dreadful with three-word queries, technical jargon, internal abbreviations or anything outside the embedder’s fine-tuning domain. The root cause is almost never the model: it is that a single pass of dense retrieval over the raw query is structurally insufficient for the cases a real user produces. In 2026 the field has consolidated the three-layer retrieval pattern: a broad layer that mixes dense + sparse (BM25 / SPLADE) via Reciprocal Rank Fusion to secure recall, a fine layer with a cross-encoder reranker (BAAI/bge-reranker-v2-m3, Cohere Rerank 3, mxbai-rerank-large) that reorders the 30-50 candidates by true relevance, and optionally a late interaction layer with ColBERT-v2 or an LLM reranker for critical cases. On top of it all, a query rewriting pattern that rewrites malformed queries and HyDE, which generates a hypothetical document so you embed the expected answer instead of the question. This post goes through them one by one: why each layer exists, what latency budget it consumes, the minimum maths of RRF and of the recall/precision trade-off, the dominant OSS stack in 2026, the on-premise hardware for serving it under data sovereignty, and the seven operational traps that kill the stage.

The analogy: the committee for a professorial chair

A university department has a chair to fill and receives 3,000 applications. Nobody is going to read 3,000 CVs in depth. The real process is organised in rounds:

Round 1, mass pre-screening. A system of automatic filters reads each CV in seconds. One route looks at the keywords of the post (publications in specific journals, languages, certifications); another route looks at the overall profile (research area, h-index, career). The two routes run in parallel. Each proposes its shortlist of 30-50 candidates. The two rankings are combined and what remains is a list of some 50-100 candidates for the next round to act on. The metric that matters here is recall: not losing the good candidate through a filter error.

Round 2, careful reading. A member of the panel reads each CV on the shortlist thoroughly and scores it against the real profile of the post. It is slow, 10-20 minutes per CV, but it discriminates far better than the automatic filter. It reorders the 100 candidates into a fine ranking and keeps the 5 finalists. The metric that matters here is precision: that the top-5 is genuinely the real top-5.

Round 3 (optional), personal interview. Long interviews are held for the 5 finalists. Face to face, questions adapted to each candidate’s profile, a check against the reality of the post. It is extremely expensive and very slow, but for critical chairs it justifies the cost.

The retrieval of a modern RAG works exactly like this. Change the vocabulary and the structure holds:

The equivalence is exact. The broad layer guarantees that the right chunk is on the shortlist (recall). The fine layer orders well inside the shortlist (precision). Each layer is justified by an operational asymmetry: the broad one is cheap and parallelises, the fine one is expensive and sequential. Solving everything with the expensive one, a single cross-encoder over the whole corpus, is operationally unsustainable. Solving everything with the cheap one, a single dense top-k, is functionally insufficient.

The problem: why dense on its own is not enough

The argument against dense-only is not theoretical. It is that the queries a real user produces have patterns the dense bi-encoder does not capture well by construction:

Short queries. “IRPF dates 2026” goes through the embedder and produces a vector close to any chunk that mentions dates and IRPF, including chunks from 2023, chunks of generic calendars, chunks of corporate IRPF when you were asking about personal income tax. BM25 disambiguates better because the weight of the infrequent terms (“2026”, “IRPF”) dominates the score.

Infrequent or internal technical terms. Product codes (FBR-X42-PRO), organisation abbreviations (PYM-23-bis), internal variable names. The embedder, trained on a public corpus, breaks them down into sub-tokens and pushes them into the “noisy” zone of the latent space. Two completely different codes can end up 0.02 cosine distance apart. BM25 treats them as exact tokens and scores them correctly.

Polysemy. “Java” as a language vs Java as an island vs Java as coffee. The dense embedding averages the meanings; the ranking becomes mediocre. Here dense can beat BM25 if the query carries enough context, but that “if” is rarely met with real user queries.

Synonyms and rephrasings. “How do I cancel my subscription” vs a chunk titled “Service termination procedure”. Here dense beats BM25: the semantic space brings them together even though they share no tokens. It is exactly the case where dense shines and BM25 fails.

The operational conclusion: no single retriever covers the four cases. The combination covers them all. And for the cases where both retrievers fail (very ambiguous queries, contradictory context between chunks), the third layer comes in: the reranker that reads the query and each chunk together with a cross-encoder, and produces a real relevance score far more discriminating than cosine similarity.

Layer 1 — Hybrid retrieval: dense + sparse

Why BM25 is still alive in 2026

BM25 is from 1994. It is not deep learning. It needs no GPU. And it is still competitive with dense retrievers of hundreds of millions of parameters on many retrieval benchmarks (BEIR, 2021-2024). Its formula is simple:

$$ \mathrm{BM25}(q, d) = \sum_{t \in q} \mathrm{IDF}(t) \cdot \frac{\mathrm{tf}(t,d) \cdot (k_1 + 1)}{\mathrm{tf}(t,d) + k_1 \cdot (1 - b + b \cdot \frac{|d|}{\overline{|d|}})} $$

Where $\mathrm{tf}(t,d)$ is the frequency of term $t$ in document $d$, $|d|$ the document length, $\overline{|d|}$ the mean length of the corpus, and IDF the inverse document frequency. $k_1$ and $b$ are hyperparameters (typically $k_1 = 1.2$, $b = 0.75$). In practice you use pyserini, Tantivy, Elasticsearch or OpenSearch and never write the formula.

SPLADE (Formal et al., 2021) is the neural generation of BM25: it learns sparse weights over the BERT vocabulary, expanding each token to its related terms. It keeps BM25’s interpretability (you can see which query terms match which document terms) and beats plain BM25 on BEIR. The price is that you need GPU inference over the query (and over each document at indexing time, once).

The rule of thumb in 2026:

Corpus < 10M chunks, queries with a broad vocabulary: plain BM25 via Elasticsearch / Tantivy. Trivial indexing, 1-5ms queries.
Corpus 10M-100M, specific domain: SPLADE-v3 indexed in Elasticsearch or Vespa. Slow indexing (you need a GPU once), 5-20ms queries.
Corpus > 100M, latency critical: BM25 with external query expansion (synonyms, query2doc) is the operational option, with SPLADE reserved for the critical segments.

Dense bi-encoder: the second lane

The bi-encoder produces one vector per chunk and another per query, and relevance is the cosine similarity (or dot product) between the two. The dominant multilingual model in 2026 is BAAI/bge-m3: 568M params, 1024 dimensions, support for up to 8,192 context tokens per chunk, multivector (dense + sparse + ColBERT-style multi-vec in the same embedder), trained on 100+ languages. Alternatives:

Embedder	Params	Dim	Multilingual	Comment
`BAAI/bge-m3`	568M	1024	yes	the default all-rounder in 2026
`intfloat/multilingual-e5-large-instruct`	560M	1024	yes	direct competitor, instruct-style queries
`nomic-ai/nomic-embed-text-v1.5`	137M	768	English	fast, variable Matryoshka dimension
`jinaai/jina-embeddings-v3`	570M	1024	yes	with task-specific LoRAs per domain
`mixedbread-ai/mxbai-embed-large-v1`	335M	1024	English	top English on MTEB
`Snowflake/arctic-embed-l-v2.0`	568M	1024	yes	enterprise-oriented, Apache licence

The practical selection criterion: multilingual no matter what if your corpus or queries need it (bge-m3, multilingual-e5, jina-v3, arctic-l-v2), Matryoshka if you want to save vector store memory by reducing dimensions without re-embedding (nomic), and the MTEB leaderboard ranking only as a tie-breaker, never as the sole criterion, since MTEB benchmarks are contaminable and a 1-2 point difference rarely translates into a real improvement in your domain.

The fusion: Reciprocal Rank Fusion (RRF)

You have two rankings, one from BM25 and one from dense. How do you combine them? The naive option, adding the scores, fails because the scores are not comparable (BM25 produces numbers of 0-30, cosine 0-1, L2 distances 0-2…). The option that has consolidated is Reciprocal Rank Fusion (Cormack, Clarke, Buettcher 2009), which ignores absolute scores and uses only the rankings:

$$ \mathrm{RRF}(d) = \sum_{r \in R} \frac{1}{k + \mathrm{rank}_r(d)} $$

Where $R$ is the set of retrievers and $\mathrm{rank}_r(d)$ is the position of document $d$ in the ranking of retriever $r$. The constant $k$ is typically $60$ and smooths the weight of the first positions.

A numerical example. You have BM25 with top-5: [d_a, d_b, d_c, d_d, d_e] and dense with top-5: [d_b, d_f, d_a, d_g, d_h]. You compute the RRF of each candidate:

Doc	BM25 rank	dense rank	$\frac{1}{60+r_{\text{BM25}}}$	$\frac{1}{60+r_{\text{dense}}}$	total RRF
`d_b`	2	1	0.01613	0.01639	0.03252
`d_a`	1	3	0.01639	0.01587	0.03226
`d_c`	3	—	0.01587	0	0.01587
`d_f`	—	2	0	0.01613	0.01613
`d_d`	4	—	0.01562	0	0.01562

d_b wins because it appears well ranked in both. d_a follows it. Those that appear in only one list stay below. With nothing tuned, RRF tends to push the consensus candidates to the top, which is exactly what you want as a coarse filter before the reranker.

Variants: weighted RRF assigns a multiplier per retriever ($w_r \cdot \frac{1}{k + \mathrm{rank}_r(d)}$) when you have reasons to trust one more; learning-to-rank trains a model (LambdaMART or a LightGBM ranker) over features from the two retrievers, a more powerful option but one that demands relevance labels most teams do not have.

In Qdrant, Vespa, Weaviate and Elastic 8.11+ the RRF fusion is built in as a native mode: you ask for hybrid search with a lexical query and a vector and they return the combined top-k without you having to compute the fusion.

Layer 2 — The reranker: bi-encoder vs cross-encoder

The bi-encoder looks at query and document separately and compares the resulting vectors. It is extremely fast (a dot product) and lets you index the document embeddings once and for all. The price is that the model never sees query and document together, and the compressed representation loses nuance.

The cross-encoder looks at query and document concatenated: [CLS] query [SEP] document [SEP] goes through the model and the output is a relevance score. It discriminates far better, since shades of ordering and of local relationship between terms are captured, but the cost explodes: each (query, doc) pair requires a full inference of the model. You cannot precompute anything. If you want to rerank 50 candidates, that is 50 inferences.

Aspect	Bi-encoder	Cross-encoder
Inference per chunk at indexing	yes, once	not precomputable
Inference at query time	1 (query) + dot product per chunk	1 per (query, chunk) pair
Typical latency for top-50	5-20 ms (CPU possible)	80-300 ms (GPU recommended)
Discrimination	medium	high
Typical use	recall (layer 1)	precision (layer 2)

This asymmetry is why the canonical pattern is a two-step funnel: the bi-encoder does cheap broad recall and the cross-encoder does expensive fine precision, over 30-50 candidates instead of millions.

2026 OSS cross-encoder reranker stack

Reranker	Params	Multilingual	top-50 latency (A100)	Comment
`BAAI/bge-reranker-v2-m3`	568M	yes (100+ languages)	~90 ms	the de facto default in 2026
`BAAI/bge-reranker-v2-gemma`	2B	yes	~250 ms	more quality, more cost
`mixedbread-ai/mxbai-rerank-large-v2`	1.5B	English	~180 ms	top English on BEIR rerank
`jinaai/jina-reranker-v2-base-multilingual`	278M	yes	~50 ms	fast and enough for many cases
`Alibaba-NLP/gte-multilingual-reranker-base`	306M	yes	~55 ms	direct competitor to jina
`Cohere Rerank 3`	(closed)	yes (100+ languages)	~120 ms (API)	commercial, not on-prem
`Voyage rerank-2`	(closed)	yes	~100 ms (API)	commercial, not on-prem

For deployments under data sovereignty the obvious choice is bge-reranker-v2-m3 as a starting point, with jina-reranker-v2-base as the lightweight alternative. If the domain is exclusively English and the hardware budget allows it, mxbai-rerank-large-v2 takes a point or two more on BEIR. Cohere and Voyage are the option when you can reach out to an external API, ruled out in ENS / NIS2 scenarios with sensitive data.

Late interaction: ColBERT-v2 as a compromise

There is a third option gaining traction: late interaction, embodied in ColBERT-v2 (Santhanam et al., 2022). The idea is that the document is embedded not as a single vector but as a matrix of embeddings, one per token. And at query time the query is also embedded as a matrix, and the similarity is the sum of the maxima per column (MaxSim):

$$ S(q, d) = \sum_{i \in q} \max_{j \in d} \langle E_q[i], E_d[j] \rangle $$

The result is retrieval with quality close to a cross-encoder but at a much lower cost (there is no need to run the whole model over each pair). The price is space: each chunk is ~150-200 vectors instead of 1, and that multiplies the index size by ~100. For small to medium corpora (<10M chunks) it is manageable; for huge corpora PLAID compression and ColBERT-v2 quantization make it viable up to 100M.

In 2026 the reference implementations are RAGatouille (Python wrapper), JaColBERT (Japanese), Vespa.ai (a search engine with native ColBERT support as a first-class citizen), and the native integration in bge-m3 that produces the ColBERT-style multivectors as part of the same dense model.

My operational recommendation: start with a cross-encoder reranker over the top-30 of the hybrid retrieval. Only if the precision numbers fall short, try late interaction as an alternative to the cross-encoder (not as an additional layer). Late interaction as a third layer rarely pays back the operational cost for most cases.

Query rewriting and HyDE: correcting the query before retrieval

The user’s query is the worst-controlled input in the whole system. The most frequent problems:

Queries that are too short: “GDPR fines” → retrieves any chunk about GDPR or about fines.

Queries with typos or jargon: “i wnat to know how to cancel the premium subscripton” → the embedder is noisy because of the typos.

Multi-intent queries: “compare the premium and enterprise plans and tell me which has the better SLA” → dense will try to embed the three questions into one vector.

Queries with references resolved in context: “and for next month?” → without the previous context, dense does not know what it is about.

Canonical 2026 patterns to mitigate:

Query rewriting with a light LLM. Before retrieval, a small model (Qwen2.5-7B, Llama-3.2-3B, Phi-4-mini) rewrites the user’s query into a “canonical” version: it expands abbreviations, fixes typos, breaks multi-intent into sub-queries, resolves references using the conversation history. Cost: one cheap inference (~50-100 ms). Benefit: the four pathologies above are noticeably reduced.

Multi-query: the LLM generates 3-5 rephrasings of the query and you retrieve with each one; then you fuse with RRF. It increases recall at the cost of time and LLM tokens.

Step-back prompting (Zheng et al., 2023): the LLM generates a more general question than the original query (“what is a premium subscription in this system?”), you retrieve with both and combine. It helps with very specific queries that need context.

HyDE — Hypothetical Document Embeddings (Gao et al., 2022): instead of embedding the question, you ask a light LLM to generate a hypothetical answer and you embed that. The reasoning is that the semantic space of answers is closer to that of the real chunks than the space of questions. It works especially well in domains with very specific vocabulary. Cost: one inference of the light LLM (~100-200 ms). Benefit: in many cases +5-15 points of Recall@10.

The operational trade-off is the added latency. Each technique adds 50-300 ms to the pipeline. In synchronous interactions with a 2s SLO there is still headroom; in agentic interactions with tool-use loops, every ms counts. The practical choice in 2026: rewriting with a light LLM as the default, HyDE for specific cases where it has been measured to help, multi-query and step-back only when the case justifies it with numbers.

Minimum maths: latency budget and metrics

Latency budget of a canonical pipeline

A typical hybrid + rerank pipeline in 2026 production over a corpus of 5M chunks with bge-m3 + bge-reranker-v2-m3 on an H100 with TEI:

Stage	p50 latency	p95 latency	Comment
Query rewrite (LLM 3B)	60 ms	150 ms	only if enabled
HyDE (LLM 3B)	90 ms	220 ms	only if enabled
BM25 / SPLADE top-50	8 ms	20 ms	Elasticsearch or local Tantivy
Dense embedding of the query	12 ms	35 ms	bge-m3 with batch=1
Dense top-50 (filterable HNSW)	6 ms	18 ms	Qdrant with scalar quantization
RRF fusion → top-30	<1 ms	<1 ms	arithmetic
Cross-encoder rerank top-30	60 ms	180 ms	bge-reranker-v2-m3 batched
Top-5 selection + format	<1 ms	<1 ms
Total without rewrite/HyDE	~90 ms	~270 ms	realistic range on an H100
Total with rewrite + HyDE	~240 ms	~640 ms	+ ~150-370 ms

On an RTX 4090 (~3× slower than an H100 on a large cross-encoder, similar on everything else) the numbers shift: ~180 ms p50 without rewrite, ~400 ms p50 with rewrite + HyDE. It is perfectly serviceable for a synchronous assistant with a 2-3s SLO, but tight for one with a 500 ms SLO.

Metrics: nDCG, MRR, Recall@k

There are three classic metrics that any retrieval eval reports:

Recall@k: is the relevant chunk among the top-k? A binary metric, it ignores ordering. The one that matters for the broad layer.

$$ \mathrm{Recall@k} = \frac{|\{\text{relevant}\} \cap \{\text{top-k}\}|}{|\{\text{relevant}\}|} $$

Mean Reciprocal Rank (MRR@k): in which position does the first relevant chunk appear? It penalises placing it far down.

$$ \mathrm{MRR@k} = \frac{1}{|Q|} \sum_{q \in Q} \frac{1}{\mathrm{rank}_q} $$

where $\mathrm{rank}_q$ is the position of the first relevant item for query $q$ (or $\infty$ if it does not appear in the top-k).

nDCG@k: it takes into account multiple relevant chunks with degrees of relevance. The most used metric for evaluating rerankers:

$$ \mathrm{DCG@k} = \sum_{i=1}^{k} \frac{2^{\mathrm{rel}_i} - 1}{\log_2(i+1)} \quad \mathrm{nDCG@k} = \frac{\mathrm{DCG@k}}{\mathrm{IDCG@k}} $$

The operational heuristic: Recall@50 measures the quality of your broad layer (we want > 95%; otherwise layer 2 will never be able to recover what was lost), nDCG@5 measures the quality of your reranker (we want > 0.75 on the golden set to consider the system “good”) and MRR@5 measures the quality of your top-1 (it matters especially in systems with top_k=1 or where the LLM’s answer is based mainly on the first chunk).

Costs and throughput

Assuming a bge-reranker-v2-m3 cross-encoder (568M params) on an H100 SXM with TEI 1.7 and dynamic batching:

Throughput: ~280 queries/sec with top-30 each (= ~8,400 chunk-query pairs/sec).
VRAM used: ~3.5 GB for the model + ~6 GB of KV cache + activations at batch=32.
Energy cost: an H100 SXM draws ~700 W. At 8,400 pairs/sec, the energy cost of pure reranking is ~0.083 mJ per pair. In Spanish industrial electricity bill terms (~0.12 €/kWh, May 2026): ~0.000003 € per rerank of 30 candidates. The bill is negligible compared with the cost of the downstream LLM.

On an RTX 4090 with TEI: ~95 queries/sec at top-30 (~2,850 pairs/sec), VRAM ~3.5 GB + ~5 GB. Serviceable for an internal assistant with modest traffic (~5 QPS sustained, peaks at 20-30).

The canonical pattern — which piece goes where

The 2026 operational reference stack for an on-premise hybrid + rerank RAG under data sovereignty:

 ┌─────────────────┐
User query ────────▶│ Query rewriter │ (optional, LLM 3B)
│ (Qwen2.5-7B-IT) │
└────────┬────────┘
▼
┌─────────────────┐
│ HyDE │ (optional)
│ (same LLM) │
└────────┬────────┘
┌─────────────────┼─────────────────┐
▼ ▼ ▼
┌──────────────┐ ┌──────────────┐ ┌──────────────┐
│ BM25/SPLADE │ │ Dense bi-enc │ │ (metadata │
│ Elasticsearch│ │ bge-m3 + TEI │ │ filters │
│ top-50 │ │ + Qdrant │ │ per tenant) │
│ │ │ top-50 │ │ │
└──────┬───────┘ └──────┬───────┘ └──────┬───────┘
└─────────┬───────┘ │
▼ │
┌──────────────┐ │
│ RRF k=60 │◀─────────────────┘
│ → top-30 │
└──────┬───────┘
▼
┌──────────────────┐
│ Cross-encoder │
│ bge-reranker-v2 │
│ TEI batch=32 │
│ → final top-5 │
└──────┬───────────┘
▼
┌──────────────────┐
│ LLM context │
│ ~2,000 tokens │
└──────────────────┘

Key services:

Sparse index: Elasticsearch 8.16, OpenSearch 2.18 or embedded Tantivy. Native BM25, SPLADE optional via plugin.
Vector index: Qdrant 1.13 (filterable HNSW, scalar quantization, multivectors). Covered in detail in the PostgreSQL + Qdrant in ingestion post.
Embedder serving: HuggingFace’s TEI (Text Embeddings Inference) for bge-m3, multilingual-e5 and compatible models. OpenAI-compatible endpoint.
Reranker serving: TEI too, for the rerankers of the bge-reranker, jina-reranker and mxbai-rerank families. Separate /rerank endpoint.
LLM for rewriter/HyDE: vLLM 0.7+ serving Qwen2.5-7B-Instruct or Phi-4-mini-Instruct. If the main LLM is the same, it shares resources; if not, a dedicated pod with GPTQ-INT4 quantization.
Orchestration: the RAG client (FastAPI, LangChain, LlamaIndex) composes the three lanes. Each parallel call (Sparse + Dense) is made concurrently; rewriter, HyDE and rerank are sequential.

Minimum Kubernetes manifests are already developed in the vLLM on Kubernetes post and the full orchestration piece fits the pattern of the multi-tenant H100 cluster.

On-premise hardware: what fits in an RTX 4090 vs a generic 4×H100 SXM configuration

The operational reality matters. Two typical profiles:

Modest configuration — 1× RTX 4090 (24 GB Ada Lovelace)

It comfortably serves:

bge-m3 (568M) for dense embedding of the query → ~1 GB VRAM
bge-reranker-v2-m3 (568M) cross-encoder → ~3.5 GB VRAM at batch=32
A light LLM for rewriter/HyDE (Qwen2.5-7B GPTQ-INT4) → ~5-6 GB VRAM
The main LLM (Llama-3.1-8B-Instruct FP8) → ~11-12 GB VRAM

Total VRAM used: ~21 GB. That leaves ~3 GB for the main LLM’s KV cache, enough for 4-8 concurrent users with ~4K token contexts. For an internal corporate assistant it is viable. If you need a larger context or more concurrency, you have to split things up: either the main LLM is served outside the 4090 (another GPU, another node), or you fall back on quantising the reranker (not advisable: you lose the nDCG points that justify the cost of adding it).

Generic configuration — 4×H100 SXM (320 GB total, NVLink)

It comfortably serves everything above multiplied by 30-50× in throughput and opens the door to:

A large main LLM (Llama 3.3 70B FP8 or Qwen2.5-72B FP8) on 2 GPUs with tensor parallelism
Large rerankers (bge-reranker-v2-gemma 2B) with quality ~2 points higher
Late interaction (ColBERT-v2) on one of the nodes for critical cases
Several light LLMs in parallel for rewriter, HyDE and the evals judge

The practical rule: the reranker is one of the most efficient things you can put on a GPU. The VRAM used is modest, the throughput is high, and the quality improvement per euro of extra hardware is among the best in the RAG stack.

The seven traps that kill retrieval

Trap 1 — Top-k to the LLM with no reranker. Dense retrieval returns a top-5 and those 5 chunks go straight to the LLM. Without a reranker, the order of the 5 is the cosine similarity order, which is not the real relevance order. The LLM answers based much more on the first chunks than the last ones, and the nuance is lost. Symptom: confident answers with citations that are not the most relevant in the corpus.

Trap 2 — Dense-only with no BM25. Short queries and queries with internal jargon work badly, the demos look pretty, real traffic complains. Symptom: on QA with real support tickets, the system fails specifically on queries of fewer than 5 words or with product codes.

Trap 3 — Reranker badly calibrated on top-k. A layer 1 top-k that is too small (top-10) cuts before the reranker can do its job. A top-k that is too large (top-200) blows up latency without improving nDCG. The canonical sweet spot is top-30 to top-50 from the hybrid → top-5 to top-10 from the reranker. Symptom: the reranker’s nDCG@5 stuck no matter which reranker model you try.

Trap 4 — Embedder and reranker in different languages. The corpus is Spanish + Catalan, the embedder is all-MiniLM (English only), the reranker is bge-reranker-v2-m3 (multilingual). The broad layer retrieves badly because of the embedder, and the reranker has little to rerank. Symptom: Recall@50 < 80%, unrecoverable by changing the reranker.

Trap 5 — RRF with k=1. Someone reads the paper, copies the formula and sets k=1 “because they do not understand what it does”. The result: the first positions of the worst ranking (whichever it is) dominate. The fusion stops averaging and becomes “winner-takes-all”. Symptom: the final ranking is almost identical to that of just one of the retrievers.

Trap 6 — A query rewriter that changes the meaning. The LLM rewriter “improves” "how do I cancel" into "termination procedure for the premium service for corporate customers". It retrieves chunks about corporate customers when the user was a private individual. Over-specification by the rewriter is worse than no rewriter. Symptom: queries from “normal” users come back worse than the old ones. Mitigation: the rewriter must keep the user’s intent and only add context, never assume.

Trap 7 — No retrieval telemetry. The system serves, the user complains, you do not know whether the fault was the corpus, the retrieval, the reranker or the LLM. Without emitting traces with retrieved_chunks_ids, retrieved_chunks_scores, rerank_scores, query_rewritten_to, selected_top_k, debugging is theatre. Symptom: each incident takes days to investigate. The tracing with OTel and MCP piece covers the canonical pattern.

All seven are operational. Just as with the corpus, retrieval does not break because the maths is wrong: it breaks because the discipline relaxes. And as in Eval, the metrics can go up while the real experience gets worse, because the golden set accommodates itself to the system instead of the system accommodating itself to the golden set.

What we have not covered (upcoming posts)

Embedding model selection and fine-tuning: how to choose among the embedders on the MTEB leaderboard without falling into goodharting, when and how to fine-tune an embedder on your own domain with MNR loss or triplet loss, which synthetic dataset is generated with an LLM for training (GPL, InPars, Promptagator), and the gotchas of massive re-embedding when you change model.
Semantic cache for RAG: how to cache semantically similar queries to serve answers without going through retrieval or the LLM, GPTCache, MeanCache, and the cache’s precision/coverage trade-off. It saves 30-70% of the cost on workloads with repeated queries.
Multi-vector and ColBERT-v2 at scale: how PLAID and CITADEL indexes are designed to serve corpora of 100M+ chunks with late interaction without burning the memory budget.
RAG-specific eval — RAGAS deep dive: faithfulness, answer relevance, context precision, context recall, noise sensitivity. How the RAG-specific golden set is built (with chunks labelled as relevant/irrelevant) and which metrics correlate with real user satisfaction.
Function calling and tool-augmented retrieval: when the LLM decides which retriever to invoke (SQL for structured data, vector for unstructured, web search for real time), the ReAct pattern, handling tool errors.
Agentic retrieval loops: when a single retrieval pass is not enough and the agent iterates (planning, sub-queries, summary-and-refine). The latency / quality trade-off and the anti-patterns (infinite loops, diverging sub-queries).

References

BEIR: Thakur et al. (2021). “BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models.” NeurIPS 2021 Datasets and Benchmarks Track. https://github.com/beir-cellar/beir
SPLADE: Formal et al. (2021). “SPLADE: Sparse Lexical and Expansion Model for First Stage Ranking.” SIGIR 2021. https://arxiv.org/abs/2107.05720
ColBERT-v2: Santhanam et al. (2022). “ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction.” NAACL 2022. https://arxiv.org/abs/2112.01488
Reciprocal Rank Fusion: Cormack, Clarke, Buettcher (2009). “Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods.” SIGIR 2009. https://dl.acm.org/doi/10.1145/1571941.1572114
HyDE: Gao et al. (2022). “Precise Zero-Shot Dense Retrieval without Relevance Labels.” https://arxiv.org/abs/2212.10496
Step-back prompting: Zheng et al. (2023). “Take a Step Back: Evoking Reasoning via Abstraction in Large Language Models.” https://arxiv.org/abs/2310.06117
BGE-M3: Chen et al. (2024). “BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation.” https://arxiv.org/abs/2402.03216
MTEB leaderboard: https://huggingface.co/spaces/mteb/leaderboard — útil como punto de partida, peligroso como criterio único (overfitting al benchmark, dataset contamination).
TEI (Text Embeddings Inference): https://github.com/huggingface/text-embeddings-inference — el motor de serving de HuggingFace para embedders y rerankers de tamaño pequeño-mediano. Endpoint OpenAI-compatible.
Qdrant hybrid search: https://qdrant.tech/documentation/concepts/hybrid-queries/ — implementación nativa de RRF y multi-vector queries.
RAGatouille: https://github.com/AnswerDotAI/RAGatouille — wrapper Python para ColBERT-v2 que reduce drásticamente la curva de entrada.
Cohere Rerank 3: https://docs.cohere.com/docs/rerank-2 — referencia técnica del reranker comercial multilingüe líder.

RAG corpus curation: the active librarian who decides what goes in, what goes out and what gets signed

Mon, 25 May 2026 11:00:00 +0200

This is the curation layer inside stage 1 (Data) of the six-stage LLMOps pipeline. It complements the other Data posts: dataset versioning for the four versionable artefacts, PostgreSQL + Qdrant ingestion in microservices for the outbox + CDC pattern, and RAG over Kafka and a datalake for streaming transport. This post is not about moving data: it is about what to do with data before you let a model read it.

TL;DR

A RAG system serving mediocre answers is rarely the retriever’s fault, nor the model’s. The root cause is usually the corpus: three nearly identical versions of the same PDF that make top-k return the same thing three times, an old manual nobody removed that contradicts the current one, a free-text field with customer numbers that the model quotes verbatim, a scanned PDF with dirty OCR that the chunker cut in the middle of a sentence. None of that is fixed by changing the model, the embedder, the reranker or the prompt. It is fixed by curating the corpus. This post takes apart the five operational layers of curation (schema-validated ingest, three-level deduplication, PII anonymisation measured with precision/recall, anti-contamination against the golden eval set, chunk→trace lineage), the minimum maths needed to avoid fooling yourself, the 2026 stack (Presidio, Unstructured, Argilla, LangChain text splitters, OpenLineage, Marquez, Great Expectations), the seven traps that reduce the stage to theatre, and the on-premise hardware to sustain all of it without sending anything sensitive to external APIs.

The analogy: the active librarian

A serious librarian does not accept books by the kilo. When someone offers a new volume:

They look at the spine, the ISBN and the stamp: is it legible? Is it catalogued correctly? Does it belong to a recognised collection? Without valid metadata, it does not get in. This is the schema check.
They check whether they already hold a copy: is it exactly the same book? Is it a later edition of the same one? Is it a translated version of something already on the shelves? If so, they decide explicitly what to do (replace, archive the old one, withdraw both from lending). This is dedup at its three levels.
They flag what is restricted: if the book contains identifiable personal data, some pages cannot be lent as they are, so they must be redacted, anonymised or moved to the reserved section. This is PII anonymisation.
They verify it is not this year’s final exam book: if it is, it stays out of the public collection until the syllabus changes, because if students can consult it the exam stops measuring what it is meant to measure. This is anti-contamination against the golden eval set.
They write it into the register: this book, this edition, this provenance, this date, this person who approved the entry. This is lineage.

If the book passes all five, it joins the collection. If it fails any of them, it goes to an auditable quarantine shelf with the reason for rejection. The difference between a good collection and a mediocre one is not size: it is how much discipline you apply across the five layers, every day, to every new book that arrives.

A RAG corpus is exactly that. The only differences are scale (thousands or millions of documents per month) and the fact that the “readers” are LLMs that cannot tell a duplicate from a reinforced truth, nor a PII value from a synthetic example, nor a contaminated fragment from an authentic one.

The four data artefacts and where the RAG corpus fits

Before going down to the five layers it is worth being clear about which corpus we are curating. The Data stage of the pipeline manages four distinct artefacts, each with its own discipline. The data versioning post lists them; here I reorder them from a curation perspective:

Artefact	Who consumes it	Dominant curation
Training dataset	Tune (fine-tuning the model or an adapter)	aggressive dedup + quality filters + label balancing
RAG corpus	Deploy (retrieval at request time)	the 5 layers in this post
Golden eval set	Eval (promotion gates)	strict hold-out + stratification + maintenance driven by incidents
Retrain enriched dataset	Retrain (closing the loop)	production feedback + human triage

The RAG corpus is the most volatile of the four and the one most exposed to the end user: every answer the system serves literally contains fragments of it. A duplicate in the training dataset degrades learning but ends up buried in the weights; a duplicate in the RAG corpus shows up in today’s answer and tomorrow’s. That justifies the extra discipline that follows.

Layer 1 — Schema-validated ingest

Every piece entering the corpus has to arrive with structured metadata validated against a schema. This is not bureaucracy: it is the only way to make the following layers (dedup, PII, lineage) work without friction.

The canonical pattern is to define a schema in JSON Schema or Pydantic that every document must satisfy:

class CorpusDocument(BaseModel):
 source_system: str # e.g. "confluence", "salesforce", "manual_pdf"
 source_id: str # unique ID in the source system
 version: str # document version (semver or date)
 language: str # ISO 639-1: "es", "en"
 title: str
 body: str
 captured_at: datetime
 captured_by: str # pipeline or human
 sensitivity: Literal["public", "internal", "restricted"]
 schema_version: str # version of the schema itself, not of the document

Any piece that fails this contract is rejected at ingest and never reaches the following layers. Validation is done with Great Expectations (declarative suites), Pandera (more pythonic, integrates with pandas) or Soda (oriented towards continuous data quality). The choice is a matter of style; what matters is:

Validation suites live in code and are versioned with the pipeline, not in a notebook off to one side.
Rejection generates an auditable event (quarantine) with the specific schema failure as its reason, not a log line lost in stdout.
The schema itself is versioned. When it changes, previous documents are reprocessed or explicit backward compatibility is maintained.

The RAG over Kafka post covers the Schema Registry pattern (Confluent Schema Registry or Apicurio) that materialises this in streaming: every message on the topic is validated against the registered schema before propagating downstream. For batch or pull, Great Expectations is the equivalent.

Common trap: leaving the body field free with no further validation. You have to tighten it up with minimum/maximum length (a PDF that yields 12 characters after extraction is almost certainly broken), valid encoding (UTF-8 with no control characters), proportion of alphanumeric characters (dirty OCR returns a soup of symbols). These are simple rules that filter 80% of the noise with no need for AI.

Layer 2 — Three-level deduplication

The most expensive and most silent mistake in a RAG corpus is the duplicate. A document that appears three times in the corpus makes retrieval’s top-k return it three times, wasting two slots and reinforcing a single source. The LLM reads it as if three independent sources agreed, when in fact it is the same thing repeated.

Deduplication is done at three levels, in this order by cost:

Level A — Exact dedup (SHA-256 hash)

Compute the hash of the normalised content (trim, lower-case where applicable, remove redundant whitespace) and compare it against an index of already ingested hashes. On a match, discard or replace. Cost: $O(1)$ per document. It catches literal duplicates (the same PDF uploaded twice, two byte-for-byte copies of the same HTML).

Level B — Near-duplicate (MinHash + LSH)

Nearly identical documents with minor differences (a different header, an updated date, one version in Castilian and another in Galician with minimal changes). The canonical algorithm is MinHash with Locality-Sensitive Hashing (LSH), which approximates Jaccard similarity over shingles of k tokens. For n documents, comparing all against all is $O(n^2)$, which is unworkable for large corpora. LSH cuts the cost to $O(n)$ most-likely buckets.

A typical threshold is Jaccard ≥ 0.80 over shingles of 5 tokens. The standard libraries are datasketch (Python, MIT) or dedup (Python, MIT). A numerical example: for 1 M short documents (300 tokens each), datasketch.MinHashLSH with 128 permutations and threshold 0.8 takes about 2 GB of RAM and processes the full corpus in around 30 minutes on a modern CPU. The fraction of duplicates found in a real enterprise corpus is usually between 5% and 25%; removing them cuts storage and improves retrieval quality at the same time.

Level C — Semantic dedup (cosine over embeddings)

Documents saying the same thing in different words, such as paraphrases, translations or rewritten versions, are not caught by MinHash. This is where semantic similarity comes in: compute the embedding of each document and compare the cosine between pairs.

The problem is quadratic cost: for n documents, computing every pair is $O(n^2)$. For n = 1 M and 768-dimensional embeddings (a typical model such as BAAI/bge-base-en-v1.5), that is 5×10^11 dot products, which is unworkable. The solution is the same idea as LSH but over dense vectors: HNSW (Hierarchical Navigable Small World) or IVF (Inverted File) to build an approximate search index. For each new document you run a k-NN query against the index and examine only the k nearest neighbours.

A sensible threshold for treating something as a semantic duplicate: cosine ≥ 0.95. Below 0.95 the documents are related but distinct; above it, they are almost always the same information rewritten. The exact threshold is calibrated by observing precision/recall over a human-annotated sample; 100 pairs confirmed by a reviewer is reasonable for setting it.

A numerical example: with qdrant or pgvector as the HNSW index and k=10 neighbours per query, deduplicating 1 M documents against the existing corpus takes on the order of 2-4 hours on an RTX 4090 (including embedding computation). If the embedder is self-hosted with vLLM, the cost per token is negligible against compute time.

Policy on what to do with a duplicate

Detection is not enough, you have to decide. Three common policies, in order of complexity:

Drop: discard the more recent one, keep the older one. Simple, with no extra lineage.
Replace: discard the old one, index the new one. More volatility but it reflects the update.
Merge with provenance: mark the new one as a “shadow” of the old one, keep both in lineage but index only one. Better for regulated audit.

The policy has to be explicit and applied uniformly, not an ad-hoc decision per document.

Layer 3 — PII anonymisation with measured precision/recall

This layer is the one that most easily turns into theatre. The typical mistake: install Presidio, run it over the corpus, assume the output is clean. Without measuring the detector’s precision and recall against an annotated golden set, you know nothing.

A PII detector can fail in two ways:

False negative (low recall): it does not detect a national ID written as “12345678-A” because your model was trained on the 12345678A format without a hyphen. The RAG serves personal data unredacted.
False positive (low precision): it redacts a number from a configuration manual thinking it is a phone number. The RAG loses useful information.

Both are problems; regulation (GDPR, ENS, NIS2) penalises the first, and user experience degrades with the second. The acceptable ratio depends on the domain. In medical data, practically zero false negatives is non-negotiable; in internal technical documentation you can tolerate more recall in exchange for less precision.

The standard metric is F1 over an annotated golden set:

\[ \text{precision} = \frac{TP}{TP + FP}, \quad \text{recall} = \frac{TP}{TP + FN}, \quad F_1 = 2 \cdot \frac{\text{precision} \cdot \text{recall}}{\text{precision} + \text{recall}} \]

To build the PII golden set, annotate around 200 documents by hand with every entity marked (national ID, IBAN, email, phone, address, personal name). Then run the detector and compute the metrics per category, not just aggregated, because a global F1 of 0.90 can hide a recall of 0.55 on IBANs.

2026 stack for this layer:

Microsoft Presidio (MIT, Microsoft): the most complete OSS option. Configurable detectors, recognises around 50 entities by default, extensible with your own regex patterns or with fine-tuned NER models.
spaCy NER (MIT, Explosion AI): a base for custom detectors; useful when Presidio does not cover a domain entity.
Llama Guard 4 (Llama Community License, Meta): a safety classifier that also detects PII in one pass, an option when you already have a GPU for inference and prefer a single pass.
DataFog (Apache 2.0): a more recent alternative, specialised in streaming pipelines.

Recommended hybrid pattern: Presidio for rule-based + regex detection (fast, deterministic) → Llama Guard as a second pass over what Presidio did not flag (an ensemble that raises recall without killing throughput). This is measured and reported as aggregate and per-category F1 on every release of the detector.

Common fallacy: trusting that a detector with F1 0.95 “is very good”. If you have 1 M documents and each contains on average 1 PII entity, F1 0.95 means 50,000 mishandled entities (between false positives and negatives). With sensitive data you have to design so that false negatives go to human quarantine, not to the public corpus.

Layer 4 — Anti-contamination against the golden eval set

If the RAG corpus contains fragments of the golden eval set, Eval metrics measure memorisation. The model returns the exact answer because it has it literally in its context, not because it generalised anything. Deploy then promotes models that shine in the exam and fail in production.

This layer is the easiest to implement and the easiest to forget:

The golden eval set has its own versioned hash.
Before indexing any new document into the RAG corpus, run a token-by-token overlap check (or by shingles, similar to MinHash) against the golden set.
If overlap exceeds a threshold (typically ≥ 30% of 5-token n-grams), the document is not indexed. It stays in quarantine flagged as “contamination risk vs golden_v12”.
A human reviews the rejections. Sometimes they are false positives (a short quotation, a boilerplate sentence). Sometimes they are real contamination that a supplier introduced without realising.

The deeper reason: the RAG corpus and the golden set are enemies by design. The golden set measures how well the system generalises to questions it has not seen. If those questions are in the RAG, the system “sees” them on every query. The metric stops measuring generalisation.

This check is computationally trivial, a hash join over n-grams. The difficulty is keeping it up: every time the golden set changes (monthly or quarterly), the full corpus has to be re-validated against the new golden set. Without that discipline, contamination sneaks in through the back door when someone updates the golden set with real cases that the RAG was already serving.

Layer 5 — End-to-end lineage: from document to trace

The last layer is the one that closes the auditable chain. Every chunk indexed in the vector store carries metadata that lets you answer the forensic question:

“The system generated this answer on 14 March at 16:23. Which exact document did the quoted fragment come from? When did that document enter the corpus? Which version of the embedder processed it? Who approved its ingest?”

Without lineage, that question is unanswerable. With lineage done properly, it is four queries.

The canonical pattern:

Every indexed chunk carries in its metadata: source_system, source_id, document_version, chunk_index, embedder_version, dataset_hash, ingested_at, ingested_by, schema_version.
Every RAG answer in production emits a trace span that includes the retrieved chunk_id values.
The central tracing system (Langfuse, Phoenix or OpenLLMetry) joins chunk_id → chunk metadata → document metadata → corpus dataset_hash → embedder version → and so on.

The tools that standardise this glue are OpenLineage (Apache 2.0, LF AI & Data) and Marquez (Apache 2.0, its server implementation). They define a lineage event schema that interoperates between systems; an ingest job emits an event “produced corpus_v12.3 from source X with embedder bge-base-v1.5”; a retrieval job emits “consumed corpus_v12.3 with query Q produced response R”. The graph is reconstructed automatically.

This layer is the only way to pass real audits under regulations such as the EU AI Act, GDPR or ENS, where traceability of which data went into which answer is a requirement, not an option. Without it, the answer “we do not know which document this came from” is not acceptable, and it is the default answer if lineage is not built from day one.

The maths that matter

Beyond dedup thresholds and PII F1 scores, there are three mathematical pieces that any serious team ends up using.

Chunk size vs retrieval quality. Chunk size affects retrieval quality in a non-monotonic way: chunks that are too small fragment ideas (retrieval returns a piece without context), and chunks that are too large dilute the signal (the embedding mixes several topics and similarity drops). The empirical sweet spot for technical text in 2026 is between 256 and 768 tokens per chunk, with 15-25% overlap between contiguous chunks to preserve continuity.

Numerically, for a corpus of 1 M documents with a mean length of 2,000 tokens, chunking at 512 tokens with overlap 100 gives $\frac{2000}{512 - 100} \approx 5$ chunks per document, a total of roughly 5 M indexed chunks. With 768-dimensional embeddings in float32, that occupies $5 \cdot 10^6 \cdot 768 \cdot 4 \approx 15$ GB of vector memory, manageable in any modern vector store.

PII golden set coverage. To know whether the annotated PII golden set is representative enough, compute the proportion of categories covered: if your golden set of 200 documents has 5 IBAN examples and production has 12,000 IBANs per day, the measured F1 on IBANs is statistical noise. Rule of thumb: a minimum of 30 examples per category for per-category metrics to mean anything.

Cost of re-embedding when rotating the model. Changing the embedder invalidates the whole index. For a corpus of 5 M chunks with a model such as BAAI/bge-base-en-v1.5 (768 dim, around 110 M parameters) served in vLLM on 1× H100, throughput is on the order of 8,000-15,000 chunks/second. Full re-embedding: about 5-10 minutes. For a larger embedder (bge-large, 1024 dim, around 335 M parameters), a factor of 3× worse, around 15-30 minutes. The bottleneck is usually vector store I/O, not GPU compute. The dual-index pattern, keeping the old index serving while the new one is built and doing an atomic swap at the end, avoids downtime and allows rollback.

Applied to typical on-premise hardware

For an on-premise deployment that keeps all curation in-house without sending data to external APIs:

RTX 4090 (24 GB): covers layer 1 (the schema check with Great Expectations is CPU-bound), layer 2 levels A and B (hash + MinHash are CPU-bound), and layer 2 level C semantic dedup with a bge-base-class embedder (8-15k chunks/s, enough for corpora of up to 5-10 M chunks in hours). For Presidio in NER mode (layer 3) it runs comfortably. It is the sensible GPU for the whole curation pipeline on a mid-size corpus.
Generic 4×H100 SXM configuration (320 GB total, NVLink): only needed if the corpus exceeds around 50 M chunks or if you want frequent re-embeddings with large models (bge-large, e5-mistral). In practice, two GPUs serve the embedder at TP=2 with throughput above 50k chunks/s, and the other two go to the PII judge (Llama Guard 4) or to serving the main inference model. Capacity for corpora of hundreds of millions of chunks.

The stubborn arithmetic: with a 4090, corpus curation is an overnight job; with 4×H100, it is minutes. The decision depends on corpus size and on how often you rotate the embedder or the PII rules.

The seven traps that kill this stage

Trap 1 — No validated schema at ingest. Malformed documents reach the chunker, the chunker slices them into nonsense, garbage embeddings enter the index. The RAG answer quotes incoherent text and nobody knows why.

Trap 2 — Dedup only at exact hash level. The corpus fills up with paraphrases and translations of the same document. Retrieval’s top-k returns the same source 3 times. The LLM reads it as three confirmations.

Trap 3 — PII detector without precision/recall measurement. Everyone assumes Presidio “works”. IBANs in non-standard formats slip through. The RAG serves personal data.

Trap 4 — Golden eval set contaminated with the corpus. Eval metrics measure memorisation. Promotions approve models that fail in real production.

Trap 5 — No lineage down to the chunk. The question “where did this quotation come from?” has no answer. Regulatory audit fails. Incidents cannot be investigated.

Trap 6 — Maintenance as a one-off event. The corpus is curated once when the system is initialised, and afterwards everyone assumes it is fine. Six months later the documents are out of date, the new PII rules are not applied retrospectively, dedup has not been re-run after adding new sources. The corpus degrades silently.

Trap 7 — Quarantine without human review. Rejected documents go into a table nobody looks at. False positives pile up, real contamination cases go uninvestigated, the team’s trust in curation erodes and the pressure to “relax the thresholds” begins.

All seven are operational, not technical. Corpus curation does not break because of an algorithm bug: it breaks because discipline slackens. It is the exact equivalent of the kind of degradation that kills Eval suites, and in both cases the symptom is the same: the metrics improve or hold steady while the real experience gets worse.

What we have not covered (upcoming posts)

Vector store versioning proper: an embedding index is not versioned like a raw dataset because it depends on the embedding model. Changing the embedder rewrites the whole index. It is a different animal with its own patterns (index branching, selective reembedding, recall-aware ANN parameters).
Streaming corpus updates with CDC: when the corpus has to be updated in near-real time from an OLTP system. The Postgres + Qdrant ingestion post covers the mechanics; still pending is the pattern for selectively invalidating chunks that depend on deleted rows.
Multi-tenant corpus isolation: how to set up a shared corpus versus one with per-tenant namespaces, with ACLs over individual chunks. Especially relevant for multi-customer RAG under data sovereignty constraints.
Federated corpus: corpora distributed across silos that the system queries without centralising the content. An emerging pattern for companies with several sites and cross-border restrictions.
Reranking-aware curation: how curation discipline changes when there is a reranker (Cohere Rerank, ColBERTv2, BGE-Reranker) reordering the top-k after retrieval. Some duplicates you would tolerate without a reranker become intolerable when the reranker pushes them up the ranking.

References

Presidio: https://microsoft.github.io/presidio/ — official docs, list of supported entities, guide to extending it with custom NER.
OpenLineage: https://openlineage.io/ — event schema spec and libraries per language.
Marquez: https://marquezproject.ai/ — server implementation of OpenLineage.
datasketch (MinHash + LSH): https://ekzhu.com/datasketch/ — the reference Python library for near-duplicate deduplication at scale.
Great Expectations: https://docs.greatexpectations.io/ — declarative data quality suites.
Unstructured: https://docs.unstructured.io/ — parsing and normalisation of heterogeneous documents (PDF, HTML, DOCX, eml) before chunking.
Argilla: https://docs.argilla.io/ — human annotation UI for building the PII golden set and other calibration sets.
Llama Guard 4: Meta technical paper, multimodal safety classifier — useful as a second PII detection layer.
GDPR, EU AI Act, ENS, NIS2 — the regulatory frameworks whose compliance depends, in practice, on the discipline of layers 3 (PII) and 5 (lineage). The CEN/CENELEC technical standards for conformity assessment of GenAI systems under the EU AI Act are still pending final publication.

Evals for LLMs: the layer after tracing that decides whether your model performs or only looks like it does

Mon, 25 May 2026 07:30:00 +0200

This is stage 3 of the LLMOps pipeline. If you arrive without context on the full journey, the six-stage LLMOps pipeline describes where Eval sits between Tune and Deploy, and the anatomy of a production LLM request shows a real eval blocking the promotion of an adapter.

TL;DR

Tracing is not evaluation. Tracing tells you what happened; eval tells you whether what happened is any good. The two layers live in tools that overlap visually (Langfuse does both), but the discipline is different: tracing is continuous capture over real traffic; eval is controlled execution against a stable dataset, with metrics that have to fail CI if they drop below a threshold. Without eval, the Tune → Deploy loop closes blind: adapter v8 goes to production because the engineer “saw it answering well” on five examples. With eval done properly, v8 only gets in if it beats v7 across a battery of 500 curated cases, evaluated by a mix of heuristics, embeddings, an LLM judge calibrated against humans, and a sample of real traffic with human annotation. This post takes apart the mechanism, the mathematics that keep you honest, the real tools in 2026 and the pitfalls that turn it into theatre.

The analogy: the academic examination board

A doctoral candidate defends a thesis. They do not defend it before a single distracted professor: they defend it before a mixed board, with a syllabus fixed in advance (not improvised on the day), and with an explicit pass mark that separates pass from fail. The board is not a single expert: it is a panel combining fast readers (the heuristics: does it have the requested format? does it include the citation?), semantic reviewers (the embeddings: does it resemble the expected answer?), an external assessor trained on the subject (the LLM judge: is it faithful to the context, is it relevant, does it sound coherent?), and human jury members on a sample of the most sensitive cases. If the candidate does not reach the mark, they are not promoted: they go back and prepare the defence again.

That analogy has three edges worth holding on to from the first minute:

The syllabus is fixed, but it is actively updated when a new topic appears in the real world that the candidate must know. If it is not updated, the exam measures something increasingly distant from what happens outside.
The board has to be trained: an LLM judge not calibrated against humans is an examiner making up the criteria.
The pass mark is published beforehand: it is not decided after seeing the result, because then it is not a pass mark, it is a justification.

These three ideas run through the rest of the post. Every tool and every piece of mathematics that follows is, at bottom, a way of operationalising them.

The mechanism itself: four layers of evaluators

An eval suite in 2026 is made up of four layers that coexist. None replaces the others; each one measures what the rest cannot measure well, and leaves unmeasured what the rest do measure well.

Layer 1 — Deterministic heuristics. Rules that return true or false with no ambiguity: the output matches a regex, contains a specific entity, does not exceed a length, follows a valid JSON schema, respects a requested format (markdown, tool_call function, mandatory citation). They are cheap, extremely fast, need neither judge nor embeddings, and catch the most frequent class of bug: the model answered in the wrong format. Their limit is obvious, in that they do not know whether the answer is correct, only whether it is well formed.

Layer 2 — Semantic metrics with embeddings. They compare the model’s output with an expected answer by computing cosine similarity between their embeddings, or by measuring whether a claim in the output is entailed by the retrieved context. They are cheap, deterministic given the embedding model, and very useful for detecting answers that drift in meaning but not in form. Their limit is equally clear: two answers can have high cosine similarity and say the opposite of each other (“the customer can cancel at any time” vs “the customer cannot cancel at any time” share 80% of their tokens).

Layer 3 — LLM-as-judge. A model, or an ensemble, evaluates the output of the model under test with a prompt designed to produce a score on a rubric. The canonical methods in 2026 are G-Eval (chain-of-thought prompting with a calibrated numeric score), Prometheus (open-source judge trained specifically for evals, reporting 0.897 correlation with humans in its v2.5 release from late 2025), and panel-of-judges setups that average votes from three heterogeneous models to reduce bias. This layer captures nuances the previous two cannot see: is it faithful to the context? is it useful? is it safe? is it complete? Its limit is cost and the need for calibration, treated as a whole section below.

Layer 4 — Humans. Trained annotators who evaluate a sample of the eval set, not all of it. They are the only layer with ultimate authority over the rubric: the LLM judge is calibrated against them, not the other way round. They are expensive (≈ 0.50–2.00 € per annotated sample when the domain is technical) and slow (a competent annotator produces 60–120 quality annotations per working day). The mistake teams make again and again is dropping this layer “because we have an LLM judge”; without humans there is no calibration, and without calibration the judge measures whatever it feels like.

Normal operation of an eval gate combines all four: heuristics remove malformed outputs before spending judge tokens, embeddings filter out the manifestly irrelevant, the judge scores the rest, and humans annotate a sample every N runs to keep the judge calibrated.

The golden dataset: a versioned syllabus

The golden dataset is the most undervalued artefact in the pipeline. It is the exam. If it is badly built, everything else, the best-calibrated judge in the world, the strictest gates, the fastest suite, measures noise. Covered at the top level in the data versioning post as one of the four artefacts to version differently; here we go into detail from the Eval perspective.

The three properties a golden dataset has to satisfy are:

Stratified representativeness. The dataset has to cover the system’s real input space in proportions that reflect production. If 30% of real traffic is in German, 30% of the golden set has to be in German; if 12% of questions are about subscription cancellation, that category cannot be 60% of the eval set just because it was easy to annotate. Stratification stays auditable: each example carries tags (lang=de, category=cancellation, tenant_type=enterprise, difficulty=hard) and the suite reports metrics per segment, not just the aggregate.

Strict holdout, no contamination with training. This rule is so obvious that almost every team believes it is following it, and almost every team breaks it without noticing. If the golden eval set gets mixed with the fine-tuning dataset, because somebody did a badly built random_split, because a purchased dataset is used by other vendors for training, because the LLM judge saw it during pretraining, then the metric stops measuring generalisation and starts measuring memorisation. The eval set hash is versioned separately (covered in the continuous fine-tuning post) and a routine leakage check is run: if an example in the golden set matches one in training token for token, it raises an alert.

Reasoned sample size, not aspirational. How many examples are needed? There is a mathematical minimum for telling two models apart with confidence. If model A is right 80% of the time and model B 85% on the same set, the 95% confidence interval for that difference of proportions (unpaired) is:

\[ \Delta p \pm 1{.}96 \cdot \sqrt{\frac{p_A(1-p_A) + p_B(1-p_B)}{n}} \]

To distinguish 80% from 85% with 95% confidence (an interval that does not cross zero), you need n ≈ 700 examples. To distinguish 90% from 91% under the same criterion, the calculation gives n ≈ 6,500. A “we have 50 examples in the golden set and we see adapter v8 scoring 90%” means nothing statistically: the confidence interval is ±8 points. The field’s practical rule in 2026 is a minimum of 300 examples to detect coarse differences, ideally 500–1,500 if you want to detect fine improvements, and to start by stratifying properly before obsessing over sample size.

On top of this comes active maintenance: the golden set is enriched with production incidents (covered in detail in the retrain post), so that every real complaint that ends in a bug becomes a curated example the next deploy candidate will have to pass. The golden set is not static: it is a living record of errors the system has already made and must not make again.

LLM-as-judge: how you calibrate an examiner

Layer 3 is the one most teams misuse. The typical mistake is: “we use GPT-4 as judge because it is the most capable”. The judge is not chosen for nominal capability; it is chosen for agreement with humans on the specific rubric you are measuring. A judge with 60% agreement is useless even if it is GPT-5; a Prometheus 7B fine-tuned for your domain with 88% agreement is worth more.

The standard metric for measuring agreement between two annotators (human vs judge, or two humans with each other) is Cohen’s kappa, which corrects for the agreement expected by chance:

\[ \kappa = \frac{p_o - p_e}{1 - p_e} \]

where $p_o$ is the observed proportion of agreement and $p_e$ is the proportion expected by chance under each annotator’s marginal distributions. The accepted interpretations in the literature are:

κ < 0.40: poor agreement. The judge says whatever it likes.
κ ∈ [0.40, 0.60]: moderate. Acceptable for coarse signals (is it toxic?), dreadful for nuance (is it faithful to the context?).
κ ∈ [0.60, 0.80]: substantial. Useful in production for most metrics.
κ > 0.80: almost perfect. The judge can be treated as a substitute for the human for that specific kind of judgement.

Numerically, consider a binary rubric (faithful / not faithful) over 200 examples annotated by a human and by a judge. If the human said “faithful” in 150 cases and the judge in 140, and they agree on 175 of the 200, then $p_o = 0{.}875$; the marginals are $p_h = 0{.}75$, $p_j = 0{.}70$, and $p_e = 0{.}75 \cdot 0{.}70 + 0{.}25 \cdot 0{.}30 = 0{.}600$. Kappa comes out at 0.6875: substantial but not excellent, usable, with an eye kept on the rubric.

Calibrating the judge means an explicit process:

Build a calibration set — 100–300 examples annotated by trained humans, with written guidelines. Inter-annotator kappa between the humans is also measured; if the humans themselves cannot agree, the rubric is badly written before you even start talking about a judge.
Iterate the judge prompt until judge agreement with the humans passes the accepted threshold (typically κ ≥ 0.7 for sensitive metrics).
Pin the judge version (claude-3-5-sonnet-20251022, gpt-4o-2024-11, prometheus-2-7b@sha256:…): any change invalidates the calibration.
Recalibrate periodically — every time the judge, the judge prompt, or the rubric changes. The frequency the field recommends in 2026 is quarterly at minimum, monthly if the rubric is new.
Persist everything in lineage — a score of “faithfulness 0.87” with no traceability of which judge, which prompt, which calibration set and which human validated it, is decorative.

The prompt versioning post covers how judge prompt versioning is materialised. The practical consequence is that the judge is versioned like any other model: your eval suite has judge_id = prometheus-2.5@v3 just as it has adapter_id = customer_support_v7.

The two cadences: CI gate and platform regression

Eval suites live in two places and run on two different cadences. Teams that conflate them turn one of the two into theatre.

CI gate (pre-merge, blocking). It runs on every pull request that modifies prompts, adapters, RAG configuration, or any artefact that could move the model’s output. It runs against the golden dataset versioned at the hash that is on main. The gate fails the merge if:

the critical metric drops more than X percentage points in absolute terms against the baseline (typically X = 2);
some safety metric (toxicity, PII leakage) crosses a hard threshold (typically tox > 0.02);
some strategic segment (a language, an enterprise-type tenant) drops more than Y points even if the aggregate improves.

This cadence has to be fast (ideally < 10 minutes over 500 examples) and cheap (LLM judge in batch mode, cached embeddings, heuristics run locally). The CI gate is not exhaustive: it is the low-latency line of defence.

Platform regression (post-deploy, continuous). It runs on a schedule (typically nightly or weekly) over sampled production traffic, not over the static golden set. It detects drift: the model has not changed, the golden set has not changed, but the users have, and quality over real traffic drops. This cadence is more expensive (judge over thousands of samples, human annotation over hundreds), tolerates latencies of hours, and its main consumer is not CI but the observability dashboard and the humans on the product team who decide whether to open a retrain cycle.

Both cadences persist results in the same store (Langfuse, MLflow, or equivalent) and connect them by model_id, prompt_id, dataset_hash and judge_id. Without that glue of identifiers, the metric that passes CI cannot be correlated with the one that fails in production three weeks later.

The minimum mathematics that matter

Beyond kappa and the interval for the difference of proportions, both already covered above, there are three more mathematical pieces that any team running evals seriously ends up using.

Confidence interval for a continuous metric. If your metric is a continuous score (faithfulness ∈ [0, 1]) and you measure the sample mean $\bar{x}$ over n examples with standard deviation s, the 95% confidence interval for the population mean is:

\[ \bar{x} \pm 1{.}96 \cdot \frac{s}{\sqrt{n}} \]

For n = 300 and s ≈ 0.2 (typical of a 0-1 score with non-degenerate variance), the margin is ±0.023. This means that differences below 2 hundredths cannot be distinguished from noise at that sample size. If your team is chasing “+0.5 pp” improvements over 100 examples, it is optimising noise.

Judge cost as a function of sample size and rubric. The cost of one eval pass with LLM-as-judge over n examples, with m metrics evaluated in a single call per example, and per-token price $c_{in}, c_{out}$ on the judge model, is:

\[ C \approx n \cdot (t_{in} \cdot c_{in} + t_{out} \cdot c_{out}) \]

where $t_{in}$ is the number of input tokens (includes context, output of the model under test, full rubric) and $t_{out}$ the output tokens (includes the judge’s CoT + score). For n = 500, $t_{in}$ ≈ 4,000, $t_{out}$ ≈ 300, a GPT-4o judge at May 2026 prices, one pass costs on the order of 8–15 USD per suite. If the pass fires on every PR and there are 30 PRs/day, that is 240–450 USD/day on CI gates alone. Multiplied by continuous regression, teams that do not control this spend four figures a month on judge tokens without realising. The canonical mitigation is a mix of layers: heuristics and embeddings filter first, the judge is only invoked on what the cheap layers cannot resolve, and for platform regression a self-hosted open-source judge is used (Prometheus 7B on your own GPU plane) instead of a commercial model.

Distinction between aggregate metric and per-segment metric. The classic eval fallacy is the hidden mean. If your suite reports faithfulness = 0.87 and the team reads it as “up 2 points against the previous adapter”, this may be what is happening: the new adapter gains 4 points in English (where 70% of the eval set is) and loses 6 points in German (where 30% is). The aggregate mean improves, the German experience gets worse. Any serious suite reports a breakdown by strategic segment (language, tenant type, question category, context length). The CI gate can also have per-segment thresholds, not just aggregate ones.

The 2026 stack: dominant tools

Tool	Main layer	Licence	Maintainer	When to pick it
DeepEval	CI gate	Apache 2.0	Confident AI	“Evals as pytest” — assertions in Python code, trivial integration with GitHub Actions. A reasonable default.
Promptfoo	CI gate	MIT	Promptfoo Inc.	Declarative YAML, matrix of prompts × providers × assertions, diff vs baseline. DevOps-friendly.
RAGAS	RAG-specific metrics	Apache 2.0	Exploding Gradients	Faithfulness, context relevancy, answer relevancy. The canonical piece if your system is RAG.
Inspect AI	Safety/capability evals	MIT	UK AI Safety Institute	Suite focused on safety and capability. Useful for regulatory gates under the EU AI Act.
Langfuse Evals	Platform regression	MIT (OSS) / EE	Langfuse GmbH	Integrated with tracing — datasets, runs and scores in the same UI as production traces.
MLflow GenAI Evals	Registry + evals	Apache 2.0	Databricks/LF AI	Good when you already have MLflow for classic models; “GenAI dashboard” since 3.10.
Phoenix (Arize)	Eval + visual drift	Elastic License 2.0	Arize AI	Focused on visual debugging of embeddings and drift; a complement to Langfuse, not a replacement.
Prometheus 2.5	Self-hosted OSS judge	Apache 2.0	KAIST + LG AI	Fine-tuned judge, high correlation with GPT-4 at zero cost per token when self-hosted.
G-Eval / JudgeLM	Prompting methods	— (techniques)	academic	Prompting frameworks for LLM-as-judge — they apply on top of any judge model.

The OSS catalogue by stage goes into detail entry by entry; the parallel OSS vs hyperscalers catalogue compares against Bedrock Evaluations, Vertex AI Eval Service and Azure AI Evaluation.

The canonical pattern in 2026 is hybrid: DeepEval or Promptfoo for CI gates + Langfuse Evals for platform regression + Prometheus 2.5 as a self-hosted judge + human annotation on Argilla or Label Studio for the calibration set. Swapping any of these pillars for equivalents (W&B Weave instead of Langfuse, Inspect instead of DeepEval) is style, not functionality; what matters is that the four functions are present and connected.

Applied to typical on-premise hardware

For an on-premise deployment that wants to avoid sending sensitive outputs to a commercial LLM judge (for data sovereignty, ENS, NIS2 or equivalents), the judge is hosted on your own GPU plane. Reference figures for May 2026, based on Prometheus 2.5 (Llama-3.1-8B fine-tuned as a judge) served on vLLM:

RTX 4090 (24 GB, Ada Lovelace): viable for small suites (< 200 examples) and for running the judge offline during development. Latency per evaluation ≈ 1.5–3 s with BF16; aggregate throughput on the order of 25–40 evaluations/min with batching. Useful for a developer’s local CI gates, not for platform regression.
Generic 4×H100 SXM configuration (320 GB total, NVLink): runs the judge in parallel at TP=2 across two GPUs, leaving two free to serve the model under test. Aggregate throughput on the order of 200–350 evaluations/min, a full 500-example suite in 2–3 min. This allows PR gates with no perceptible waiting and nightly platform regression over thousands of samples with no per-token cost.

The comparative cost arithmetic is stubborn: self-hosting Prometheus 2.5 pays off an H100 in roughly 6 months if the team fires ≥ 30 PRs/day with eval gates. Below that volume, the commercial judge still wins unless sovereignty is a requirement, and under ENS / NIS2 it is.

The seven pitfalls that kill this stage

Pitfall 1 — Aged golden dataset. It is not enriched with production incidents. After months it measures a world that no longer exists. The metric climbs comfortably while real users complain more.

Pitfall 2 — Contaminated or uncalibrated judge. The LLM judge evaluates with criteria it invents itself. Without a human calibration set as reference, there is no way to know whether its 0.89 is generous, severe or random.

Pitfall 3 — Insufficient sample size. A 50-example suite, 1-point differences the team treats as significant. The confidence interval is ±10 points. They are measuring noise and making real decisions on it.

Pitfall 4 — Runaway cost. A GPT-4 batch-mode judge fired on every PR over 1,000 examples, with no prior filtering by cheap layers. The eval bill overtakes production serving. It happens more often than people admit.

Pitfall 5 — Aggregate metric hiding segments. Global mean improves 2 points, German drops 6, enterprise tenants drop 3. With no explicit per-segment breakdown, the gate approves what it should not.

Pitfall 6 — Judge with a floating version. Judge model updated without recalibrating the rubric. Thresholds lose their statistical meaning. Last month’s regressions are not comparable with those from two months ago.

Pitfall 7 — Eval gate that is not enforced. The gate exists in the documentation but not in the real workflow: the suite takes 30 minutes, developers skip it with --no-verify, managers ask for one-off exceptions that become the norm. An eval gate without enforcement is ornament.

All seven are operational, not technical. The Eval layer does not break because the mathematics are wrong: it breaks because the discipline slackens. It is the same thing that happens with unit tests in any project that grows, except that here, without the discipline, the system improves its metrics while getting worse, and that makes the degradation invisible until it is already ungovernable.

What we have not covered (upcoming posts)

Guardrails and online safety: the inline eval layer that filters outputs in real time, not in CI. Conceptually a cousin of Eval, but with very different latency constraints. Covered in detail in the guardrails post: four lines of defence (input, retrieval, tool, output), OWASP LLM Top 10 (2025), OSS catalogue and deployment patterns.
Judge ensembles and agreement between multiple judges: how to reduce bias by combining three heterogeneous models as a panel, which aggregation function works (simple majority, trimmed mean, calibrated meta-judge).
Meta-eval: how you evaluate the suite itself. If the eval improvement from adapter v7 to v8 does not translate into a real improvement for the user, the suite is wrong, and that is also measured, by correlating eval metrics against product metrics.
Metamorphic testing: evaluating robustness against input perturbations (typos, paraphrasing, alternative language) as an extra gate. Beyond nominal agreement, it measures consistency.
Adversarial evals with red teaming: introducing prompt injection and jailbreak attacks as part of the gate.
Privacy in the judge: how to stop sensitive outputs from the model under test travelling to an external judge when regulation forbids it — homomorphic judge, judge in a TEE, or simply a self-hosted judge.

References

G-Eval: Liu et al., “G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment” — the reference paper for the chain-of-thought LLM judge method.
Prometheus: Kim et al., “Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models” — OSS judge with a reported 0.897 correlation vs humans.
RAGAS: Es et al., “RAGAS: Automated Evaluation of Retrieval Augmented Generation” — the paper that standardises faithfulness, context relevancy and answer relevancy.
Cohen’s kappa: Cohen, J. (1960). “A Coefficient of Agreement for Nominal Scales.” Educational and Psychological Measurement — the classic metric for inter-annotator agreement, still the operational reference.
DeepEval docs: https://docs.confident-ai.com/
Promptfoo docs: https://promptfoo.dev/docs/
Langfuse Evals: https://langfuse.com/docs/scores
Inspect AI: https://inspect.ai-safety-institute.org.uk/
EU AI Act, relevant articles on mandatory evaluation of high-risk systems — pending publication of the CEN/CENELEC technical standards on conformity assessment for GenAI.

The OSS catalogue for LLMOps in six stages: tool by tool, what each one does and when to choose it

Sat, 23 May 2026 07:30:00 +0200

TL;DR

For each of the six LLMOps stages (Data, Tune, Eval, Deploy, Observe, Retrain) and the two cross-cutting components (prompt + data versioning), the open source ecosystem has canonical pieces that this blog has been citing again and again. This post gathers them in one place with cards of roughly 150 words per core tool: what it does, how it differs from the alternatives inside the same bucket, its licence and governance model, and a typical gotcha that you only learn in production. Plus more alternatives as bullets, a decision matrix per stage according to the case (small / large corpus, single tenant / multi-tenant…), a diagram of the connected OSS stack and a master table of licences / EE offerings. The intention: that the reader closes the post knowing what is available, which company maintains it, what gap each piece fills, and when to choose it. This is not opinion: it is a curated catalogue.

You are here: every stage, but by OSS column

This post shares a map with the two previous ones in the series, the six stages and the two cross-cutting pieces are all active, but it zooms in on the open source column.

The analogy: the electrician’s toolbox

A professional electrician arrives at an installation with a box organised into compartments. Nothing is improvised: for every type of cable there is a specific wire stripper, for every screw a screwdriver of the exact gauge, for every measurement a multimeter and a clamp meter, for every connection the right terminal block or connector. The difference between a professional electrician and a handyman is not that one knows more theory, the handyman has often read the manuals, it is that the right tool is within arm’s reach and he knows when to use each one. The day the specific stripper is missing, improvising with a box cutter breaks the insulation, leaves a badly terminated cable, and the panel comes back under warranty two months later.

The OSS LLMOps stack works the same way. For every canonical problem, versioning a dataset, indexing a corpus for retrieval, serving tokens with dynamic batching, propagating trace_id end-to-end, managing prompts with a production label, orchestrating retraining pipelines, there is a canonical piece of the open source ecosystem that solves it, maintained by a serious community or foundation, with a clear licence and a well documented gotcha. The consultant who knows which tool to use for each job builds a robust system in weeks; the one who improvises with “whatever the team already knows” pays later in operations, usually once the system is carrying real load and any replacement is expensive.

This post opens the toolbox and shows each card. It is not a user manual, that is what the deep-dive posts linked at the end are for; it is the curated catalogue.

Diagram of the connected reference OSS stack

The catalogue makes sense once you see how the pieces connect into a single coherent architecture, the one this blog has been describing throughout the series. The boxes do not float; they talk to each other over stable contracts (HTTP, gRPC, OTel, Kafka, S3/MinIO API).

Solid arrows mark data / control flow; the dashed blue ones are OTel traces. The K8s plane holds everything up. The control plane at the bottom is where the retraining pipelines, the evals in CI, the versioned prompts and the lineage live. The data plane on the left feeds both the serving side (RAG, configs) and the control plane (datasets, lineage). The observability plane receives from serving and from everything else.

Now we go stage by stage. Each one opens with a context paragraph, then cards for the core tools (about 150 words each), bullets with relevant alternatives, and a specific decision matrix at the end.

Stage 1 — Data + cross-cutting Data versioning

The Data stage solves three distinct problems that beginners confuse: versioning datasets (so that (dataset_id, version, hash) exists and propagates), storing and serving the operational corpus (object store + vector index + structured text), and moving it between systems with CDC and stable schemas. Covered in detail in the posts on data versioning with DVC and lakeFS, PostgreSQL + Qdrant in ingestion and RAG over Kafka.

DVC (Data Version Control)

DVC puts datasets under version control with the same discipline that git applies to code. The .dvc pointers live in git (plain text, about 200 bytes per dataset), the bulk content lives in a remote object store (S3, MinIO, Azure Blob, GCS). Each dvc add computes a SHA-256 hash of the dataset, uploads it to the remote and stores the pointer. The key line: the dataset_hash becomes the luggage tag that travels to the trainer, to experiment tracking and to lineage. The same dataset retrained twice produces the same hash, and therefore reproducible experiments. DVC integrates with MLflow and W&B as an input artefact. Gotcha: it works well for datasets that change by replacement (I swap train.jsonl for a new version) and worse for datasets with thousands of small files that change individually. For that case, combine it with lakeFS. Licence Apache 2.0, maintained by Iterative.ai since 2017. There is DVC Studio (managed) and dvc data (pure CLI) on different planes.

lakeFS

lakeFS brings git semantics (branch, commit, merge, rollback) to an entire S3/MinIO/ADLS bucket. Where DVC versions individual files as pointers in git, lakeFS versions the whole bucket: you can branch the corpus, ingest new data into the branch, validate that it passes checks (recall@10 over golden queries for embeddings, completeness for a tabular corpus), and only then merge into main. It is the piece that makes continuous RAG safe: the production corpus is always on main, updates are tested on branches. It offers hooks (pre-merge, pre-commit) that fire automatic validations, and time-travel to reproduce the state of the bucket at a past date. Gotcha: the manifest overhead on huge buckets (hundreds of millions of objects) deserves sizing; lakeFS keeps metadata in its own Postgres, not in the bucket. Licence Apache 2.0, maintained by Treeverse since 2020. Managed offering: lakeFS Cloud.

MinIO

MinIO is the S3-compatible object store that fills the “S3 on-premise” gap without surprises. API identical to S3 (the AWS SDKs work by pointing them at a different endpoint), its own CLI client (mc), erasure-coded mode for fault tolerance, bucket-to-bucket replication, encryption at rest. It is the base on which the other data plane components are built: DVC remote, lakeFS underlying storage, Postgres snapshots, MLflow artefacts, eval datasets, saved models, distributed KV cache fabric. In small deployments it runs single-node multi-disk; in serious ones, distributed clusters. Gotcha: the licence changed to AGPLv3 in 2021 (it was Apache 2.0 before), which means distributing software connected to MinIO obliges you to open the code that connects to it. For internal on-premise use this is not a problem; for a vendor packaging MinIO inside a commercial product, it is. Maintained by MinIO Inc. with a SUBNET enterprise offering and a community fork called AIStor launched in 2025.

Qdrant

Qdrant is the OSS vector database most aligned with this blog’s “RAG corpus per tenant with strict ACLs” pattern. Written in Rust, it exposes a REST + gRPC API, indexes with HNSW + scalar/binary quantisation to cut memory, supports efficient payload filtering (it is not post-filtering: it integrates the filter into the HNSW search), and allows collections isolated per tenant. For the multi-tenant chatbot scenario, Qdrant is where the tenant_<id>_kb_v3 collections with strict ACL live. It scales well horizontally (sharding by payload) and vertically (millions of chunks on a node with 64GB RAM). Gotcha: binary quantisation is aggressive, it cuts VRAM by 32× but degrades recall by 10-20 %; enabling it without re-tuning the threshold breaks retrieval silently. Licence Apache 2.0, maintained by Qdrant Solutions GmbH (Germany). There is Qdrant Cloud (managed) and EU-only support for ENS cases.

PostgreSQL + pgvector

Postgres 18 with the pgvector extension is the stack’s “hidden vector database”: when the corpus is small (under a million embeddings) and Postgres is already in production for operational data, running a separate Qdrant is expensive operations. pgvector adds a vector(dim) type, HNSW and IVF indexes, and the operators <->, <#>, <=> for cosine, L2 and dot product. Combined with tsvector (Postgres full-text search) it allows hybrid search, dense + sparse, in a single SQL query. Version 0.8 (2025) introduced halfvec and bit support to reduce size by 4×-8×. Gotcha: HNSW in pgvector consumes a fair amount of RAM to build the index (roughly 2× the size of the embeddings) and blocks inserts during the build; in production you build on a secondary, promote it, and discard the primary. Licence PostgreSQL License (permissive BSD-style) for both the core and pgvector. Maintained by the PostgreSQL Development Group plus pgvector by Andrew Kane + Crunchy Data + Neon.

Apache Kafka + Debezium

Kafka is the event bus where “everything that happens in the company is a stream” becomes real. For LLMOps in production it serves two functions: CDC from source systems (Debezium captures changes in Postgres / MySQL / MongoDB and publishes them as topics) and LLM event buffer (every request, every piece of feedback, every eval result ends up in a topic with the trace_id propagated). As the post on RAG over Kafka explains, the RAG corpus is kept fresh by capturing changes from the CMS / source system as CDC, running the embedding in Flink streaming, and ingesting into Qdrant continuously. Gotcha: a badly sized Kafka with long retention plus multi-client topics turns into a fast disk sink; measuring throughput per topic and key cardinality before production is mandatory. Kafka licence Apache 2.0 (ASF project); Debezium Apache 2.0 (project incubated by Red Hat). Kafka-compatible drop-in alternative: Redpanda (BSL, restricted commercial use).

Apache Flink (brief mention)

Flink processes streams with sub-second latency and exactly-once semantics. On the LLM plane it is used to: run embeddings in streaming (over CDC topics), aggregate online metrics, materialise features for retraining. Licence Apache 2.0, ASF. Common alternative: Spark Structured Streaming (also ASF, micro-batch latency).

More options for Data, mentioned on the blog:

Ceph — object store for large clusters with geo-distributed replication. LGPL/Apache licence, Red Hat / IBM.
Milvus — C++ vector database, an alternative to Qdrant; better for corpora of billions. Apache 2.0, Zilliz.
Karapace — Confluent-compatible OSS Schema Registry. Apache 2.0, Aiven.
DataHub / Apache Atlas / OpenMetadata — catalogue + lineage. Apache 2.0, Acryl Data / ASF / Collate respectively.
OpenLineage — cross-system lineage event standard. Apache 2.0, Linux Foundation AI&Data.

Decision matrix — Data:

If your case is	Choose
Corpus < 1M embeddings, you already have Postgres	pgvector (one component fewer)
Corpus 1M-100M, multi-tenant with ACL	Qdrant (integrated filtering, ACLs per collection)
Corpus > 100M, aggressive sharding	Milvus (scales linearly better into the billions)
Training datasets + experiment tracking	DVC over MinIO + MLflow integration
RAG corpus with controlled releases	lakeFS over MinIO + pre-merge hooks
You want both	DVC + lakeFS as complements (the blog’s recommendation)

Stage 2 — Tune

The Tune stage produces a new model_id, model_version, typically a LoRA adapter over a stable base, with lineage back to the dataset and experiment tracking to reproduce it. Detail in the post on continuous fine-tuning.

HuggingFace Transformers + PEFT

transformers is the canonical library for loading and training models of the decoder-only family (Llama, Mistral, Qwen, Gemma…) and encoder-decoder. peft (Parameter-Efficient Fine-Tuning) is the complement that adds declarative support for LoRA, QLoRA, IA3 and assorted adapters. Together they form the mandatory core of the OSS Tune stack: any higher-level framework (Axolotl, LLaMA-Factory) uses them underneath. PEFT allows you to train an adapter of about 280 MB (order of magnitude) instead of a full model of about 140 GB, with a functionally equivalent result on most style / domain adjustment tasks. Gotcha: PEFT with a badly configured target_modules trains an adapter that covers only Q and V of the attention, leaving out key, output proj and MLP. The result looks trained but performs poorly; adding target_modules=["all-linear"] fixes it (at the cost of a larger adapter). Licence Apache 2.0, maintained by Hugging Face SAS (a French company); open governance model with active external maintainers.

bitsandbytes

bitsandbytes implements weight quantisation to 8-bit and 4-bit with NF4 for models loaded with transformers. It reduces the 140 GB of Llama 3 70B FP16 to about 40 GB in NF4, allowing QLoRA training on a single H100 80GB. The trick is that the weights stay quantised in memory while the sensitive computations (attention, gradient updates in the adapter) are done in FP16/BF16 with on-the-fly dequantisation. Ideal for fine-tuning on limited hardware and for serving with vLLM when you want to cut VRAM. Gotcha: NF4 quantisation is lossy; on small models (< 7B) the quality degradation is noticeable. For production serving of models under 7B, INT8 is preferred (more memory, less loss) or FP8 if the hardware supports it (the H100 does). Licence MIT, maintained by Tim Dettmers (originally at U. Washington, now with support from Anthropic and HuggingFace).

MLflow Tracking

MLflow is the reference OSS experiment tracking tool: each trainer run records parameters (lr, batch size, epochs, target_modules), metrics (loss curves, eval scores), artefacts (model, tokeniser, configs) and, crucially, input artefacts (dataset_id, dataset_hash, parent_run). The model registry associates each model_version with a reproducible run_id. The line of continuity between Tune and Deploy passes through here: the deployment reads the model to serve from the registry, with its lineage made explicit. MLflow 2.x integrates MLflow Prompts (prompt registry) and MLflow Tracing (OTel-compatible spans), cutting the number of components needed. Gotcha: the default backend store is SQLite, which works for personal experiments and breaks on a shared cluster. In production: Postgres as the backend store + MinIO/S3 as the artefact store. Licence Apache 2.0, maintained by LF AI & Data (donated by Databricks in 2020).

Axolotl

Axolotl wraps transformers + PEFT + bitsandbytes + DeepSpeed + FSDP in a declarative YAML configuration: instead of writing a script of about 300 lines to configure a fine-tuning run, you define a config.yml with base model, dataset path, LoRA config and training hyperparameters, and run it in one line. It supports Llama, Mistral, Qwen, Gemma, Phi… loads. It keeps compatibility with the HuggingFace Hub to download models and datasets, and with MLflow / W&B for tracking. It is the convenience framework this blog cites when it talks about “productive fine-tuning without reinventing the wheel”. Gotcha: the pace of change in the community is fast; a config.yml that worked six months ago can break with a current version because of internal refactors. Pinning the exact Axolotl version in the environment mitigates this. Licence Apache 2.0, maintained by the OpenAccess AI Collective (community-driven). A very similar alternative, more widely used in China: LLaMA-Factory (Apache 2.0, Beihang U.).

Ray Train

Ray Train scales fine-tuning to multiple nodes by distributing workers across a Ray cluster. While DeepSpeed and FSDP are intra-job parallelism (several GPUs collaborating on one job), Ray Train is the orchestration plane that builds the cluster, launches workers, manages checkpoints, recovers from node failures, and integrates with Slurm or Kubernetes. For training runs beyond 8 GPUs on changing clusters, Ray Train avoids the operational burden of “manually launching N torchrun processes with NCCL”. It combines with MLflow for tracking. Gotcha: the Ray learning curve is real; for a single node with 4-8 GPUs, torchrun or Hugging Face Accelerate are simpler. Ray Train shines when there are N changing nodes. Licence Apache 2.0, maintained by Anyscale Inc. (commercial backer) + community. A more K8s-native alternative: Kubeflow Training Operator (Apache 2.0, LF AI & Data).

More options for Tune:

DeepSpeed — ZeRO parallelism in 3 stages, mixed precision, CPU/NVMe offload. MIT, Microsoft.
FSDP (Fully Sharded Data Parallel) — native PyTorch parallelism, an alternative to DeepSpeed. BSD, Meta.
LLaMA-Factory — equivalent to Axolotl with a focus on the Llama family. Apache 2.0, Beihang University.

Decision matrix — Tune:

If your case is	Choose
Fine-tune on 1 GPU with 24GB (RTX 4090)	QLoRA with bitsandbytes NF4 + Axolotl
Fine-tune on 1 H100 80GB, models < 13B	LoRA bf16 + Axolotl
Fine-tune on 4-8 GPUs, single node	transformers + PEFT + Accelerate + MLflow
Multi-node fine-tune on a K8s cluster	Kubeflow Training Operator or Ray Train
Reproducible tracking is mandatory	MLflow + DVC input artefact
You want the minimum viable	Axolotl + MLflow

Stage 3 — Eval + Guardrails

Eval validates candidates pre and post promotion against a golden set with operational metrics; Guardrails runs safety online. Detailed in the posts on evals and guardrails.

DeepEval

DeepEval is the “pytest-style” OSS eval suite: you define tests with assertions over faithfulness, answer relevancy, contextual precision, hallucination rate, summarisation quality… and run them in CI. Each metric is an evaluator: some rule-based, others LLM-as-judge with auditable prompts. The philosophy is “evals as unit tests”: parameterisable by dataset, failable in CI, integrable with GitHub Actions. Gotcha: LLM-as-judge metrics vary between judge model versions, so if the judge moves up a version, the thresholds lose their earlier statistical meaning. Explicit pinning of the judge model in config plus periodic threshold recalibration is mandatory discipline. Licence Apache 2.0, maintained by Confident AI (a company); parallel commercial SaaS offering. Comparable: TruLens (MIT, TruEra) and G-Eval (academic).

RAGAS (RAG Assessment)

RAGAS specialises in evaluating RAG pipelines. It defines four canonical metrics: faithfulness (the answer is supported by the retrieved chunks), answer relevancy (the answer addresses the query), context precision (the retrieved chunks are relevant), context recall (all the relevant chunks were retrieved). Each metric is computed with LLM-as-judge over a dataset of (query, context, expected answer). For a RAG system, RAGAS is the evaluator that measures whether retrieval is aligned with generation. It integrates with Langfuse and MLflow to store results. Gotcha: RAGAS works well with golden sets of under 1000 examples; over huge golden sets the cost of the LLM judge per evaluation explodes, and the practice is to sample. Licence Apache 2.0, maintained by Exploding Gradients (the authors’ company).

Promptfoo

Promptfoo is the declarative, CI-oriented evaluator: in promptfooconfig.yaml you define a set of prompts and a set of assertions (contains text X, does not contain Y, faithfulness > 0.8, judge approves…), point at a provider (OpenAI compatible, vLLM, Ollama…), and promptfoo eval runs the matrix of prompts × providers × assertions, returns a diff against the baseline and fails CI if anything regresses. It is the most “DevOps-friendly” piece of the evals ecosystem: it integrates trivially with GitHub Actions, GitLab CI or Jenkins. Gotcha: assertion thresholds have to be calibrated with real data; starting at a default > 0.5 produces false positives that erode the team’s trust. Calibrate after the first week. Licence MIT, maintained by Promptfoo, Inc. (a company); a commercial Promptfoo Cloud SaaS offering exists but the OSS version is complete.

NeMo Guardrails

NeMo Guardrails is NVIDIA’s framework for defining and enforcing policies in LLM systems through a DSL called Colang. It lets you express rules such as “if the user asks about topic X, answer with template Y” or “if the model tries to do Z, block it” in a conversational-script syntax rather than in Python. It runs as middleware between the app and the model: input rails (validate what comes in), output rails (validate what goes out), dialog rails (control the flow). Designed for complex multi-turn systems where the policies are nontrivial. Gotcha: Colang adds latency per turn (roughly 50-200 ms depending on the policy graph); for high-throughput conversational chat, dialog rails are disabled and only input + output remain. Licence Apache 2.0, maintained by NVIDIA.

Microsoft Presidio

Presidio is the most mature OSS PII (Personally Identifiable Information) detector in the ecosystem. It detects DNI, NIE, IBAN, phone numbers, emails, physical addresses, credit card numbers, personal names, dates of birth… with recognisers based on regex + NER (spaCy) + custom validators. It supports redaction (replace with placeholders), masking (asterisks) or deterministic anonymisation (repeatable hash). For ENS/NIS2 scenarios, it is the piece placed in front (on input) and behind (on output) the LLM to guarantee that no PII is processed or emitted. Gotcha: the built-in recognisers cover English well and the rest badly; for Spanish, Catalan and Basque you have to add custom recognisers, which is disciplined work but doable. Licence MIT, maintained by Microsoft.

More options for Eval:

Phoenix Arize OSS — combines tracing + evals, an alternative to Langfuse Evals. ELv2, Arize AI.
lm-eval-harness — academic suite with standard benchmarks (MMLU, HellaSwag…). MIT, EleutherAI.
HELM — holistic academic evals. Apache 2.0, Stanford CRFM.
Guardrails AI — a pythonic alternative to NeMo Guardrails. Apache 2.0, Guardrails AI Inc..
LlamaGuard / PromptGuard / ShieldGemma — safety models, not frameworks. Open weights, Meta / Google.

Decision matrix — Eval + Guardrails:

If your case is	Choose
Eval in CI, “pytest for LLMs” style	Promptfoo + GitHub Actions
Eval specific to a RAG pipeline	RAGAS + Langfuse datasets
General eval with custom metrics	DeepEval + MLflow dataset
Dialog policy with declarative rules	NeMo Guardrails (Colang)
PII redaction in/out only	Presidio (you do not need NeMo)
Open safety model in Spanish	LlamaGuard 3 or ShieldGemma

Stage 4 — Deploy

Deploy serves tokens to the user with predictable throughput and latency, adapter hot-swap and multi-tenancy where it applies. Covered in the posts on vLLM on K8s, LLM operators, multi-tenant cluster, KV cache, PagedAttention and disaggregated serving.

vLLM

vLLM is the reference OSS inference engine. It implements PagedAttention (KV cache paging in the style of virtual memory, avoiding fragmentation), continuous batching (requests join the batch as they arrive, rather than waiting for the next batch), prefix caching (common prefixes, system prompts, do not recompute the KV cache), LoRA hot-swap (--enable-lora allows loading and unloading adapters without restarting the engine), an OpenAI-compatible API, and disaggregated prefill/decode support since 2025. It covers almost everything from Llama 3 / Mistral / Qwen / DeepSeek. Gotcha: maximum throughput is only reached with --max-num-seqs and --gpu-memory-utilization tuned for the specific model and hardware; the defaults are conservative. The initial tuning session pays off: 2-3× throughput. Licence Apache 2.0, originated at UC Berkeley, today maintained by the vLLM Project / LF AI & Data plus a broad community (Red Hat, NVIDIA, AWS, IBM contribute). Serious alternatives in the same bucket: TGI (Apache 2.0, Hugging Face), SGLang (Apache 2.0, LMSys), TensorRT-LLM (Apache 2.0, NVIDIA, requires conversion).

KServe

KServe is the Kubernetes operator for serving ML models, LLMs included, in a declarative pattern: you define an InferenceService YAML with the model and predictor (which can be vLLM, TGI, Triton, or a custom container) and KServe handles scheduling onto GPU nodes, autoscaling (scale-to-zero included), traffic splitting for canary, and model registry integration. It is the layer that standardises “how a model is deployed on K8s” across multiple engines, instead of inventing engine-specific YAML. It supports multi-model with Inference Graphs (chaining preprocessor → model → postprocessor) and integrates with KEDA/Karpenter for GPU pool autoscaling. Gotcha: scale-to-zero on GPU works badly in practice because the warm-up (loading weights into VRAM) takes tens of seconds; minReplicas: 1 is better. Licence Apache 2.0, maintained by Kubeflow / LF AI & Data. Alternatives: KubeRay (Apache 2.0, Anyscale), llm-d (Apache 2.0, CNCF), KAITO (MIT, Microsoft Azure).

Triton Inference Server

Triton serves heterogeneous models on a single backend: LLMs (via the vLLM or TensorRT-LLM backend), traditional models (ONNX, TorchScript, TensorFlow), custom models. For systems that mix LLM inference with traditional classifiers, embedding encoders, reranking models, OCR and so on, Triton avoids having N different engines in N pods. It supports ensemble models (chaining models in a single request), dynamic batching, model versioning, model warmup. Gotcha: Triton is flexible but heavy to operate; for systems serving only LLMs, vLLM directly is simpler and more optimised. Triton shines when there is real heterogeneity. Licence BSD-3-Clause, maintained by NVIDIA.

Envoy AI Gateway

Envoy AI Gateway is the CNCF ecosystem’s “LLM-aware API gateway”. Built on Envoy Proxy, it adds knowledge of the OpenAI-compatible APIs (chat completions, embeddings, etc.), routing across multiple backends (local vLLM + OpenAI + Anthropic + Bedrock), token-based rate limiting (limits by tokens per minute, not by requests), intelligent retries, fallback between providers, and built-in OTel observability. It is the piece that makes “AI Gateway” real as an architectural category. Gotcha: integration with authentication (OIDC, JWT) is flexible but requires detailed Envoy configuration; an “out of the box” AI Gateway with no configuration produces an Envoy that passes everything through. Licence Apache 2.0, maintained by CNCF since the initial donation from Tetrate. Alternatives: LiteLLM Proxy (MIT, BerriAI), Portkey (MIT, Portkey AI), Kong AI Gateway (Apache 2.0 base + EE, Kong Inc.).

llama.cpp

llama.cpp serves LLMs on CPUs (and Apple Silicon, GPUs via Vulkan/Metal/CUDA) with very aggressive quantisation (GGUF format, down to 2-bit). It is the canonical option for inference on hardware without a dedicated GPU: edge devices, workstations, development machines. It covers everything from small models (Phi-3, Gemma 2B) to Llama 70B on hardware with enough RAM. Gotcha: latency on CPU is orders of magnitude worse than on a dedicated GPU, so it is useful for offline evals, drift checks and local development, not for production serving under real load. Licence MIT, maintained by Georgi Gerganov + community.

More options for Deploy:

TensorRT-LLM — maximum optimisation on NVIDIA Hopper/Ada. Apache 2.0, NVIDIA.
SGLang — good for workloads with structured generation and JSON. Apache 2.0, LMSys.
TGI — a mature alternative, focused on the HuggingFace ecosystem. Apache 2.0, HuggingFace.
NVIDIA Dynamo — multi-node disaggregated serving. Apache 2.0, NVIDIA.
llm-d — K8s operator specific to LLMs. Apache 2.0, CNCF.

Decision matrix — Deploy:

If your case is	Choose
Production serving on NVIDIA H100/A100	vLLM (the safe default)
Absolute throughput squeezing on Hopper	TensorRT-LLM + vLLM plugin or standalone
Edge / local dev without GPU	llama.cpp
Multi-model (LLM + classifiers + encoders)	Triton with the vLLM backend
Declarative K8s with autoscaling	KServe + vLLM as predictor
AI Gateway with token rate limiting	Envoy AI Gateway
Multi-node disaggregated GPU cluster	NVIDIA Dynamo over vLLM

Stage 5 — Observe

Observe propagates trace_id end-to-end, emits runtime metrics, runs an LLM judge over sampling and detects drift. Detailed in tracing with AgentSight, MCP observability with OTel and eBPF + drift.

OpenTelemetry Collector

The OTel Collector is the agent that receives traces, metrics and logs in OTel format (or in any other via receivers), processes them (filters, sampling, attribute enrichment, PII redaction), and routes them to one or several backends (Tempo, Jaeger, Prometheus, Loki, Langfuse…). It is the piece that decouples the apps from the observability backend: moving from Tempo to Jaeger means changing the Collector’s exporter, not the app. For LLMOps it matters especially because the OTel GenAI semantic conventions spec defines the attributes gen_ai.request.model, gen_ai.prompt.version, gen_ai.response.tokens and so on, which stitch the trace_id to the system’s lineage. Gotcha: the Collector configuration tends to grow; without discipline and periodic review it ends up as an 800-line YAML that nobody understands. Modularising with extensions helps. Licence Apache 2.0, maintained by the CNCF / OpenTelemetry Project.

Tempo (traces) + Jaeger

Grafana Tempo is the distributed trace backend optimised for cost: it uses an object store (S3/MinIO) instead of Elasticsearch, deduplicates by trace_id, and integrates natively with Grafana for visualisation. For LLMOps, where a real request generates 10-30 spans (gateway, prompt pull, RAG retrieval, prefill, decode N times, scoring), Tempo copes with high volumes at a reasonable cost. Jaeger is the more established CNCF alternative, better for cases under 100k traces per day, worse for native object store. Gotcha: Tempo has no traditional indexing; searches like “traces that took more than 5s and touched tenant X” require TraceQL + Grafana and are not as fast as Jaeger with Elasticsearch. For immediate ad-hoc diagnosis it is worth keeping a parallel Jaeger with aggressive sampling. Licences AGPL 3.0 (Tempo) and Apache 2.0 (Jaeger), maintained by Grafana Labs and CNCF respectively.

Prometheus + Grafana

Prometheus is the time-series metrics foundation of the ecosystem. Pull model (it scrapes /metrics endpoints), PromQL for queries, exporters for everything (Postgres, Kafka, NVIDIA GPU via dcgm-exporter, vLLM natively). Grafana visualises Prometheus + Tempo + Loki on a single plane. For LLMOps, the critical metrics are gpu_utilization, kv_cache_usage_pct, tokens_per_second, prefill_latency_p95, decode_latency_p95, queue_depth, aggregated by tenant. Gotcha: Prometheus is very good up to about 1M active series; above that it is worth moving to Thanos or Mimir for long retention and horizontal scalability. For the blog’s typical LLM cluster (4-8 H100), Prometheus alone is enough. Licences Apache 2.0 (Prometheus, CNCF) and AGPL 3.0 (Grafana 10+, Grafana Labs).

Langfuse

Langfuse is the LLM-specific OSS observability + prompt management tool. It captures spans with LLM semantic conventions (input, output, model, tokens, latency, score, user_id, session_id), visualises them as conversational traces (not just span trees), manages prompts versioned with a production label and allows curated datasets + evals from the same UI. For serious LLMOps, Langfuse fills the gap that neither Tempo nor Jaeger covers: a tracing UI designed to be LLM-first. Gotcha: Langfuse maintains its own store (Postgres + ClickHouse for high volume); on large clusters the operational side of ClickHouse deserves attention. To get started, Postgres alone copes. Licence MIT for the OSS core, EE Enterprise Edition with additional features (SSO, audit logs, advanced RBAC). Maintained by Langfuse GmbH (Berlin, German). There is Langfuse Cloud (SaaS).

Phoenix Arize OSS

Phoenix is Arize AI’s OSS for LLM observability + evals, an alternative to Langfuse with a different emphasis: more oriented to evaluation and visual debugging (embedding drift, cluster analysis), less to prompt management. A good pairing with Langfuse when you want a dual approach: Langfuse for “production conversational traces”, Phoenix for “exploratory investigation of model behaviour”. Gotcha: Phoenix duplicates functionality with Langfuse and with MLflow; having all three in production multiplies the operational burden. Pick one as the main tool and the others as complements. Licence Elastic License 2.0 (not strictly OSI), maintained by Arize AI.

Cilium Tetragon + Hubble

Tetragon (eBPF runtime security observer) and Hubble (eBPF network observer) are the low-level pieces that give real runtime visibility into the cluster: which processes run in which pods, what syscalls they make, what network connections they open, in real time. For ENS/NIS2 environments that demand “prove what ran in production”, Tetragon is the irrefutable audit layer: every process execution with its parent, its capabilities, its K8s context. Hubble visualises network flows by pod, namespace, service. Gotcha: the volume of events generated is high; without kernel-side filtering (which Tetragon supports with TracingPolicy), it saturates the observability plane fast. Discipline in policies. Licence Apache 2.0 for both, maintained by Cilium / CNCF / Isovalent.

Evidently AI

Evidently is the OSS library for drift detection: it compares input and output distributions between two time windows (training vs production, current week vs previous week), applies statistical tests (KS, PSI, Wasserstein, chi-square) and generates HTML reports. For LLMOps it detects when the distribution of prompts changes (new topics, new lengths, new languages) or when the model starts answering shorter / longer / differently. Gotcha: Evidently is oriented to tabular data and embeddings; for raw text it is best combined with an encoder embedder that produces vectors before applying the tests. Licence Apache 2.0, maintained by Evidently AI (a company). Alternatives: NannyML (Apache 2.0, NannyML BV), Alibi Detect (Apache 2.0, Seldon).

More options for Observe:

Loki — Prometheus-style logs backend for Grafana. AGPL 3.0, Grafana Labs.
Pixie — auto-instrumented eBPF observability. Apache 2.0, CNCF.

Decision matrix — Observe:

If your case is	Choose
Minimum viable stack	OTel Collector + Tempo + Prometheus + Grafana + Langfuse
Traces with strong ad-hoc search	Add Jaeger with aggressive sampling
ENS / NIS2 compliance runtime audit	Tetragon + Hubble + mandated retention
Exploratory investigation of the model	Phoenix Arize OSS alongside Langfuse
Statistical drift detection	Evidently over embeddings + inputs
Cluster beyond 1M Prometheus series	Mimir (Grafana Labs) or Thanos

Stage 6 — Retrain + cross-cutting pieces

Retrain closes the loop feedback → triage → enriched dataset → new adapter. Prompt versioning and data versioning stitch lineage across stages. Detailed in retrain, prompt versioning and data versioning.

Apache Airflow

Airflow is the most established OSS DAG scheduler. You define workflows as Python code (DAGs), each DAG with tasks (operators) that run according to declared dependencies plus a cron schedule. For retraining: a weekly DAG that extracts feedback from Postgres, triages it with an LLM-as-classifier, enriches the enriched dataset in DVC, launches the fine-tuning job on Kubernetes, runs evals against the golden set, and promotes if it passes the gates. There is a huge ecosystem of operators for everything (S3, Postgres, Kafka, Slack, K8s, Spark…). Gotcha: Airflow 2.x improved a great deal over the chaos of 1.x, but the scheduler is still a component that deserves operational attention (Postgres backend, executor pool, sidecar workers); for simple pipelines it is over-engineering. Licence Apache 2.0, maintained by the ASF.

Argo Workflows

Argo Workflows is the K8s-native equivalent of Airflow: each step is a container, the DAGs are defined as K8s YAML, the executor is Kubernetes itself. For environments where everything is K8s, Argo fits without an extra component to maintain. Long tasks (a six-hour fine-tuning run) execute as Pods that survive control plane failures. It integrates trivially with Kubeflow Pipelines (which is built on top). Gotcha: Argo’s YAML syntax is verbose; for complex DAGs, Argo feels less productive than Airflow in Python. Solutions: Hera (Python DSL for Argo, a DataBricks contribution) or Argo + custom CRDs. Licence Apache 2.0, maintained by CNCF.

Kubeflow Pipelines

Kubeflow Pipelines is the layer above Argo Workflows oriented specifically to ML: artefact tracking, experiment tracking, reusable pipeline templates, versioned components. Built on Argo, it adds the ML conceptual model (input artefact, output artefact, metrics) that raw Argo does not have. For cyclical retraining on a K8s cluster, it is the most “ML-ready” option in the OSS ecosystem. Gotcha: Kubeflow as a full suite is heavy (10+ components); many organisations install only Pipelines + Training Operator + Katib and skip Notebook Server / legacy KFServing. Licence Apache 2.0, maintained by CNCF / LF AI & Data.

Feast

Feast is the most widely used OSS feature store. It defines feature views over batch sources (BigQuery, Postgres, Parquet) and online sources (Redis, DynamoDB, Postgres with an extension), exposes a consistent API for read-during-training and read-during-inference (point-in-time correctness), and guarantees that the features of the model in production are the same ones it was trained with. For LLMOps where the model needs consistent user / session / context features (latest plan, tenure as a customer, recent tickets), Feast provides the discipline. Gotcha: for many pure LLM systems (a RAG chatbot with no complex features), Feast is over-engineering, Postgres is enough. When there are real features (recommendation, scoring, ranking), Feast shines. Licence Apache 2.0, maintained by LF AI & Data.

Argilla

Argilla is the OSS annotation + HiL (human-in-the-loop) platform most aligned with modern LLMOps. It creates annotation projects with templates (classification, ranking, span annotation, RLHF preference, free-form text), connects to HuggingFace datasets, and integrates with Langfuse to import traces from production as cases to annotate. It supports multiple annotators with reconciliation, kappa scoring and quality control. To enrich retrain datasets with cases from the “blunt tone” cluster of the Retrain post, Argilla is the front end. Gotcha: Argilla requires Elasticsearch for production performance; for small experiments SQLite is enough. Licence Apache 2.0, maintained by Argilla, Inc. (acquired by Hugging Face in 2024). Alternative: Label Studio (Apache 2.0, HumanSignal), more generalist, less LLM-first.

Langfuse Prompts + MLflow Prompt Registry

Langfuse Prompts manages prompts as versioned entities with labels (production, staging, experiment). The client reads the active prompt from Langfuse on the request path (with a local cache of a few seconds) and propagates prompt_id, prompt_version to the OTel span, exactly as the forensic post does. MLflow Prompt Registry does the same with a slightly different conceptual model (no labels-as-pointers; it uses stages, like the Models registry). Both are valid; the choice depends on which tracking tool is already in place. Gotcha (Langfuse): labels are mutable, so changing production points at another version with no explicit audit; prompts are better deployed via PR against the configs repo, not manually in the UI. Licences and governance covered above.

More options for Retrain + cross-cutting pieces:

Prefect — “modern” Python DAGs, an alternative to Airflow. Apache 2.0, Prefect Tech.
Dagster — DAGs with a strong focus on data assets. Apache 2.0, Dagster Labs.
Label Studio — generalist annotation. Apache 2.0, HumanSignal.
OpenLineage — cross-system lineage event standard. Apache 2.0, LF AI & Data.
DataHub / Apache Atlas / OpenMetadata — catalogue + lineage with a UI. Apache 2.0.

Decision matrix — Retrain + cross-cutting pieces:

If your case is	Choose
Simple pipelines with a catalogue of operators	Airflow
Everything is K8s, minimal components	Argo Workflows
ML pipelines with artefact tracking	Kubeflow Pipelines
HiL annotation for LLM retraining	Argilla + Langfuse integration
Features shared between training and inference	Feast
No complex features, only prompts + LLM	Skip Feast
Lightweight prompt registry	Langfuse Prompts
MLflow is already centralised	MLflow Prompt Registry

Master table: licence, governance and enterprise offering

Tool	Licence	Governance / maintainer	Commercial EE / SaaS
DVC	Apache 2.0	Iterative.ai	DVC Studio
lakeFS	Apache 2.0	Treeverse	lakeFS Cloud
MinIO	AGPL v3	MinIO Inc.	SUBNET / AIStor
Qdrant	Apache 2.0	Qdrant GmbH	Qdrant Cloud
pgvector	PostgreSQL License	Andrew Kane + community	— (built-in Postgres clouds)
PostgreSQL	PostgreSQL License	PostgreSQL Global Dev Group	several managed (Crunchy, Neon, Aiven, EDB)
Apache Kafka	Apache 2.0	ASF	Confluent Cloud
Debezium	Apache 2.0	Red Hat / ASF	Debezium Server / Confluent Connectors
Apache Flink	Apache 2.0	ASF	Ververica Platform, Aiven
HF Transformers	Apache 2.0	Hugging Face SAS	HF Inference Endpoints / Enterprise Hub
PEFT	Apache 2.0	Hugging Face SAS	— (part of the HF offering)
bitsandbytes	MIT	Tim Dettmers + community	—
MLflow	Apache 2.0	LF AI & Data	Databricks MLflow
Axolotl	Apache 2.0	OpenAccess AI Collective	—
Ray (Train)	Apache 2.0	Anyscale + community	Anyscale Platform
DeepSpeed	MIT	Microsoft	—
DeepEval	Apache 2.0	Confident AI	Confident AI SaaS
RAGAS	Apache 2.0	Exploding Gradients	—
Promptfoo	MIT	Promptfoo, Inc.	Promptfoo Cloud
NeMo Guardrails	Apache 2.0	NVIDIA	NeMo Microservices
Presidio	MIT	Microsoft	—
Phoenix (Arize)	Elastic v2	Arize AI	Arize Platform
vLLM	Apache 2.0	vLLM Project / LF AI & Data	several (Red Hat, AWS, IBM, NVIDIA)
TGI	Apache 2.0	Hugging Face SAS	HF Inference Endpoints
SGLang	Apache 2.0	LMSys + community	—
TensorRT-LLM	Apache 2.0	NVIDIA	NVIDIA AI Enterprise
llama.cpp	MIT	Georgi Gerganov + community	—
Triton Inference Server	BSD-3	NVIDIA	NVIDIA AI Enterprise
KServe	Apache 2.0	LF AI & Data (Kubeflow)	—
Envoy AI Gateway	Apache 2.0	CNCF / Tetrate	Tetrate Service Bridge
LiteLLM	MIT	BerriAI	LiteLLM Cloud
OpenTelemetry	Apache 2.0	CNCF	several vendors (Honeycomb, Datadog, Grafana)
Tempo	AGPL 3.0	Grafana Labs	Grafana Cloud Tempo
Jaeger	Apache 2.0	CNCF	—
Prometheus	Apache 2.0	CNCF	Grafana Cloud, AMP, GCP Managed Prom, Azure
Grafana	AGPL 3.0	Grafana Labs	Grafana Cloud, Grafana Enterprise
Loki	AGPL 3.0	Grafana Labs	Grafana Cloud Loki
Langfuse	MIT (core) / EE	Langfuse GmbH	Langfuse Cloud
Tetragon	Apache 2.0	Cilium / CNCF / Isovalent	Isovalent Enterprise
Hubble	Apache 2.0	Cilium / CNCF	Isovalent Enterprise
Evidently AI	Apache 2.0	Evidently AI	Evidently Cloud
Apache Airflow	Apache 2.0	ASF	Astronomer, MWAA, Cloud Composer
Argo Workflows	Apache 2.0	CNCF	—
Kubeflow Pipelines	Apache 2.0	CNCF / LF AI & Data	—
Feast	Apache 2.0	LF AI & Data	Tecton (commercial)
Argilla	Apache 2.0	Hugging Face	HF Hub features
OpenLineage	Apache 2.0	LF AI & Data	—
DataHub	Apache 2.0	Acryl Data	Acryl Cloud

The pattern to watch for when reading the table: AGPL 3.0 and Elastic v2 are the ones that add the most friction in companies with strict licensing policies (legal asks for a specific review). The Apache 2.0 ones pass compliance without argument. Those with an “EE Enterprise” tier or equivalent hide a decision: the OSS version is functionally complete for production, but team features (SSO, audit, advanced RBAC) live in the commercial version. For ENS customers under a HIGH declaration, the EE features (SSO with corporate SAML/OIDC, immutable audit logs) are usually mandatory, so it is worth knowing the price beforehand.

When to move from the “minimum stack” to the “full stack”

The whole catalogue can be intimidating. But you do not build all of it on day one. There is a reasonable order that this blog has been validating in posts throughout the series. The minimum viable stack that serves an LLM API with acceptable discipline:

Serving: vLLM on Kubernetes plus an Envoy AI Gateway in front.
Data: Postgres + pgvector (no Qdrant), MinIO for object store, no Kafka.
Tune: Axolotl + MLflow, no Ray Train.
Eval: Promptfoo in CI, no RAGAS and no judge in production.
Observe: OTel Collector + Prometheus + Grafana + Langfuse, no Phoenix and no Tetragon.
Retrain: feedback in Postgres + crontab scripts, no Airflow.
Versioning: prompts in Langfuse + datasets in DVC over MinIO, no lakeFS.

That is about 8-10 components and it serves a reasonable LLM system for a single tenant with moderate traffic. As the system grows, there are identifiable moments where adding each piece pays off:

Trigger	Component to add
Multi-tenant with isolated corpora	Qdrant (collections per tenant, ACL)
The corpus is refreshed often and periodically breaks	lakeFS (branches with pre-merge hooks)
The embedding pipeline needs streaming	Kafka + Debezium + Flink
Retraining moves from monthly to weekly	Airflow or Argo Workflows
Shared features appear (customer profile, scoring)	Feast
Annotation outgrows informal capacity	Argilla
RAG eval needs specific metrics	RAGAS + Langfuse datasets
ENS compliance demands runtime audit	Tetragon + Hubble
Drift is invisible and shows up late	Evidently
A single stack stops covering multi-model	Triton or KServe with several predictors
Multiple simultaneous multi-tenant adapters	vLLM Production Stack + a dedicated Operator

Each jump adds 1-2 components and is worth the cost only when the trigger is clear. Adding Kafka “just in case” when the corpus is updated once a month is net negative work.

What we have not covered (yet)

Some pieces still deserve a post of their own:

Schema Registry for LLM data and prompts (Confluent OSS, Karapace, JSON Schema Registry).
Catalogue + lineage in depth: DataHub vs Atlas vs OpenMetadata + OpenLineage taken seriously.
Federated learning over OSS (Flower, FedML) for scenarios where the data is not centralised.
OSS MCP Servers and their place in the stack as a tools / actions layer.
“Agentic” evals specific to multi-step systems with tool use.
Upgrade best practices for each component (vLLM every 6 weeks, Kafka major every 18 months, etc.).

References

vLLM · vLLM Production Stack · TGI · SGLang · TensorRT-LLM · llama.cpp — OSS inference engines.
Triton Inference Server · KServe · Envoy AI Gateway · LiteLLM — orchestration and AI gateway.
Qdrant · pgvector · Milvus — vector databases.
DVC · lakeFS · MinIO — versioning and object store.
Apache Kafka · Debezium · Apache Flink — streams and CDC.
Hugging Face Transformers · PEFT · bitsandbytes · Axolotl — fine-tuning.
MLflow · Ray Train · Kubeflow Training Operator — training orchestration.
DeepEval · RAGAS · Promptfoo · NeMo Guardrails · Presidio — evals and guardrails.
OpenTelemetry · OTel GenAI semconv · Tempo · Prometheus · Grafana · Loki — observability foundation.
Langfuse · Phoenix Arize · Evidently AI — LLM observability and drift.
Cilium · Tetragon · Hubble — eBPF runtime.
Apache Airflow · Argo Workflows · Kubeflow Pipelines · Feast · Argilla — orchestration + retrain + annotation.

The parallel catalogue: the six LLMOps stages in open source and on the hyperscalers (AWS, GCP, Azure)

Sat, 23 May 2026 07:00:00 +0200

TL;DR

The previous forensic post used a single request to walk the six stages of the LLMOps pipeline and the two cross-cutting components. This post walks the same stages but crosses them with three extra columns: how each stage is built in open source on-premise, and what the equivalent services are on AWS, GCP and Azure. It is not a migration guide nor a cost benchmark: it is a catalogue of equivalences with its gaps. The general pattern you will see: OSS gives you control, sovereignty and free composition in exchange for expensive operations; the hyperscalers give you integration and time-to-market in exchange for lock-in on margins, data contracts and political dependency. For scenarios subject to ENS / NIS2 with critical customer data, OSS wins by default; for discovery projects where time-to-market is the deciding metric, the hyperscaler wins by default. The interesting part is in the middle. As a concrete thread, at the end we take the multi-tenant chatbot from the previous post and port it to AWS piece by piece to show what disappears, what appears, and where the lock-in materialises.

You are here: the same six stages, but by column

This post shares a map with the previous one, the six stages and the two cross-cutting pieces are all active, but it changes the cut: instead of following a request horizontally, it makes the vertical cut and shows which tools live in each stage depending on the deployment model.

The analogy: your own bakery and the franchise

A baker opens a business. There are two possible models.

He can open his own bakery: rent the premises, buy the oven, choose the flour suppliers, hire his master baker, write his recipes, set the prices, dress the shop window. The day he wants to launch an organic rye sourdough loaf, he asks nobody for permission. The day the price of flour goes up, he finds another supplier. But he pays for all of it: the initial investment, the risk, the daily operations, the months when he misreads the neighbourhood. The bakery is his.

Or he can join a franchise: the franchisor hands over the turnkey premises, the oven with a maintenance contract, the suppliers already negotiated, the operating manuals, the written recipes, the centralised marketing, the loyalty app, the POS system. The learning curve is weeks, not years. But the recipes belong to the franchisor, so do the suppliers, the price of bread is in the catalogue, and the day the croissant formula changes he gets an informational email, not a business decision.

Both bakeries turn out bread. Both meet health regulations and generate income. The operational difference is enormous and it is not about technology: it is about ownership, control and timescale.

The parallel with LLMOps is direct. The on-premise OSS stack is the bakery you own. The managed stack on the hyperscalers is the franchise. The pieces that appear in each stage are functionally equivalent, at the end of the day both solve the same technical problem, but the governance model, the operational cost, the lock-in and the compliance guarantees are different. This post builds the parallel catalogue so that the choice is not made by default.

Quick recap of the previous post

In the forensic post we followed one specific request: a premium-es user of an insurance company asking “How do I cancel my premium subscription?” to the multi-tenant support chatbot of the SaaS provider that hosts it. The route crossed the six stages of the LLMOps pipeline, Data, Tune, Eval, Deploy, Observe, Retrain, plus the two cross-cutting components, prompt versioning and data versioning, over an on-premise infrastructure: RKE2 with Cilium BGP, a 4×H100 SXM cluster, an RTX 4090 for development, vLLM on Kubernetes, Langfuse + OTel + Prometheus + Tempo, Postgres + Qdrant, DVC + lakeFS + MinIO, Kafka and MLflow. The system meets ENS / NIS2 and keeps trace_id propagated end to end.

What comes now is that same system, piece by piece, showing for each box which tool does the job if you are on public cloud, because the integrator’s question is rarely “OSS yes or no?”: it is “what do I lose and what do I gain if I take this box managed?”. And the answer differs box by box.

Stage 1 — Data

The problem. There are three sub-problems the Data stage solves, frequently confused with one another. First, versioning and identity of the corpus and of the training datasets (making sure a dataset_id, dataset_version exists and propagates). Second, storage and serving of the operational corpus (object store + vector index + structured text). Third, streams and ingestion from source systems with CDC, transformation and stable schemas (Schema Registry).

Reference OSS stack. Versioning lives in DVC (pointers in git, content in an object store) combined with lakeFS for branch/merge semantics over data. The post on data versioning goes into the functional difference. The object store is MinIO or Ceph. The vector index is Qdrant or Milvus for large corpora (millions of chunks) and pgvector on Postgres 18 for small cases where having one component fewer pays off operationally. The stream layer is Kafka (pure Apache or Redpanda) with a Schema Registry (Confluent or Karapace OSS), CDC with Debezium or Flink CDC, transformation with Flink or Spark Structured Streaming. The catalogue / lineage is DataHub, Apache Atlas or OpenMetadata with OpenLineage events between systems. The post on PostgreSQL + Qdrant ingestion and the post on RAG over Kafka cover the detailed operations.

Hyperscaler equivalents. On AWS, the corpus lives in S3 (with versioning enabled, which is the cheap substitute for serious data versioning), tabular queries in Athena or Redshift, the vector index in Amazon OpenSearch with the vector plug-in or in Amazon Aurora pgvector. The stream layer is MSK (managed Kafka) or Kinesis Data Streams, CDC with AWS DMS, transformation with Glue Streaming or MSK Connect. The catalogue is AWS Glue Data Catalog + AWS Lake Formation for data governance. And for the RAG case there is also Amazon Bedrock Knowledge Bases, the managed shortcut: you give it S3, it indexes into OpenSearch or Aurora pgvector, exposes a retrieval API and the operations end there, in exchange for paying per indexed chunk and per query.

On GCP, the corpus lives in Cloud Storage (with object versioning), the analytical warehouse is BigQuery (with BigQuery Vector Search already integrated), the dedicated vector store is Vertex AI Vector Search (formerly Matching Engine). The stream layer is Pub/Sub + Dataflow, CDC with Datastream. Catalogue and lineage are Dataplex (which in 2024-2025 absorbed Data Catalog and added automatic lineage). The managed equivalent of Knowledge Bases is Vertex AI Search (formerly Discovery Engine).

On Azure, the corpus lives in ADLS Gen2, tabular queries in Microsoft Fabric / Azure Synapse, the vector index in Azure AI Search (vector mode) or Azure Cosmos DB for PostgreSQL with pgvector. The stream layer is Event Hubs + Stream Analytics or Microsoft Fabric Real-Time Intelligence, CDC with Azure Data Factory. The catalogue is Microsoft Purview, which covers catalogue, lineage and data governance integrated with Entra ID.

Summary table — Data stage.

Functional piece	OSS on-premise	AWS	GCP	Azure
Object store	MinIO, Ceph	S3	Cloud Storage	ADLS Gen2
Dataset versioning	DVC, lakeFS	S3 Versioning (limited), Lake Formation	GCS Versioning, Dataplex	ADLS versioning, Purview
Vector index	Qdrant, Milvus, pgvector	OpenSearch, Aurora pgvector, Bedrock KB	Vertex Vector Search, BigQuery VS	Azure AI Search, Cosmos pgvector
Stream + CDC	Kafka + Debezium + Flink	MSK / Kinesis + DMS + Glue	Pub/Sub + Datastream + Dataflow	Event Hubs + ADF
Schema Registry	Karapace, Confluent OSS	Glue Schema Registry	Pub/Sub schemas	Schema Registry (Event Hubs)
Catalogue + lineage	DataHub, Atlas, OpenLineage	Glue Catalog + Lake Formation	Dataplex	Purview
Managed end-to-end RAG	— (you build it)	Bedrock Knowledge Bases	Vertex AI Search	Azure AI Studio Knowledge

Where the names mislead. S3 Versioning is not DVC. It keeps object versions but has no notion of a dataset (which objects together make up version 3 of the enriched set?), does not propagate dataset_hash to the trainer, does not integrate with experiment tracking, and does not fail a CI run if a dataset breaks schema. Covering that properly on AWS requires combining it with Lake Formation, Glue Data Catalog and your own records in SageMaker Experiments. The same applies on GCP with Dataplex and on Azure with Purview. The gap is real and is paid for in operations or in broken lineage.

Stage 2 — Tune

The problem. Producing a new model_id, model_version, typically a LoRA adapter over a stable base, as the post on continuous fine-tuning explains, with lineage back to the dataset that trained it and experiment tracking that lets you reproduce it six months later.

OSS stack. Technical core: HuggingFace Transformers + PEFT (LoRA, QLoRA), bitsandbytes for quantisation, DeepSpeed or FSDP for parallelism. Experiment tracking: MLflow (authoritative) or Weights & Biases self-hosted. Convenience frameworks: Axolotl and Llama Factory wrap the machinery above in declarative configuration. Distributed orchestration: Kubeflow Training Operator or Ray Train. On small infrastructures, direct scripts with Slurm or K8s Jobs over GPU pools. The lineage chain dataset → run → model is closed by registering the dataset as an MLflow input artefact.

Hyperscaler equivalents. On AWS, SageMaker Training Jobs covers most workloads, SageMaker HyperPod covers large training runs with resilience to node failures, and SageMaker JumpStart offers click-to-train fine-tuning over a catalogue of pre-curated models. For fine-tuning Bedrock models (hosted Claude, Llama, Mistral) there is Bedrock Custom Models: you upload the dataset to S3, Bedrock trains, and it returns a private endpoint with provisioned throughput. The equivalent experiment tracking is SageMaker Experiments or managed MLflow in SageMaker (yes, AWS has officially hosted MLflow since 2024).

On GCP, Vertex AI Custom Training runs any container with GPUs or TPUs; Vertex AI Tuning is the managed API for fine-tuning Gemini and Model Garden models. Experiment tracking in Vertex AI Experiments (with MLflow compatibility).

On Azure, Azure ML Training Jobs on your own clusters or managed compute; Azure OpenAI fine-tuning for fine-tuning GPT and the o-series; Azure ML Experiments with MLflow natively integrated since 2022.

Summary table — Tune stage.

Functional piece	OSS on-premise	AWS	GCP	Azure
Training framework	HF Transformers + PEFT	SageMaker SDK	Vertex AI SDK	Azure ML SDK
Quantisation / parallelism	bitsandbytes, DeepSpeed, FSDP	SageMaker libs + HF support	Vertex + HF support	Azure ML + HF support
Managed fine-tuning (black box)	—	Bedrock Custom Models, JumpStart	Vertex Tuning (Gemini)	Azure OpenAI fine-tuning
Distributed on a cluster	Kubeflow, Ray Train, Slurm	SageMaker HyperPod	Vertex AI Training (multi-node)	Azure ML compute clusters
Experiment tracking	MLflow, W&B self-hosted	SageMaker Experiments, managed MLflow	Vertex Experiments	Azure ML + MLflow
Access to the base model	Whatever you download (Llama, Mistral, Qwen)	Bedrock catalog + HF Hub	Vertex Model Garden + HF Hub	Azure ML model catalog + HF Hub

Where the names mislead. The managed fine-tunings (Bedrock Custom, Vertex Tuning, AOAI fine-tuning) are a black box: you do not choose hyperparameters beyond a handful, you do not see the trainer’s detailed logs, and you cannot inspect the dataset once it is inside their pipeline. The experiment tracking they offer is not comparable to MLflow sitting next to the trainer, where you can capture any metric and artefact. For scenarios where operationally you do not need inspection this is liberating; for ENS / NIS2 scenarios where you have to prove what trained what, the black box fails by construction.

Stage 3 — Eval

The problem. Validating candidates before and after promotion against a golden set, with operational metrics (faithfulness to the RAG, tone, format compliance, toxicity, jailbreak resistance, PII leakage) run as gates in CI and as online sampling. Covered in the post on evals and the one on guardrails.

OSS stack. Eval suites: DeepEval, RAGAS (specialised in RAG), Promptfoo (declarative, ideal for CI), lm-eval-harness (academic), HELM. Evals integrated with tracing: Langfuse Evals, Phoenix Arize OSS. LLM-as-judge judges: any local OSS model; in serious systems, two different judges to reduce bias. Safety and guardrails: NeMo Guardrails (NVIDIA), Guardrails AI, LlamaGuard + PromptGuard (Meta), ShieldGemma (Google, open weights), PII detectors such as Presidio (Microsoft) on-prem.

Hyperscaler equivalents. On AWS, Bedrock Model Evaluation offers automatic evals (toxicity, accuracy, robustness) and human-in-the-loop, Bedrock Guardrails covers the safety layer (denied topics, PII, prompt injection, contextual grounding check), and SageMaker Clarify adds bias and explainability over general models.

On GCP, Vertex AI Evaluation Service runs evals with automatic metrics and an LLM judge, Vertex AI Model Armor and the safety filters built into the Gemini API cover the guardrails layer. Vertex AI Studio exposes interactive Eval for prompt iteration.

On Azure, the Azure AI Evaluation SDK runs offline evals against datasets, Azure AI Content Safety covers safety (Prompt Shields against jailbreak, Groundedness detection, content categories, PII detection). All accessible from Azure AI Foundry.

Summary table — Eval stage.

Functional piece	OSS on-premise	AWS	GCP	Azure
Automatic eval suite	DeepEval, RAGAS, Promptfoo	Bedrock Model Evaluation	Vertex AI Evaluation Service	Azure AI Evaluation SDK
LLM-as-judge	Any OSS model	Bedrock judge models	Vertex judge (Gemini)	Azure OpenAI judges
Golden set management	Langfuse datasets, manual	SageMaker Ground Truth datasets	Vertex Datasets	Azure ML Datasets
Guardrails (jailbreak, PII, prompt injection)	NeMo Guardrails, LlamaGuard, Presidio	Bedrock Guardrails	Vertex Model Armor + Gemini safety	Azure AI Content Safety (Prompt Shields, Groundedness)
Eval in CI	Promptfoo + GitHub Actions	Bedrock Eval API + CodeBuild	Vertex Eval API + Cloud Build	Azure AI Eval + Azure Pipelines

Where the names mislead. Managed guardrails are convenient but opaque: the Bedrock Guardrails rules are configurable but the detection implementation cannot be inspected; the same goes for Azure AI Content Safety. In OSS, NeMo Guardrails shows you the Colang graph and Presidio shows you the recognisers, auditable and modifiable. For regulated systems where an auditor asks “how exactly does it detect PII?”, OSS answers with code; the cloud answers with documentation.

Stage 4 — Deploy

The problem. Serving tokens to the end user with predictable latency and throughput, a decent cost / token ratio, hot-swap adapter support, and multi-tenancy if the business demands it. Covered in the posts on KV cache, PagedAttention, disaggregated serving, vLLM on K8s, LLM operators and multi-tenant cluster.

OSS stack. Inference engine: vLLM (PagedAttention, prefix caching, LoRA hot-swap, OpenAI-compatible API) as the reference, TensorRT-LLM for maximum optimisation on Hopper / Ada, SGLang for workloads with heavy prompt restructuring, TGI (Hugging Face) as a mature alternative, llama.cpp for edge and CPUs, NVIDIA Dynamo for multi-node disaggregated serving on large clusters. Orchestration on Kubernetes: KServe, KubeRay, dedicated operators such as llm-d, vLLM Production Stack and KAITO. Gateway / control plane: Envoy AI Gateway, LiteLLM Proxy, Portkey AI Gateway, Kong AI Gateway. Triton Inference Server covers mixed workloads (LLM + traditional) where a single backend matters.

Hyperscaler equivalents. On AWS, two different routes. The managed-by-model route is Amazon Bedrock: a catalogue of hosted models (Claude, Llama, Mistral, Cohere, Titan), pay per token or Provisioned Throughput with an SLA, Bedrock Prompt Caching as the conceptual equivalent of vLLM’s prefix caching, and integrated Bedrock Agents and Bedrock Knowledge Bases. The managed-by-infrastructure route is SageMaker Endpoints (real-time, async, serverless, batch) with Inference Components to densify multiple models on one instance. Own hardware: AWS Inferentia and Trainium via the Neuron chip, an alternative to NVIDIA with a better cost / token on stable workloads if you compile your model.

On GCP, Vertex AI Prediction Endpoints runs your containers or models from the Model Garden, the Gemini API via Vertex AI offers the managed Gemini models, and Cloud TPU v5e / v5p / Trillium (v6) is the in-house hardware competing with the H100 for training and inference. For sovereignty there is Google Distributed Cloud air-gapped, which brings Vertex AI to a certifiable on-premise rack.

On Azure, Azure OpenAI Service serves OpenAI models (GPT-4.1, o-series, GPT-image), Azure ML Managed Online Endpoints runs any model (OSS included, via container), and Azure AI Foundry models absorbed in 2025 the catalogue of open models served as-a-service. Hardware: Azure ND H100 v5, ND H200 v5, ND GB200 v6 and the in-house bet Microsoft Maia 100 for internal inference.

Summary table — Deploy stage.

Functional piece	OSS on-premise	AWS	GCP	Azure
Inference engine	vLLM, TensorRT-LLM, SGLang, TGI	Bedrock (managed model), SM Endpoints (your container)	Vertex Prediction, Gemini API	Azure OpenAI, Azure ML Endpoints
Prefix / prompt caching	vLLM native	Bedrock Prompt Caching	Vertex AI context caching	Azure OpenAI prompt caching
Adapter hot-swap (LoRA)	vLLM `--enable-lora`, S-LoRA	Bedrock Custom Models endpoints	Vertex Tuning endpoints	Azure OpenAI fine-tuned deployments
Disaggregated serving	NVIDIA Dynamo, vLLM PD-disagg	— (managed internally, not exposed)	— (managed internally, not exposed)	— (managed internally, not exposed)
Accelerator hardware	NVIDIA H100/H200/B200, AMD MI300	Inferentia, Trainium, NVIDIA	TPU v5/v6, NVIDIA	Maia, NVIDIA
AI Gateway / proxy	Envoy AI Gateway, LiteLLM, Portkey, Kong	API Gateway + Bedrock	Vertex AI + Apigee	Azure API Management + AOAI
K8s orchestration	KServe, KubeRay, llm-d, KAITO	EKS + SageMaker Operators	GKE + Vertex AI	AKS + KAITO

Where the names mislead. Bedrock Prompt Caching and Vertex context caching sound equivalent to vLLM’s prefix caching, but operationally they are different: the cache lives on the hyperscaler’s plane, its eviction policy is opaque, its cost is billed separately, and you cannot easily see the hit ratio per tenant. In vLLM you see the hit ratio in Prometheus metrics and you decide the policy. The same applies to disaggregated serving: the hyperscalers implement it internally to cut their own cost of serving, but they do not expose prefill/decode control to the user. If you need your workload’s TTFT to be controlled separately from TPS, that lever is not available.

Stage 5 — Observe

The problem. End-to-end LLM traces with trace_id propagated through every component, runtime metrics per tenant, online scoring (LLM judge over sampling), statistical drift, and safety / guardrails monitoring. Covered in the posts on AgentSight tracing, MCP observability with OTel and eBPF + drift.

OSS stack. Base standard: OpenTelemetry (specification + collector + SDKs) with the gen_ai semantic conventions that stabilised in 2025. Backends: Tempo or Jaeger for traces, Prometheus for metrics, Loki for logs, Grafana as the common UI. LLM-specific layer: Langfuse (self-hosted with an optional EE licence) and Phoenix Arize OSS. eBPF layer for low-level observability: Pixie, Hubble, and Cilium Tetragon for runtime security. Drift: Evidently AI, NannyML, Alibi Detect.

Hyperscaler equivalents. On AWS, CloudWatch (metrics + logs) + AWS X-Ray (traces) form the base, CloudWatch Application Signals adds OTel-compatible APM, and Amazon Managed Prometheus and Amazon Managed Grafana serve the plane if you want to keep Prom + Grafana without operating them. Bedrock logging integrates with CloudWatch and S3. ADOT (AWS Distro for OpenTelemetry) is the official collector.

On GCP, Cloud Monitoring + Cloud Logging + Cloud Trace + Cloud Profiler form the quintet, all OTel compatible. Vertex AI Model Monitoring offers drift detection (feature skew, prediction drift) integrated with runs.

On Azure, Azure Monitor + Application Insights + Log Analytics cover the APM stack with native OTel, Azure ML Model Monitor adds drift and data quality, and Azure OpenAI diagnostic logs enrich the traces with token and model metadata.

Summary table — Observe stage.

Functional piece	OSS on-premise	AWS	GCP	Azure
Traces (OTel)	OTel + Tempo / Jaeger	X-Ray + ADOT, App Signals	Cloud Trace	App Insights + Azure Monitor
Metrics	Prometheus + Grafana	CloudWatch + AMP / AMG	Cloud Monitoring	Azure Monitor Metrics
Logs	Loki, ELK	CloudWatch Logs	Cloud Logging	Log Analytics
LLM-specific (prompt, scores, sessions)	Langfuse, Phoenix Arize OSS	Bedrock logging + CW + custom	Vertex AI tracing + custom	App Insights + AOAI logs + custom
Drift detection	Evidently, NannyML, Alibi Detect	SageMaker Model Monitor	Vertex AI Model Monitoring	Azure ML Model Monitor
eBPF / runtime	Pixie, Hubble, Tetragon	— (no direct equivalent)	GKE Dataplane v2 / Cloud Service Mesh	Azure CNI + Defender for Cloud

Where the names mislead. The classic cloud APM tools (X-Ray, Cloud Trace, App Insights) do not understand prompt versioning or adapter id as native concepts. They accept the gen_ai.* attributes as dimensions, but the UIs do not prioritise those views. Langfuse and Phoenix do, because they are designed for LLM. In cloud, the usual pattern is to send dual: APM to the managed service for infrastructure + self-hosted Langfuse / Phoenix for the LLM plane. That pays off.

Stage 6 — Retrain + cross-cutting pieces

The problem (Retrain). Closing the loop feedback → triage → enriched dataset → new adapter, with a mixed cadence (quarterly + incident-driven). Covered in the Retrain post.

OSS Retrain stack. Orchestration: Apache Airflow, Prefect, Dagster or Argo Workflows and Kubeflow Pipelines for K8s-native. Feature store where applicable: Feast. Annotation and human-in-the-loop: Argilla, Label Studio, Trubrics. Structured feedback capture: your own Postgres table + Langfuse scores + Phoenix annotations. Closed-loop lineage: OpenLineage tying dataset → run → model → deployment → feedback → next dataset.

Hyperscaler Retrain equivalents. On AWS, SageMaker Pipelines orchestrates the cycle, SageMaker Ground Truth and A2I (Augmented AI) manage annotation and HiL, and SageMaker Model Monitor fires alerts that can invoke retrain pipelines. AWS Step Functions serves as a more general alternative orchestrator.

On GCP, Vertex AI Pipelines (based on Kubeflow Pipelines, compatible) orchestrates, Vertex AI Data Labeling Service annotates, Vertex AI Feature Store manages features, and Workflows or Cloud Composer (managed Airflow) act as orchestration alternatives.

On Azure, Azure ML Pipelines orchestrates, Azure ML Data Labeling annotates, Azure ML Feature Store manages features.

The problem (cross-cutting: prompt + data versioning). Making sure prompt_id, prompt_version and dataset_id, dataset_version propagate through the whole system and show up in spans, runs and metrics. Covered in the posts on prompt versioning and data versioning.

Prompt versioning equivalents. OSS: Langfuse Prompts, MLflow Prompt Registry. AWS: Bedrock Prompt Management (catalogue, versions, labels, integrated A/B testing) and SageMaker Prompt Hub. GCP: Vertex AI Prompt Management inside Vertex AI Studio. Azure: Azure AI Foundry Prompt flow and prompt versioning in Azure OpenAI deployments.

Data versioning equivalents. OSS: DVC + lakeFS (already covered under Data). AWS: S3 Versioning + Lake Formation + Glue Catalog (they are not DVC but together they cover part of it). GCP: Cloud Storage versioning + Dataplex (same). Azure: ADLS Gen2 versioning + Purview (same). The real gap here is that no hyperscaler offers DVC natively, so dataset-as-first-class-citizen operations still require a layer of your own.

Summary table — Retrain stage + cross-cutting pieces.

Functional piece	OSS on-premise	AWS	GCP	Azure
ML pipeline orchestration	Airflow, Dagster, Argo, Kubeflow	SageMaker Pipelines, Step Functions	Vertex AI Pipelines, Cloud Composer	Azure ML Pipelines
Feature store	Feast	SageMaker Feature Store	Vertex AI Feature Store	Azure ML Feature Store
Annotation / HiL	Argilla, Label Studio	SageMaker Ground Truth, A2I	Vertex Data Labeling	Azure ML Data Labeling
Feedback capture	Postgres + Langfuse scores	Bedrock + custom + Ground Truth	Vertex + custom	App Insights + custom
Prompt versioning	Langfuse Prompts, MLflow Prompts	Bedrock Prompt Management	Vertex Prompt Management	Azure AI Foundry Prompt flow
Data versioning	DVC + lakeFS + OpenLineage	S3 Versioning + Lake Formation	GCS + Dataplex	ADLS + Purview
Cross-system lineage	OpenLineage + DataHub	SageMaker Lineage Tracking	Dataplex lineage	Purview

The chatbot from the previous post ported to AWS

To stop the catalogue being abstract, we take the full scenario from the previous post, the multi-tenant support chatbot for insurers on an on-premise OSS stack, and describe it component by component with an AWS stack. It is not an executable migration; it is the map of what disappears, what appears and where the lock-in shows up.

The network plane. Edge LB and WAF: AWS WAF + CloudFront. Ingress to the cluster: AWS Load Balancer Controller over EKS. What was Cilium BGP + RKE2 is replaced by EKS with VPC CNI (or Cilium on EKS, which is possible). The conceptual equivalent of Tetragon is Amazon GuardDuty for EKS + optional Falco. Moderate lock-in: network control couples to VPC.

The chat gateway and auth. What was a bespoke API gateway with JWT verification materialises as Amazon API Gateway + Amazon Cognito (or IAM Identity Center if it is B2B). The gateway’s AI-aware routing is covered with Bedrock + per-customer tags, or with AWS API Gateway custom authorizers invoking a Lambda for tenant resolution. High lock-in on the identity layer if Cognito is chosen.

The inference engine. Three different options, with a clear trade-off.

Bedrock with a managed model (Claude / Llama / Mistral): all the operations of vLLM, K8s Operators, KV cache and disaggregated serving disappear. You move to Provisioned Throughput for a latency guarantee. You gain time-to-market; you lose control over prefill/decode, over custom LoRA adapters (Bedrock accepts Bedrock-managed fine-tunes but not arbitrary LoRAs), and you enter model lock-in (moving from Claude to Llama means changing API).
SageMaker Endpoints with your own vLLM container: you keep vLLM and its optimisations, but K8s disappears and SageMaker replaces it as the orchestration plane. Inference Components allow you to densify multiple adapters. KV cache, prefix caching and LoRA hot-swap work the same. Moderate lock-in on the SageMaker SDK and on the Inference Components format.
EKS with vLLM (the minimalist option): basically the OSS stack but with EKS instead of RKE2 and EBS/EFS instead of Ceph. Low lock-in, limited benefit from the cloud.

Data layer. The corpus moves to S3 with versioning, the embeddings to Amazon OpenSearch Service or to Aurora pgvector. The radical managed option is Bedrock Knowledge Bases: you upload documents to S3, it indexes them, it exposes a retrieval API. We remove Qdrant, we remove manual embedding pipelines, we remove part of Kafka + Flink. But control over custom reranking, fine-grained per-chunk ACL and the possibility of re-embedding with a new encoder of your own disappears. Bedrock KB uses the Titan or Cohere embedders available in Bedrock, and changing them means changing the whole index. ENS compliance: you have to validate that the buckets and the index live in EU regions and that the embedding model does too.

Stream + CDC. Kafka + Debezium is replaced by MSK + MSK Connect or by Kinesis + DMS. Schema Registry: Glue Schema Registry. The events remain functionally equivalent. Moderate lock-in if you go to Kinesis (Kinesis is not Kafka), low if you go to MSK (Kafka compatibility).

Data versioning. Here the gap is clear. S3 Versioning + Lake Formation + Glue Catalog is not DVC. To preserve the discipline of the previous post, (dataset_id, dataset_version, sha256_hash) propagated as an input artefact to the trainer, you can keep DVC over S3 (DVC works perfectly with S3 as a remote) or accept the limitation and register lineage manually in SageMaker Lineage Tracking. The first option preserves the practice; the second accepts degradation.

Tune stage. The LoRA adapter customer_support_v7 is trained with SageMaker Training Jobs on ml.p5.48xlarge instances (8× H100), using a standard HuggingFace + PEFT container. MLflow managed by SageMaker or your own MLflow on EC2 covers tracking. Alternative: if the black box is acceptable, Bedrock Custom Models with a dataset in S3 produces a fine-tuned Bedrock model without instantiating a GPU manually, at the cost of not being able to inspect the run.

Eval stage. Promptfoo + RAGAS in CI runs the same over CodeBuild. Bedrock Model Evaluation replaces a good part of the automatic eval suite. Bedrock Guardrails replaces NeMo Guardrails + Presidio + LlamaGuard, with the loss of transparency noted earlier.

Deploy stage. If Bedrock was chosen as the engine, this stage vanishes: Bedrock serves. If SageMaker Endpoints + vLLM was chosen, KServe is replaced by SageMaker Operators (or KServe is kept over EKS). The AI Gateway that in OSS was Envoy AI Gateway or LiteLLM becomes API Gateway + Bedrock or API Gateway + Lambda + SageMaker.

Observe stage. The OTel Collector remains the standard. Traces to AWS X-Ray + CloudWatch Application Signals. Metrics to Amazon Managed Prometheus. Logs to CloudWatch Logs + optionally OpenSearch for search. Langfuse is hosted on ECS Fargate or EKS because the cloud has no native equivalent of integrated prompt + traces + scores. Drift: SageMaker Model Monitor replaces Evidently / NannyML. eBPF (Pixie / Hubble / Tetragon) has no direct equivalent in managed AWS, so Falco or installing Tetragon on EKS is still the route.

Retrain stage. SageMaker Pipelines orchestrates the quarterly cycle. SageMaker Ground Truth + A2I replace Argilla. The feedback_signals table in Postgres stays as it is (RDS Postgres) or moves to DynamoDB for large scales.

How much the lock-in weighs. The component with the highest lock-in is Bedrock + Bedrock Knowledge Bases + Bedrock Guardrails: leaving that requires rewriting the inference plane and reindexing the whole RAG. Next is the SageMaker SDK (Pipelines, Endpoints, Training), where leaving costs but means rewriting scripts, not data. Data in S3 is portable (S3 → MinIO with rclone works). OTel observability is portable at almost no cost if the collector is kept as the abstraction. The auth gateway is the other high lock-in point if it uses Cognito.

What you gain. A dramatic reduction in GPU infrastructure operations, K8s patching, CUDA driver management, prefill/decode sizing, Ceph / MinIO management. A very short start-up curve: one request served in less than a sprint versus several weeks of bring-up for the OSS stack. Explicit provider SLAs.

What you lose. Contractual data sovereignty (the data stays in EU regions if configured that way, but the operator is a US third party under the Cloud Act). Visibility of the whole stack (Bedrock is a black box from the model downwards). Roadmap independence (the decision to discontinue a model, raise prices or change guardrails is not controlled by the customer). Fine-grained cost-per-token optimisation (the levers are the ones the provider exposes). For ENS customers under a HIGH declaration or NIS2 essential category, several of these points are non-compliance, not preference.

Master table: the entire parallel catalogue

Stage / component	OSS on-premise (the blog’s reference)	AWS	GCP	Azure
Data	DVC + lakeFS + MinIO + Qdrant + Kafka + Debezium	S3 + Lake Formation + OpenSearch / Aurora pgvector + MSK + DMS	GCS + Dataplex + Vertex Vector Search + Pub/Sub + Datastream	ADLS Gen2 + Purview + Azure AI Search + Event Hubs + ADF
Data versioning (cross-cutting)	DVC + lakeFS + OpenLineage	S3 Versioning + Lake Formation + Glue Catalog	GCS Versioning + Dataplex lineage	ADLS versioning + Purview
Tune	HF Transformers + PEFT + bitsandbytes + MLflow + Ray/Kubeflow	SageMaker Training + HyperPod + Bedrock Custom + SM Experiments	Vertex AI Training + Vertex Tuning + Vertex Experiments	Azure ML Training + Azure OpenAI fine-tuning + Azure ML + MLflow
Eval	DeepEval + RAGAS + Promptfoo + Langfuse Evals + NeMo Guardrails	Bedrock Model Evaluation + Bedrock Guardrails + SageMaker Clarify	Vertex AI Evaluation Service + Model Armor + Gemini safety	Azure AI Evaluation SDK + Content Safety (Prompt Shields, Groundedness)
Deploy	vLLM + KServe + LLM Operators + Envoy AI Gateway	Bedrock + SageMaker Endpoints (+ Inferentia / Trainium)	Vertex AI Prediction + Gemini API (+ TPU)	Azure OpenAI + Azure ML Endpoints (+ Maia)
Observe	OTel + Tempo + Prometheus + Loki + Langfuse + Phoenix + Hubble	CloudWatch + X-Ray + ADOT + AMP/AMG + SM Model Monitor	Cloud Monitoring + Cloud Trace + Vertex Model Monitoring	Azure Monitor + App Insights + Azure ML Model Monitor
Retrain	Airflow / Argo / Kubeflow Pipelines + Argilla + Feast	SageMaker Pipelines + Ground Truth + A2I + SM Feature Store	Vertex AI Pipelines + Data Labeling + Vertex Feature Store	Azure ML Pipelines + Data Labeling + Azure ML Feature Store
Prompt versioning (cross-cutting)	Langfuse Prompts + MLflow Prompt Registry	Bedrock Prompt Management + SM Prompt Hub	Vertex AI Prompt Management	Azure AI Foundry Prompt flow

When to choose each side — the real decision

The right question is not “OSS or cloud?”. It is stage by stage.

The OSS side wins by default when there is:

Data subject to ENS HIGH category, NIS2 essential sectors or equivalents (identifiable health data, regulated banking, critical infrastructure). Here provider traceability and the processing contract are not negotiable; using a service whose operator is subject to the Cloud Act, FISA 702 or equivalent compromises the legal basis.
A requirement for auditable inspection of the model, the guardrails and the whole pipeline. If a regulator asks “how exactly does it detect PII?” and an answer that ends in open code is mandatory.
Large volumes with stable workloads. Above a certain threshold of tokens per month, the cost of Bedrock / AOAI / Vertex diverges from the amortised cost of your own GPU cluster. The threshold depends on the workload, but it typically sits between 5 and 50 billion tokens per month for models in the Llama 70B range.
Roadmap independence as a priority. The day the provider discontinues a model or raises the price by 40 %, the organisation has to be able to ignore it.

The hyperscaler side wins by default when there is:

Critical time-to-market, an MVP in weeks. The operational weight of the OSS stack is too much for a project that has not yet proved product-market fit.
A small team with no SREs / MLEs specialised in GPU inference. Operating KServe + vLLM + KV cache + multi-tenancy is not trivial; if the team cannot sustain it, hosting is the way.
Variable workloads / unpredictable spikes. Bedrock on-demand and SageMaker serverless charge for what you use; your own cluster pays for the GPU whether it is busy or not.
A need for specific proprietary models (Claude, GPT-4.1, Gemini Pro) with no acceptable OSS equivalent for the case.

Mixed stages are frequent and reasonable. In practice, a common pattern in 2026 is: data, observe and retrain in self-hosted OSS (lineage and sovereignty), tune in OSS on your own cluster, eval in OSS + managed guardrails depending on the safety profile, deploy managed for proprietary models and self-hosted for open models. The question to ask for each stage is: “if the provider raises prices by 50 % or discontinues a component tomorrow, how much does it cost to move it?”. The parallel catalogue in this post gives the answer for each box.

What we have not covered (yet)

Some pieces still deserve a post of their own:

OpenAI / Anthropic API directly (not through Bedrock or AOAI): another level of managed, another contract.
Serious hybrids: AWS Outposts, GCP Distributed Cloud, Azure Stack HCI / Azure Local, the hyperscaler in your own room.
Per-tenant cost accounting compared OSS vs cloud: how the bill is built and where attribution breaks.
A real migration OSS → cloud or cloud → OSS: steps, scripts, gotchas.
Concrete European sovereignty: GAIA-X, EuroHPC, the European cloud offering (OVHcloud, Scaleway, IONOS, Aruba), compared with the big three for ENS / NIS2 cases.
AWS Inferentia / Trainium, GCP TPU v6 Trillium, Azure Maia: in-house chips and how they change the cost / token calculation.

References

vLLM documentation and vLLM Production Stack — the reference OSS inference engine.
Amazon Bedrock documentation — AWS managed model catalogue, Knowledge Bases, Guardrails and Prompt Management.
Amazon SageMaker AI — training, endpoints, pipelines, model monitoring.
Google Vertex AI documentation — training, prediction, evaluation, model monitoring.
Azure AI Foundry documentation — Microsoft’s unified plane for AI applications.
Azure OpenAI Service documentation — OpenAI models hosted on Azure.
OpenTelemetry GenAI semantic conventions — the standard that stitches observability across the OSS / cloud boundary.
Langfuse documentation and Arize Phoenix — the reference OSS LLM observability tools.
DVC and lakeFS — OSS data versioning.
NeMo Guardrails — OSS safety + dialog policy.
ENS (Esquema Nacional de Seguridad) and NIS2 (Network and Information Security Directive 2) — the compliance frameworks that have the last word in the OSS vs cloud choice for regulated EU customers.

Data versioning for LLMOps: DVC, lakeFS and the reproducible golden dataset challenge

Fri, 22 May 2026 11:00:00 +0200

TL;DR

The Data stage of the six-stage LLMOps pipeline has a silent link on which everything else depends: versioning the datasets with the same discipline the code is versioned with. It is not optional. An LLM system in production consumes at least four distinct types of dataset (training/fine-tuning, RAG corpus, golden eval set, enriched dataset from the Retrain loop) and each has its own demands. Git solves code but fails on data for two technical reasons (size and a useless binary diff) and one operational one (it does not propagate lineage as far as the weights bucket of the trained model). The two dominant OSS tools, DVC and lakeFS, unified in November 2025 under a single organisation with a roadmap oriented towards LLM training and RAG datalakes; they are still complementary projects (file-level vs whole-bucket branching) but now under common governance. The productive pattern the market has consolidated on: identify each artefact with an immutable (dataset_id, version), propagate the pair as far as the experiment tracking (MLflow / W&B), version the dataset’s schema too (not just its content), apply a strict holdout to the golden eval set so you are not measuring memorisation, and maintain bidirectional traceability dataset_version ↔ model_version ↔ deployment ↔ trace_id. Without this, the promise of “we can audit which model answered what” collapses at the first serious incident.

You are here: Data (with a cross-cutting effect on Tune, Eval and Retrain)

This post goes into the detail of the versioning link inside stage 1 · Data. Versioning belongs operationally to Data, but the artefacts it produces travel to Tune (training set), Eval (golden set) and Retrain (enriched dataset). That is why the diagram marks Data as active and shows a cross-cutting band indicating end-to-end lineage.

The master analogy: batch traceability in a serious factory

A serious pharmaceutical factory does not produce without batch traceability. Every box of pills carries a printed batch number; that batch is associated with manufacturing dates, with the specific batches of each raw material used, with the quality tests it passed, and with the technicians who signed off each step. If a patient reports an adverse effect, the factory can rewind within hours: this package → this batch → these raw materials → this shift → this production line → this quality control result. Without that chain, the incident is a permanent mystery.

A serious LLM system works the same way. The “package” is the answer a user saw in production. The “batch” is the combination of model, adapter, prompt, context and configuration that generated it. And the “raw materials” are the datasets: the training set the base model was trained on, the adapter’s fine-tuning dataset, the RAG corpus that feeds retrieval, the golden eval set that validates promotion. If a customer says “what data was the model trained on that answered X to my question Y on 14 March?”, without batch traceability the answer is “we do not know”. And that, with a customer under compliance pressure, kills the contract.

Git versions the recipe (the code). Data versioning versions the ingredients. Without both, there is no auditable factory.

The four artefacts worth versioning (with differentiated demands)

Not all datasets are versioned the same way or at the same frequency. The typical LLM system in production handles four artefacts worth governing separately.

Artefact	What it is	Typical size	New-version frequency	Who consumes it
Training / fine-tuning dataset	Input/output pairs (or conversations) that train the adapter or the model.	10⁴ – 10⁷ examples · 1 – 100 GB	Per Tune experiment	Trainer (Axolotl, TRL, Unsloth)
RAG corpus	Indexed documents that feed retrieval.	10⁵ – 10⁹ chunks · 10 GB – 10 TB	Almost continuous (streaming ingest)	Indexer + vector store
Golden eval set	Curated examples with an expected answer to measure quality.	10² – 10⁴ examples · MB	Per product release	Eval gates in CI
Enriched retrain dataset	Cases where the system failed + the human correction.	Hundreds to thousands per quarter	Per retrain cycle	Next Tune

All four have common requirements (immutable identity, lineage, schema) and relevant differences:

The training set tends to be large, stable per experiment, and the cost of an error is a lost experiment (expensive but bounded).
The RAG corpus is enormous, in continuous change, and its versioning is handled through periodic snapshots of the index (not of the raw text). Usually lakeFS or bucket branches; DVC is not the best fit.
The golden eval set is small but critical: errors here contaminate the whole promotion chain. Here the rigidity of the versioning matters more than anywhere else.
The enriched retrain dataset is incremental by nature: each Retrain cycle contributes a delta over the previous one. The new version does not overwrite; it inherits and adds.

Confusing them, treating the RAG corpus as if it were the training set, or the golden eval as if it were just another dataset, is the source of half the operational problems in data versioning.

Why Git is not enough

The obvious question: if Git already solves code, why does it not solve data too? Three reasons, two technical and one operational.

Reason 1: size. A Git repository with a 50 GB dataset becomes unmanageable. git clone pulls the entire history; git status walks every file; the pack file in .git/objects bloats to up to twice the size of the dataset. Git LFS solves the first part (the binary leaves the pack) but introduces its own complexity without addressing the other two reasons.

Reason 2: useless binary diff. Git assumes text diffs are useful. When a column changes in an 8 GB parquet, the diff is opaque, because the file is binary, compressed and columnar. You cannot do code review on a dataset change the way you do on a function change. You need a semantic diff: how many rows changed, which columns changed, which distribution moved. No native Git gives you that.

Reason 3: lineage that crosses repository boundaries. This is the most important and the most subtle. The training dataset lives in a bucket. The trainer code lives in a Git repo. The trained model is published to a model registry. Inference in production generates traces in an observability system. Connecting dataset_v3 → adapter_v7 → deployment_d2 → trace t_x9 requires propagating identifiers across four different systems, not inside one repo. Git has no opinion on this.

Data versioning tools (DVC, lakeFS, Pachyderm, Quilt) exist because they solve the three problems at once: they hang the data outside the Git repo, they offer some form of semantic diff, and they expose stable identities that can be propagated towards experiment tracking and the model registry.

DVC vs lakeFS before the unification

Until November 2025, the two dominant OSS tools coexisted as complementary approaches.

Axis	DVC	lakeFS
Mental model	“Git for data”	“Branching for the data lake”
Granularity	Individual file	Whole bucket (with per-branch namespacing)
Storage	Remote-agnostic (S3, GCS, Azure, MinIO, SSH)	S3-compatible (S3, MinIO, Ceph)
Workflow	`dvc add` + `dvc push` + `dvc.yaml` pipelines	`lakectl commit` + branches/merges over the bucket
Diff	File hash + external metadata	Object-level diff + commit log
Strong cases	Discrete training datasets, model files, reproducible pipelines	Large RAG corpora, branching a shared data lake, parallel experiments without duplicating data
Git integration	Deep (the `.dvc` files are committed to Git)	Tangential (lakeFS lives alongside)
Who operates it	MLE team	Data engineering team

In practice, many teams used them at the same time: DVC for the discrete datasets that fed an experiment (they fit in a Git repo thanks to the indirection of the .dvc pointers), and lakeFS for the large RAG corpus bucket they wanted to branch without duplicating terabytes.

What changed with the November 2025 acquisition

lakeFS acquired DVC in November 2025. The operational consequence as of May 2026 is modest but relevant:

There is (as yet) no technical merger of the projects. DVC is still DVC and lakeFS is still lakeFS. The current CLIs, formats and workflows have not changed.
An explicit combined roadmap towards LLM training and RAG datalakes. The merged organisation has stated specific priorities: consistent branching between the dataset and the trained model, native integrations with MLflow / W&B / Langfuse, support for the typical LLM formats (jsonl, parquet with embedded tokenisation), and branch-aware vector indexing.
Convergence expected in 2026-2027. The market anticipates a single registry with two operating modes (file-level + bucket-branching) under a unified CLI. As of today, teams still combine both.

The practical reading for 2026: adopt DVC for discrete training/eval datasets and lakeFS for the RAG corpus, but design the lineage so that a future unified registry can absorb both without re-versioning everything. Concretely: use stable identifiers (dataset_id, version, commit_hash) that can be propagated regardless of the tool.

The operational pattern: four-hop lineage

Once you accept that datasets have to be versioned, the question is not “which tool” but “which chain of identifiers connects production to the source data”. The pattern the market has consolidated on has four hops:

(dataset_id, dataset_version)
│ versioned in DVC or lakeFS
▼
(model_id, model_version)
│ registered in MLflow / W&B with the dataset as input
▼
(deployment_id, prompt_version)
│ registered in the model registry + prompt registry
▼
(trace_id)
│ emitted by the inference engine with OTel
▼
answer visible to the user

Each arrow is a metadata write that crosses the boundary between two systems. If a single arrow is missing, the lineage breaks and the promise of auditability evaporates.

A concrete example of the flow, using DVC + MLflow:

# Data stage: version the dataset
dvc add data/finetune_v3.jsonl
git add data/finetune_v3.jsonl.dvc data/.gitignore
git commit -m "data: finetune dataset v3"
dvc push # uploads the binary to the remote (MinIO/S3)

# Tune stage: train while recording lineage
mlflow run train.py \
 -P dataset_id=finetune \
 -P dataset_version=v3 \
 -P dataset_hash=$(dvc get-url data/finetune_v3.jsonl | sha256sum)
# The run records: input dataset + model output

# Eval stage: validate while recording lineage
mlflow run eval.py \
 -P model_id=adapter_customer_v7 \
 -P golden_set_id=customer_support \
 -P golden_set_version=v12

# Deploy stage: the deployment inherits the dataset + golden ids
# Every trace in Observe carries model_version + prompt_version
# which rewind back to dataset_version

The equivalent version with lakeFS over the RAG corpus:

# Branch for the new corpus embeddings
lakectl branch create lakefs://corpus/embed-2026q2 --source main

# Index the corpus on that branch
python index_corpus.py --branch embed-2026q2

# Validate before merging to main
python eval_retrieval.py --branch embed-2026q2 \
 --metric recall@10 --threshold 0.78

# If it passes, merge (changes the corpus that serves production)
lakectl commit lakefs://corpus/embed-2026q2 -m "embed: corpus 2026q2"
lakectl merge lakefs://corpus/embed-2026q2 lakefs://corpus/main

The virtue of the second flow: while the new corpus is being validated, the production system keeps serving from main with no interference. The parallel branch works as a real staging area over the whole bucket.

Schema contracts: data versioning without them is an illusion

Versioning a dataset’s content without versioning its schema is a frequent mistake. The problem: a versioned dataset with an implicit schema still breaks silently when a producer (the ingestion team, the annotation team, an ad-hoc script) changes a field.

A concrete case: a customer support golden eval set, 1000 examples, an expected_output field originally string. Someone decides they need to capture several valid answers and changes the field to list[string]. The eval loader accepts both formats by chance (Python is lax) but the downstream LLM judge receives a different object. The eval keeps passing but now it measures something else.

Productive pattern: the dataset is versioned with DVC/lakeFS and its schema is versioned with a Schema Registry (Confluent or Apicurio) or, in less mature systems, with a JSON Schema embedded alongside the dataset. CI blocks any PR that breaks the contract without a version bump.

A minimal schema for a golden eval entry (illustrative):

$schema: https://json-schema.org/draft/2020-12/schema
$id: https://example.org/schemas/golden_eval_entry/v3.json
type: object
required: [example_id, input, expected_outputs, rubric, segment]
properties:
 example_id: {type: string, format: uuid}
 input:
 type: object
 required: [user_query, retrieved_context]
 properties:
 user_query: {type: string}
 retrieved_context: {type: array, items: {type: string}}
 expected_outputs:
 type: array
 minItems: 1
 items: {type: string}
 rubric:
 type: object
 required: [must_include, must_not_include, format]
 properties:
 must_include: {type: array, items: {type: string}}
 must_not_include: {type: array, items: {type: string}}
 format: {enum: [text, json, markdown]}
 segment: {type: string}
 difficulty: {enum: [easy, medium, hard]}
 added_at: {type: string, format: date-time}
 curated_by: {type: string}

Operational rules:

Explicit forward/backward compatibility: adding an optional field is backward-compatible; removing a required one is breaking. The policy is enforced with a compatibility check in CI.
The schema version embedded in every row of the dataset (a _schema_version field). The loader validates that the version matches what the consuming code expects.
The schema registry as the single source of truth, not as an optional copy of the JSON Schema in four repos.

Without this level of discipline, “we have data versioning” means “we store the bytes but we do not control what they mean”.

Golden eval set: the most critical version

Of the four artefacts, the golden eval set is the one that demands the most rigour. A failure here contaminates the whole promotion chain: if the eval lies, the gates approve models that should not pass.

Three extra disciplines over the golden set:

Annotation with measured quality. Each example is labelled by a human, and a percentage (10-20 %) is annotated by two independent people. The inter-annotator agreement (Cohen’s kappa or pairwise F1) is measured and published; a golden set with kappa < 0.7 is measuring human noise, not model behaviour. Argilla and Label Studio provide the mechanics; what matters is the discipline, not the tool.

A strict holdout against contamination. The golden set must never enter the training set. The concrete mechanism: hash each golden set input (sha256 normalised by lowercasing + stripping trivial punctuation) → check in CI against all the training set hashes. If there is an intersection, CI blocks until it is resolved. Without this check, the model passes the eval by memorisation, not by capability. The consequence in production is disastrous: the “validated” model fails on cases analogous to the golden set that it had not memorised.

Additive versioning, never destructive. When the golden set grows (each retrain cycle adds cases), golden_v3 = golden_v2 ∪ new_examples. Never golden_v3 = a different new set. Only that way can you compare two models trained months apart over the same base + the new delta. If you rewrite the golden set, you cannot tell whether the March model was worse than the May one or whether you were simply measuring different things.

Summary table of the discipline per artefact:

Practice	Training set	RAG corpus	Golden eval set	Enriched retrain
Immutable versioning	Yes	Yes (snapshots)	Yes, critical	Yes
Schema with a contract	Yes	Recommended	Yes, critical	Yes
Double annotation	No	Not applicable	Yes (10-20 %)	Yes (10-20 %)
Holdout vs other datasets	N/A	N/A	Yes, hash check	Yes (vs golden)
Drift check vs previous version	Recommended	Yes	Recommended	Yes
Lineage to deployment	Yes	Yes	Yes	Yes

Promotion gates: the dataset is promoted like the model

A candidate dataset (a freshly enriched golden_v13, an enriched_retrain_2026_q2 resulting from the Retrain cycle) does not enter production just by being in the bucket. It goes through gates equivalent to the model’s or the prompt’s:

Schema validation — the contract is met. Blocked in CI if not.
Quality validation — a random 5-10 % sample reviewed by a human with a quality score ≥ 4/5. Blocked if the sample fails.
Holdout segregation check — for golden sets and enriched datasets, a hash check against all the other active datasets. Blocked if there is overlap.
Drift check vs previous version — a KS test over the distribution of the input embeddings, or simpler metrics (mean length, segment distribution, the ratio of each label). A warning if the drift is high with no documented cause; a block if it is very high.
Lineage check — the dataset explicitly declares which version it inherits from and what changed. Without that metadata, it does not go in.

Only when all five gates pass is the dataset tagged production-ready and the downstream pipelines that depend on it unblocked (the next Tune, the next product release, the next eval cycle).

The on-premise stack applied

On a generic infrastructure with an RTX 4090 (24 GB VRAM, a development / small-batch profile) and a 4×H100 SXM cluster (80 GB VRAM each, NVLink, training and production inference), data versioning fits without a dedicated GPU for the versioning itself, since versioning lives on CPU + storage, but it does touch the GPU for the drift checks that require embeddings.

Typical topology:

┌────────────────────────────────────────────────────────────┐
│ Object store (MinIO or Ceph) │
│ buckets: /training-sets /corpus-rag │
│ /golden-evals /enriched-retrain │
└────────────────────────┬───────────────────────────────────┘
│
┌─────────────────┼──────────────────┐
│ │ │
┌───▼────┐ ┌────▼────┐ ┌────▼─────┐
│ DVC │ │ lakeFS │ │ MLflow │
│ remote │ │ branches│ │ Tracking │
└───┬────┘ └────┬────┘ └────┬─────┘
│ │ │
└─────────────────┴──────────────────┘
│
┌──────▼──────┐
│ CI/CD gates │
│ (Forgejo / │
│ GitLab) │
└──────┬──────┘
│
┌──────────┴───────────┐
│ │
┌─────▼──────┐ ┌─────▼─────┐
│ RTX 4090 │ │ 4×H100 │
│ (drift │ │ (training │
│ embeds, │ │ + │
│ validates)│ │ serving) │
└────────────┘ └───────────┘

Operational notes:

The object store (MinIO or Ceph) serves both as the DVC remote and as lakeFS storage. One storage plane, two views.
The schema checks and holdout hashing are fast CPU-bound tasks; the CI runner executes them without a GPU.
The embedding drift check needs an encoder; the RTX 4090 handles this without touching the production cluster. A small encoder (BGE-small, E5-small, ~100M parameters) processes 10⁴ examples in a few minutes.
The H100 cluster stays free for training and serving, with no contamination from versioning jobs.

When do you NOT need DVC/lakeFS?

There is an opposite position defended with figures in the continuous fine-tuning post: for small systems with a single team, datasets < 1 GB and a handful of adapters, Postgres + pgvector + an S3 bucket + hashed filenames are enough. The operational complexity of DVC/lakeFS does not pay for itself.

The dividing line is reasonable:

You do not need DVC/lakeFS: a single team, small datasets, few adapters, no multiple products sharing data.
You do need it: multiple teams, datasets > 10 GB, several products sharing a golden eval set, external compliance demanding batch traceability, or an institutionalised quarterly retrain cycle.

Adopting DVC + lakeFS before you need them is overhead. Adopting them six months late is losing six months of lineage irrecoverably.

Seven pitfalls that turn data versioning into theatre

Versioning the data but not the schemas. The content is versioned, the contract changes silently, the system breaks without the versioning capturing it. A Schema Registry is not optional; it is half the problem.
The same S3 path overwritten. “Upload training.jsonl to the bucket” and the next experiment rewrites the file. S3 versioning (if it is enabled) saves your skin, but without an immutable identifier propagated to MLflow you cannot rewind. The correct pattern: training_v3.jsonl or training/2026q2/<sha>.jsonl, never the same name.
A golden eval set without a strict holdout. Without a hash check against training, the model memorises the eval and passes without having learned. It is the LLM equivalent of an exam the teacher announces in advance: everyone passes, nothing has been measured.
Not recording dataset → model lineage. When an incident requires knowing what data a certain model was trained on, the correct answer is a query to MLflow / W&B. If the answer is “let’s ask whoever trained it” (assuming they are still on the team), the lineage does not exist.
DVC added six months late. Adopting versioning in month 1 = a nuisance. Adopting it in month 6 = an irrecoverable loss of six months of datasets that can no longer be reconstructed. The curse of “we’ll add it later”.
lakeFS with branches that never get merged. Parallel branches over the corpus are useful for experimenting; kept indefinitely without merging, operations turn into a graveyard of half-updated branches. Explicit policy: merge or destroy within N weeks.
Schema validation only in production. The contract is validated when the dataset is already in production and the model already trained. By then, the incident has already happened. The validation has to be in CI, before the merge, over the delta the PR introduces.

A dataset’s cycle on one screen

┌─────────────────────────────────────────────────────────────┐
│ Producer (ingest / annotation / retrain loop) │
└────────────────┬────────────────────────────────────────────┘
│
▼ (commit a candidate version)
┌─────────────────────────┐
│ CI gates │
│ - Schema validation │
│ - Quality sampled │
│ - Holdout hash check │ ── fails → PR blocked
│ - Drift vs previous │
│ - Lineage declared │
└────────────┬────────────┘
│ passes
▼
┌─────────────────────────┐
│ DVC tag or lakeFS commit│
│ + MLflow registry │ ← immutable version
│ + Schema Registry │
└────────────┬────────────┘
│
▼
┌─────────────────────────┐
│ Downstream pipeline │
│ Tune / Eval / Deploy │
└────────────┬────────────┘
│
▼
┌─────────────────────────┐
│ Production trace │
│ → rewinds to dataset │
└─────────────────────────┘

What we have not covered

At top level, this post leaves out:

Vector store versioning proper: an embedding index is not versioned like a raw dataset because it depends on the embedding model. Changing the embedder rewrites the whole index. It is a different animal and deserves separate treatment (recall, ANN parameters, index branching vs full re-embedding).
Standardised lineage tooling (OpenLineage, Marquez): how to emit and consume lineage events interoperably between systems.
Data quality frameworks (Great Expectations, Soda, Deequ): how to write suites of “expectations” over a dataset and enforce them on every version.
Privacy-preserving versioning: federated learning without centralising the dataset, differential privacy applied to the version that gets distributed.
Contamination between third-party golden sets (HumanEval, MMLU, etc.) and the training datasets of open models: the problem of “the model passes HumanEval because HumanEval is in its pretraining”.

Each one is worth a post of its own when the field justifies it.

References

DVC documentation — versioning workflows, pipelines and remotes.
lakeFS documentation — branching, merging and commits over the bucket.
lakeFS acquires DVC, November 2025 — announcement and combined roadmap.
Confluent Schema Registry and Apicurio — schema contracts for streaming data.
OpenLineage and Marquez — open standard for lineage events.
Great Expectations — data quality expectations in CI.
MLflow Tracking — input datasets as first-class artefacts since MLflow 2.4.
Pachyderm and Quilt — historical alternatives to DVC/lakeFS.
On eval set contamination: “Stop Uploading Test Data in Plain Text” (Magar & Schwartz, 2022) and later work on detecting contamination in pretraining corpora.

Retrain: closing the loop between the production incident and the adapter that fixes it

Fri, 22 May 2026 07:45:00 +0200

TL;DR

The Retrain stage of the six-stage LLMOps pipeline is the one that closes the cycle. Without it, the deployed system is a project that ends; with it, it is a living practice that improves every quarter. The mechanics at the top level fit into five sequential sub-processes: capture feedback (explicit via thumbs + implicit via latency, abandonment, retries), triage incidents by root cause (model issue, retrieval issue, prompt issue, infra issue), enrich the dataset with the cases where the system failed plus the correct answer labelled by a human, decide the cadence (quarterly scheduled by default + incident-driven when a pattern crosses a threshold), and promote the candidate by taking it through Tune → Eval → Deploy with gates against the model in production. The tools the market has consolidated on in 2026: Langfuse for feedback collection in the UI, Argilla and Label Studio for human annotation of the enriched dataset, MLflow stages for promotion. The most lethal trap, and the most common, is the open loop: having all the pieces but no structured channel connecting them, so the Retrain stage boils down to “we will retrain when we need to” and therefore never.

You are here: Retrain (closes the cycle back to Data)

This post goes into the detail of stage 6 of the LLMOps pipeline. What follows takes apart the five Retrain sub-processes at complete top level, without descending into the internal mechanics of Tune (covered in the continuous fine-tuning post) or into the implementation of the eval suites (covered in the evals post).

The master analogy: the hospital mortality committee

A serious hospital holds periodic morbidity & mortality (M&M) meetings: the doctors review, without blame but without omitting anything, the cases where a patient died or had a serious complication. They look for root cause, identify patterns, adjust protocols, and leave a record. The committee does not meet when someone “remembers”; it is scheduled and mandatory. And when there is a catastrophic incident outside the cycle, an extraordinary M&M is convened within 48 h.

The Retrain stage is exactly that for an LLM system:

The morbidity is the minor incidents: answers the user marked with a thumbs-down, sessions where they retried the same question three times, examples where the eval score dropped but not below the alert threshold.
The mortality is the serious incidents: the system gave a dangerous answer, a key customer cancelled over a string of errors, the agent executed a tool it should not have.
The periodic meetings are the quarterly scheduled retrain: you look at the accumulated feedback, prioritise, and decide what goes into the enriched dataset for the next training run.
The extraordinary M&Ms are the incident-driven retrains: faced with a problematic pattern that crosses a threshold, a mini-cycle is fired off outside the cadence.

Without this discipline, incidents are anecdotes that get forgotten and the system does not learn.

Sub-process 1 — Feedback capture

The first link in the loop is observing what the system does badly. There are two families of feedback, and they complement each other.

Explicit feedback

The user tells you directly that the answer was bad. Mechanisms:

Thumbs up/down in the UI: the classic, low latency (1 click). Coverage: typically 1-5 % of traffic. Bias: users vote more when they are annoyed than when they are happy.
Annotation by power users: internal customers or experts who leave structured comments (“the answer is correct but the format does not follow our style guide”). Far lower coverage but high quality.
“What went wrong?” forms when the thumbs-down is clicked: predefined options (hallucination, format, tone, incomplete, off-topic) + optional free text. Enables automated triaging.
Re-editing: if the system writes a draft (an email, some code) and the user edits it before sending, that edit is rich feedback. The diff between what was generated and what was sent is an explicit signal of the failure.

All explicit feedback travels labelled with trace_id, prompt_version, model, user_id (anonymised where required), timestamp, and enters the feedback store. Langfuse, Phoenix and LangSmith have built-in UI for this; what matters is that every thumbs-down materialises as a row in a table, not as an event that gets lost.

Implicit feedback

The user tells you nothing but their behaviour gives the problem away. Typical signals:

Anomalous latency: the system’s TTFT was 8 s when the average is 800 ms. Indicates overload, heavy retrieval, an unexpectedly large prefill. Covered at top level in evals and ebpf+drift.
Abandonment rate: the user leaves the session before reading the full answer. If the ratio goes from 5 % to 15 % in one segment, something is wrong.
User retries: the user asks the same question (or a very similar one) 2-3 times. Indicates the first answer was no use to them.
Aborted sessions: the user closes the chat before the model finishes generating. In streaming, a high abort ratio is a strong indicator.
Workflow exit: in an agent, the user cancels the plan before execution. The agent’s trajectory did not convince them.
Statistical drift in the distribution of inputs or outputs (KS test, PSI, embedding-space shift). Covered at top level in eBPF + drift.

Implicit signals are noisier but cover 100 % of traffic, not the 1-5 % of explicit feedback. Combining them with explicit feedback gives the full picture.

Typical storage pattern

All the feedback, explicit and implicit, ends up in a common table with a minimal schema:

CREATE TABLE feedback_signals (
 signal_id UUID PRIMARY KEY,
 trace_id UUID NOT NULL,
 request_id UUID NOT NULL,
 signal_type VARCHAR NOT NULL, -- 'thumbs', 'retry', 'abandon', 'drift', ...
 signal_value JSONB, -- feedback payload (thumbs-down text, latency, etc.)
 prompt_id VARCHAR,
 prompt_version INT,
 model VARCHAR,
 user_segment VARCHAR, -- tenant, plan, geo
 occurred_at TIMESTAMPTZ NOT NULL,
 triaged BOOLEAN DEFAULT FALSE,
 triage_label VARCHAR -- filled in by sub-process 2
);

Postgres is more than enough for reasonable volumes (millions of rows a month). Langfuse uses Postgres underneath. For high volumes you can divert to ClickHouse or BigQuery, but it is rarely worth the extra complexity.

Sub-process 2 — Triage by root cause

Having feedback is not enough. You have to categorise each incident by its root cause before deciding what to do with it. Without triage, the enriched dataset is a junk drawer and the next retrain fixes nothing in particular.

The four canonical categories:

Category	Means	Typical action
Model issue	The model answered badly to something that was within its theoretical capability.	Candidate case for the enriched dataset for the next Tune.
Retrieval issue	The RAG did not retrieve the right context. The model answered reasonably from poor context.	Adjust reranker, chunking, indexing — the Data stage, not Tune.
Prompt issue	The system prompt does not cover the case, or covers it badly.	New version of the prompt (the cross-cutting stage of prompt versioning).
Infra issue	Latency, timeout, 5xx error, overload.	Adjust capacity / autoscaler — the Deploy stage.

Triage can be done:

Manually: a human (typically: the MLE team / data scientist) reviews the feedback in the Langfuse / Phoenix / LangSmith UI, looks at the complete trace, labels it. Cost: 2-5 min per incident. Sustainable up to around 50-100 incidents/week per person.
Assisted by LLM-as-classifier: an LLM classifies the incident into one of the four categories with a structured prompt. 80-90 % coverage automated, the rest escalated to a human. State of the art in 2026: GPT-5, Claude 4, Llama 3 70B-instruct with a careful prompt give F1 > 0.85 against calibrated internal rubrics.
Heuristic rules for the obvious ones: a 5xx error is always infra; latency > 5σ is always infra; a thumbs-down over RAG with context_relevance < 0.3 is retrieval. Captures 30-50 % of the volume at zero cost.

The productive pattern is: rules → LLM classifier → human, cascaded, escalating only what the previous level does not resolve with confidence.

New feedback
│
▼
[heuristic rules]
│
├── high confidence → automatic label
│
▼ (rest)
[LLM-as-classifier]
│
├── high confidence → suggested label
│
▼ (rest, or disagreement with the rules)
[human review]
│
└── final label → feedback_signals.triage_label

Sub-process 3 — Dataset enrichment

Once the incidents are triaged with a model issue label, those cases are candidates to enter the enriched dataset that will feed the next Tune. But they do not go in as they are: you need the correct answer labelled by a human.

How an enriched case is built

Each enriched case is a minimal tuple:

case_id: enrich-2026-05-22-0142
source_trace_id: trace-xyz
prompt_input:
 system: "You are a support assistant..."
 user: "I cancelled my order on Tuesday but I can still see the charge"
prompt_version_at_failure: customer_support_v3@v2
model_at_failure: llama-3-70b-instruct
failure_response: "Sorry for the inconvenience. The charge should be reversed within 5-7 business days."
human_corrected_response: "Sorry for the inconvenience. I have checked your account and I can see the refund was processed on Wednesday. It will show in your account within a further 24-48 h depending on your bank. Here is the refund ID: ABC123."
labeler: "support_agent_M3"
labeled_at: "2026-05-22T09:30:00Z"
quality_score: 4 # 1-5, eval by a second human before promoting to the dataset
notes: "The model gave a generic answer without checking the real state of the refund. It needs the account_lookup tool."

What matters is that the enriched case has enough context to be reproduced: original prompt, prompt version, model, failed answer, correct answer. Without this, the case is a loose data point, useless for training.

Annotation tools

Three dominant options in 2026:

Argilla (OSS, maintained by Hugging Face since 2024). Designed specifically for LLM datasets: annotation of (input, output) pairs, preference data (DPO/RLHF), instruction tuning. Python-friendly UI. Native integration with HuggingFace datasets and with MLflow.

Label Studio (OSS from Heartex). More generalist, also works for LLM. Rich, configurable, multi-modal UI. Better when the team already uses it for other tasks.

Langfuse built-in UI. Lets you annotate existing traces directly with thumbs + text + categorical labels. Useful for lightweight feedback; for building serious preference or instruction tuning datasets, Argilla and Label Studio are a better fit.

Typical pattern: Langfuse for traffic feedback + Argilla to build the formal enriched dataset that goes into the Tune pipeline. The traces marked as candidates in Langfuse are exported periodically to Argilla, where a human produces the correct answer and validates quality.

Quality validation before promoting

Not every annotated case enters the dataset. Good discipline requires:

Double annotation on at least 10-20 % of the critical cases (two independent annotators; if they disagree, a third resolves it).
Quality score per case (1-5 or equivalent): only cases with a score ≥ 4 enter the dataset.
Dataset versioning with DVC + lakeFS or equivalent, just like the rest of the Data stage datasets.
Reserved holdout: a portion of the enriched dataset is set aside to evaluate the retrained adapter, without it entering training. If the dataset is enriched with cases where the model failed and the same dataset is used to evaluate, you are measuring memorisation, not learning.

Sub-process 4 — Cadences: scheduled vs incident-driven

Once enriched dataset accumulates, it remains to decide when the retrain is launched. There are two complementary cadences.

Scheduled retrain (quarterly by default)

A process fixed in the calendar. Every quarter, in a specific week, the team:

Closes the accumulated feedback capture cycle.
Counts the enriched cases available (typically tens to hundreds per quarter).
Launches the fine-tuning pipeline with the aggregated dataset (golden dataset + the quarter’s enriched cases).
Evaluates the candidate against the full suite + the enriched holdout.
Promotes it if it passes the eval gates.

Advantages: predictable capacity planning, a closed budget, controlled risk, a team that is not burnt out. The default.

Incident-driven retrain

When a serious incident crosses a threshold, a mini-cycle fires off outside the cadence. Typical triggers:

Drift detected in the input/output distribution over threshold (KS p-value < 0.01, PSI > 0.25, embedding-space shift > 2σ).
A failing segment: a cluster of users or a type of question shows a 3× error rate over baseline for > 48 h.
A successful prompt injection or jailbreak attack that crosses the severity threshold (covered in guardrails).
External domain change: the customer changes policy, a new regulation comes out, and so on. The model trained against the old version stops being valid.

A typical mini-cycle: feedback from the last 7-14 days, a dataset focused on the problematic segment, fast fine-tuning on top of the existing adapter (not a full retrain), a segment-specific eval gate, canary deployment, promotion if it passes.

Cost: around 3-7 days of team work depending on severity. It is not optional for critical cases: if the failing segment is regulatory or reputational, the cost of not responding quickly is far higher than that of the mini-cycle.

Anti-pattern: “we will retrain when we need to”

The most lethal phrase in LLMOps. Without explicit scheduling, the scheduled retrain never arrives; without explicit thresholds, the incident-driven one never fires either. The system accumulates silent debt until a catastrophic incident forces a retrain that is already late.

The minimum discipline: a date in the calendar for the next scheduled run + 3-5 explicit incident-driven thresholds in writing. Without this, the Retrain stage is theatre.

Sub-process 5 — Promotion: the candidate enters production

Once the candidate adapter exists, it does not go straight into production. It goes through the same flow as any release: Tune → Eval → Deploy with gates.

Candidate adapter (from Tune)
│
▼
[full eval suite]
- historical golden dataset
- the quarter's enriched holdout
- regression vs production
│
passes? → no → block + alert
│
yes
▼
[non-regression eval gate]
- make sure it does not degrade
segments that already worked
│
passes? → no → block + alert
│
yes
▼
[canary deployment]
- 5-10% of traffic to the new
adapter for 24-72 h
- online metrics vs production
│
metrics OK? → no → rollback
│
yes
▼
[full promotion]
- move the label in the model registry
- MLflow stages: Staging → Production
- The previous one goes to Archived (preserves
historical reproducibility)

The registry tools:

MLflow Model Registry stages (Staging, Production, Archived) is the canonical pattern. Promotion is an API call: mlflow.models.transition_stage(name, version, "Production"). Audited, revertible.
A private Hugging Face Hub with a repo per adapter is the “Git for models” equivalent: versioned by commit hash, branches for staging/production, deploy via PR.
vLLM multi-LoRA hot-swap (described in continuous fine-tuning) loads the new adapter without restarting the server, so the physical promotion takes seconds.

Applied to typical on-premise hardware

Retrain as a stage does not need big hardware. The calculation:

Feedback collection: a small table in Postgres. Trivial on any node.
Manual / assisted triage: the LLM-as-classifier runs on the same inference engine that serves production, during low-demand hours, at spot priority. Tens of thousands of incidents a month consume on the order of minutes of GPU per day.
Dataset enrichment: human annotation, no GPU cost. Negligible storage.
Tune (mini-cycle or quarterly): here there is a cost. Fine-tuning a LoRA adapter over Llama 3 70B with a dataset of a few thousand examples costs on the order of 2-8 hours on a single H100. On 4 H100s with tensor parallel: 30-90 min. It fits comfortably into any low-demand overnight window.
Full eval suite: minutes on an engine with prefix caching active (covered in pagedattention deep-dive).
Canary deployment: zero additional cost, since the new adapter coexists in the same engine via multi-LoRA hot-swap.

For an RTX 4090 serving Llama 3 8B with a small team: a monthly or quarterly scheduled retrain overnight, an enriched dataset of 50-100 cases per cycle, annotation with self-hosted Argilla on the same node. Plenty.

For a 4×H100 SXM cluster serving several tenants: an enriched dataset segregated by tenant (each with its own holdout and eval suite), a retrain pipeline orchestrated with Argo Workflows or equivalent, a centralised MLflow registry, multi-LoRA hot-swap per tenant.

Common operational traps

The open loop. The system captures feedback, saves it to a table, and there it dies. Nobody triages, nobody enriches, nobody retrains. The deployed model ages silently. Fix: an explicit internal SLO (for example, “any feedback more than a week old without triage gets reported in standup”), with an assigned owner.

Human feedback that gets lost. Thumbs-downs with no structured capture (the event is logged but the reason is not), or the reason is logged but nobody indexes it for queries. Fix: an explicit schema like the one above, a weekly dashboard of “top thumbs-down reasons”.

Undefined cadence. “We will retrain when we need to” — never. Fix: a date in the calendar + 3-5 written thresholds.

No holdout test set. The enriched dataset is mixed with the golden dataset both to train AND to evaluate. The adapter looks like it improved because it “memorised” the enriched cases, but it generalises badly to new similar cases. Fix: a holdout reserved before training; the eval against the holdout is the metric that decides promotion.

Ad-hoc triage by one person. The senior data scientist triages when they can; while they are on holiday it piles up; they come back and give up because there are 400 incidents waiting. Fix: automate 70-80 % with an LLM-as-classifier, leave only the hard part to a human; rotate the “triage oncall” so one person does not get saturated.

Promotion without canary. The adapter passes the offline eval and is deployed to 100 % directly. A regression in production takes a while to be detected, until the online metrics show it, and by then the damage is done. Fix: a mandatory 5-10 % canary for 24-72 h.

No reproducibility of the original incident. The team goes to investigate why the model failed in the 22 May incident and discovers the prompt was different (it was changed two weeks ago), the model was too, and the logs did not keep the RAG context. Fix: strong traceability (covered in prompt versioning and MCP observability). Without reproducibility, retrain is guesswork.

The enriched dataset contaminates the Eval data. The team confuses “cases where it failed” (which go into the enriched training set) with “the golden regression dataset” (which has to stay stable to detect drift). Mixing them invalidates the eval. Fix: two different datasets, two different routes.

Recommended operational pattern: the quarterly cycle on one screen

A serious team with Retrain properly implemented has this flow every 3 months:

Week 1 (cycle close): new capture is frozen for the cycle, a snapshot is taken of the accumulated feedback. Automated report: how many thumbs-downs, how many incidents triaged, the distribution by category, top patterns.

Week 2 (triage and annotation): the MLE team plus annotators process the untriaged model issue cases. Human annotation in Argilla. Cross-validation on samples.

Week 3 (training and eval): the pipeline is launched with dataset = golden + this_quarter’s_enriched - holdout. Fine-tuning the adapter overnight. Eval against the full suite + holdout. If it passes the gates, candidate v_new.

Week 4 (canary and promotion): deploy the candidate as an alternative adapter in vLLM, route 5-10 % of traffic to the candidate for 48-72 h. Online metrics: latency, complaint rate, implicit eval in production. If everything is OK, full promotion; if not, rollback and analysis.

Week 5+ (next cycle): the v_new adapter is now production. Feedback capture for the next quarter begins. The previous v_old goes to Archived but stays accessible for historical reproducibility.

Quarterly, that cycle plus whatever incident-driven mini-cycles show up in between. Operational, predictable, auditable.

What we have not covered (upcoming posts)

Online DPO and continuous on-policy learning: how the cycle shortens to hours or days (Fast-Slow Chasing, iterative RLOO). 2026 state of the art, still emerging in production.
Machine unlearning for GDPR: when a user exercises the right to be forgotten and their interactions formed part of the enriched dataset of an adapter in production. Negative LoRA, selective retraining.
Constitutional AI runtime: continuous alignment that replaces or complements periodic retraining.
Eval gates with metamorphic testing: evaluating robustness against input perturbations (typos, paraphrasing, language) as part of the promotion gate.

References

Argilla documentation, Building Datasets for LLM Fine-Tuning: https://argilla.io/docs.
Label Studio documentation, LLM Annotation: https://labelstud.io/templates/llm.
Langfuse documentation, User Feedback and Dataset Management: https://langfuse.com/docs/scores/user-feedback.
MLflow Model Registry stages: https://mlflow.org/docs/latest/model-registry.html.
Ethayarajh et al., KTO: Model Alignment as Prospect Theoretic Optimization (2024) — reference for the feedback cycle as an alignment signal.
Google Cloud, Continuous Training and MLOps for GenAI (2025).
DataRobot, MLOps Best Practices: Closing the Loop (2025).
Eugene Yan, Feedback Loops in LLM Systems (blog, 2025).

Prompt versioning: the contract that stops a five-word change sinking your system

Fri, 22 May 2026 07:30:00 +0200

TL;DR

In a classic software system, the most dangerous line a team can change is a SQL migration. In an LLM system, it is a line of prompt. The prompt determines the output as much as the model or more, it does not show up in unit tests, it does not appear in the logs by default, and if it is changed without leaving a trail there is no way to know which version generated which response. Prompt versioning is the discipline that turns the prompt into a first-class artefact: with a unique identifier, history, deployment labels, an associated eval suite, and per-request traceability. The field has settled on three primitives (immutable version, mutable label, read cache) and two dominant tools (Langfuse OSS with a built-in UI, MLflow Prompts integrated into the registry since MLflow 3.10). This article covers the pattern at first level: why it matters, how it is implemented, which tool to choose, and how it fits with Eval, Deploy and Observe.

You are here: cross-cutting (touches Data, Tune, Eval, Deploy and Observe)

Prompt versioning does not live in one stage, it cuts across five. It appears as a cross-cutting component in the master map of the six-stage LLMOps pipeline precisely for that reason: the prompt version is necessary metadata at every stage, not the responsibility of a single one.

The master analogy: the prompt is an invisible SQL migration

A serious backend team would never accept someone modifying a column directly in production without going through a versioned migration. Even if the change “works” at the time, without a migration there is no way to:

Reproduce the previous state if something fails.
Know who applied the change and when.
Apply the same change in staging before prod.
Test the new version against an automated suite before promoting.
Know, two months later, why the table has the shape it has.

The LLM prompt occupies exactly that position in an inference system. Changing "You are a helpful assistant." to "You are a helpful, concise assistant. Answer in fewer than 3 sentences." can:

Cut the average cost per response by 30 % (the responses are shorter).
Or degrade quality in a segment where concision breaks necessary nuance.
Or change the distribution of tools the agent decides to invoke.
Or alter the behaviour of the downstream LLM judge that assumes a certain length.

And most importantly: if the change is made by editing a constant in the app code and deploying, when two weeks later someone asks “why did the complaint rate go up in the finance segment?”, there is no way to know which prompt was being served at each moment. The logs store the response and, with luck, the model invoked; the prompt is rarely stored explicitly.

Prompt versioning solves the same problem Flyway/Liquibase/Alembic solved for SQL: turning an invisible change into an auditable artefact.

The three primitives of the pattern

Whatever the tool, the systems that work in 2026 share three operational primitives worth fixing before looking at products.

1. Immutable version

Every time the prompt content changes (template, system message, available variables, recommended model parameters such as temperature), a new version is generated with a unique identifier. The version is immutable: once created, it is not overwritten; if something needs changing, v+1 is created.

prompt_id: customer_support_v3
versions:
v1 (2026-03-12): "Eres un asistente de soporte..."
v2 (2026-04-08): "Eres un asistente de soporte... formato JSON..."
v3 (2026-05-21): "Eres un asistente de soporte... formato JSON... 3 frases máx..."

Immutability is what allows a trace from two months ago to be reproduced: if the trace says “customer_support_v3@v2 was served”, version v2 exists literally and can be reloaded.

2. Mutable label (deployment alias)

Versions are immutable, but which version is in production changes. That decision is materialised in labels: pointers with a semantic name (production, staging, canary) that point at a specific version and can be re-pointed.

prompt_id: customer_support_v3
labels:
production → v2 (servida al 100% del tráfico)
canary → v3 (servida al 5% del tráfico via gateway)
staging → v3

Promoting a version is moving a label, not editing the prompt. Rollback is moving the label back, not copying text. The operation reduces to an atomic mutation of a (label, version) tuple.

3. Read cache

The prompt is read on every request to the model. If every read calls the prompt registry service, you add latency and a dependency. The standard solution is a local cache in the client (TTL on the order of minutes) that invalidates when the label changes or when the TTL expires.

Langfuse implements a native client cache with configurable TTL and lazy invalidation; MLflow Prompts leaves the responsibility to the client or to a gateway layer. In both cases, in production the client serves the prompt from memory with negligible overhead (<1 ms), and only goes to the registry when it refreshes.

┌──────────────────┐
│ Client (app) │
│ - local cache TTL=60s
│ - lookup label "production"
│ - gets template
│ - renders variables
│ - sends to LLM
└─────────┬────────┘
│ (when the TTL expires or a change event arrives)
▼
┌──────────────────┐
│ Prompt registry │
│ - Langfuse / MLflow
│ - GET label="production"
│ - response: version_id + template
└──────────────────┘

With these three primitives, any reasonable tool is equivalent in the essentials. What distinguishes one from another is UI, integrations, RBAC, eval integration, and so on.

The two dominant tools in 2026

The field has converged on two main options. Any serious production deployment uses one of the two (sometimes both, for different teams).

Langfuse (OSS, prompt-management UI built in)

Langfuse is the prompt-first system: it was born for tracing and observability, and prompt management is one of its core layers. Key features for versioning:

Built-in UI to create, edit and version prompts. Versions are generated automatically on save; the history is visible and diffable.
Arbitrary labels beyond the usual ones (production, latest). You can define eu-prod, internal-only, customer-a for fine-grained routing.
Native client cache in the official SDKs (Python, JS), with configurable TTL, event-driven invalidation and fallback to the last-known-good if the registry is down.
Native tracing integration: when you record a call to the LLM, Langfuse automatically associates the prompt_id@version that served it. In the UI you see: this trace, this span, this prompt version X.
Eval integration: Langfuse lets you register eval suites that fire when a new prompt version is created. The results stay linked to the prompt_id@version and are the natural gating for promoting staging → production.
Self-hosted or cloud: the core is OSS (MIT), runs on Docker compose or Helm; the cloud version adds SLA, SSO and support.

When Langfuse suits:

Teams that want a rich UI so product/PM/analyst can manage prompts without touching code.
OSS-first deployments where control of the runtime and of persistence is a requirement (on-premise, ENS).
When LLM observability is already in Langfuse: prompt management is marginal in setup.

MLflow Prompts (included in MLflow 3.10, March 2026)

MLflow Prompts is the classic MLOps ecosystem’s answer for LLMs. Features:

Integrated into the MLflow Model Registry: prompts are first-class registry artefacts, with the same stage semantics (Staging, Production, Archived) MLOps teams already know.
API consistent with the rest of MLflow: mlflow.register_prompt(), mlflow.load_prompt(name, stage="Production"). The learning curve for teams already using MLflow for models is nil.
Automatic versioning with a numeric version_id (1, 2, 3, …) and optional comments on promotion.
No built-in UI dedicated to prompts (the MLflow UI does the job, but it is designed for models; the flow is less polished than in Langfuse).
No native GenAI-aware tracing (MLflow Tracing in the 3.10 GenAI dashboard provides it, but the trace↔prompt integration is more manual than in Langfuse).
Compatible with any model registry backend MLflow supports (filesystem, Postgres, MySQL, S3, GCS, Azure Blob).

When MLflow Prompts suits:

Teams already operating MLflow for classic ML who want to extend the same discipline to LLMs without adding vendors.
Deployments where the centre of gravity is the model registry and the prompt is one more artefact.
CI/CD pipelines that already speak MLflow (CLI, REST API).

Comparison

Feature	Langfuse	MLflow Prompts
Core licence	MIT (OSS)	Apache 2.0 (OSS)
Prompt-first UI	✅	⚠️ via Model Registry
Immutable versioning	✅	✅
Mutable labels	✅ (arbitrary)	✅ (Staging/Production/Archived)
Native client cache	✅	❌ (DIY)
Integrated tracing	✅ native	⚠️ via MLflow Tracing
Eval gating on promotion	✅	⚠️ DIY with MLflow Recipes
Easy self-hosting	✅ Docker/Helm	✅ standard MLflow
Curve if you come from MLOps	medium	nil
Curve if you come from DevOps	nil	medium

In May 2026, the most widespread hybrid pattern is to use MLflow for the models+adapters registry and Langfuse for prompts+tracing, connected by trace_id and prompt_id travelling in the OpenTelemetry span attributes. Covered in evals and MCP observability.

Minimum schema of a versioned prompt

Whatever the tool, what the registry stores in each version has a reasonable minimum schema:

# prompt_id: customer_support_v3, version: 3
template:
 system: |
 Eres un asistente de soporte de {{company_name}}.
 Responde en español neutral, máximo 3 frases.
 Formato de respuesta: JSON {"answer": "...", "needs_human": bool}
 user: |
 Pregunta del cliente: {{user_message}}
 Contexto del ticket: {{ticket_context}}

variables:
 required: [company_name, user_message, ticket_context]
 defaults: {}

recommended_params:
 model: "llama-3-70b-instruct"
 temperature: 0.3
 max_tokens: 300
 response_format: "json_object"

metadata:
 author: "mlops@empresa.com"
 created_at: "2026-05-21T14:23:00Z"
 commit_message: "Añade límite de 3 frases tras feedback ticket #1842"
 eval_suite: "customer_support_v3_evals"
 related_traces: ["trace_id_x", "trace_id_y"]

This is the minimum contract. What sets a serious deployment apart:

variables.required is validated in the client before sending to the model. A missing variable blows up at client time, not in a confusing model response.
recommended_params.model ties the prompt version to a model. Changing model opens a debate (does the new version work with Llama 3 70B and with GPT-4o?). If it is not tied, the model is one more variable that wrecks reproducibility.
metadata.eval_suite is what the eval suites hook into: on creating v3, MLflow/Langfuse fires customer_support_v3_evals automatically.

Integration with eval gates: governed promotion

The real value of prompt versioning appears when it is integrated with eval. The canonical pattern:

Developer edits the prompt in the UI (Langfuse) or via the API (MLflow). v4 is created.
Automatic trigger: the prompt_created event fires the associated eval suite (the eval_suite in the metadata).
The suite runs against the golden dataset (questions+answers labelled by a human). Covered at first level in the evals post.
Results are attached to the version: v4 now has eval_score: 0.84, regression_vs_v3: -0.03.
Promotion gate: if eval_score >= threshold and regression < tolerance, the staging label moves to v4 automatically. If not, it alerts the developer.
Manual promotion to production: with the eval passed, someone with permission moves production from v3 to v4. Atomic, auditable, reversible.

Developer edits prompt → v4 created
│
▼
[eval suite trigger]
│
▼
Golden dataset 200 examples
│
▼
score = 0.84 (vs 0.87 for v3)
│
├── If it passes the threshold → label staging → v4
│ └── Manual promotion to production after review
└── If it does not pass → block + alert the developer

This flow turns the prompt change from “someone touched the code and we prayed” into “a prompt change is a PR that passes CI”. It is the same discipline classic MLOps applied to models.

Per-request traceability: which version served each response

The last piece is operational traceability: given a model response in production, which prompt version generated it?

The pattern is to propagate the version as a span attribute in OpenTelemetry, following the gen_ai.* semantic conventions we covered in MCP observability:

# In the client (common pseudo-code)
prompt = registry.load("customer_support_v3", label="production") # v3 → v_id=14

with tracer.start_as_current_span("llm_call") as span:
 span.set_attribute("gen_ai.prompt.id", "customer_support_v3")
 span.set_attribute("gen_ai.prompt.version", "14")
 span.set_attribute("gen_ai.prompt.label", "production")
 span.set_attribute("gen_ai.request.model", prompt.params.model)

 response = llm.complete(prompt.render(user_message=msg), **prompt.params)

 span.set_attribute("gen_ai.usage.input_tokens", response.usage.input)
 span.set_attribute("gen_ai.usage.output_tokens", response.usage.output)

In any trace (Langfuse, Phoenix, Jaeger, Honeycomb) you can see which exact version served that response. In an incident (“customer X received this on 22 May”) you reproduce literally the version and the model that generated the output.

Without that traceability, the incident stays an anecdote; with it, it is debuggable.

Applied to typical on-premise hardware

Prompt versioning is a computationally light layer compared with the inference engine or the fine-tuning pipeline. Its requirements:

Storage: the prompt registry typically weighs megabytes (hundreds to thousands of prompts with their versions). Postgres with a prompts(id, version, template, params jsonb, metadata jsonb, created_at) schema is more than enough. Langfuse uses Postgres by default; MLflow uses it for metadata (the blobs go to object storage or the filesystem).
Registry compute: a small instance (1-2 vCPU, 2 GB RAM) serves tens of thousands of reads per minute if the client cache is enabled. Without a cache, it scales linearly with QPS but is still trivial.
Triggered eval compute: here there is a real cost. Every time a new version is created, the eval suite runs. If the suite does LLM-as-judge over 200 examples and each eval costs 4 K tokens, one promotion costs on the order of 1 M tokens, minutes on a decent cluster, seconds if the suite already has its hot prefix cache.

For an RTX 4090 serving Llama 3 8B with a self-hosted prompt registry (Langfuse or MLflow): the registry runs on the same node in a sidecar container, the local app caches in RAM, the eval triggers run against the same inference engine at low priority. Full setup in a morning.

For a 4×H100 SXM cluster serving a large model to several tenants: registry in a dedicated K8s pod with replicated Postgres, eval suites running in pods with the spot priority class (covered in the cluster as a platform), OTel tracing propagating prompt_id+version to a central Langfuse.

Traps and things that are not what they look like

Prompts hardcoded in the app code. The most common antipattern. The prompt lives in a prompts.py file or templates/customer.txt that gets deployed with the app. There is no real versioning (git history is not a substitute: it does not tie commit ↔ production trace operationally). Migrating to a registry is 1-2 sprints of work; it is worth every hour.

Badly calibrated cache. A TTL of hours with a mutable label means a rollback takes time to propagate. A TTL of seconds overloads the registry. The reasonable default is 60-300 seconds with event-driven invalidation (the registry emits a message to Kafka/Redis when a label changes, and clients invalidate immediately).

Unvalidated variables. The template uses {{user_name}} but the app passes {{username}}. The render produces a prompt with a literal {{user_name}}. The model answers something bizarre and nobody knows why. Validating required variables in the client before sending to the model is the minimum discipline.

Prompts inside chains evaluated at runtime. If your stack uses LangChain, LlamaIndex or similar with chains that compose prompts at runtime, the final prompt the model sees may not be in the registry because it was composed from several fragments. Solutions: either register the chains as artefacts, or log the effective composed prompt in every trace.

Eval suite not hooked to the prompt_id. Without that link, a prompt change gets promoted without passing evals. The integration has to be a field in the prompt metadata (eval_suite: ...) that the system reads and fires automatically. If it depends on the developer “remembering”, the pattern will fail.

Non-existent RBAC roles. Anyone with access to the UI can move production to any version. Without separating editor (creates versions) from releaser (moves production labels), a junior developer can break production with an accidental promotion. Langfuse Enterprise has granular RBAC; MLflow has it via the server backend with per-experiment/registry permissions.

Prompts with sensitive data inline. The prompt template includes few-shot examples with real names, addresses, customer IDs. The registry stores that indefinitely. Under GDPR, the right to erasure applies to the registry too. Good practice: variables for sensitive data, not inline; periodic audit of the registry content.

Recommended operational pattern: the cycle on one screen

A serious team with prompt versioning properly set up has the following cycle, repeatable and cheap:

Developer opens a PR in the repo: changes the app code if necessary, but does not touch the prompt there.
Edits the prompt in the Langfuse/MLflow UI: creates v_new. Adds a commit message (“adds a 3-sentence limit after feedback on ticket #1842”).
The eval suite fires automatically: it runs against the golden dataset, results appear in the UI within minutes.
If it passes eval: the staging label moves to v_new automatically. The developer can test staging with controlled traffic.
Human review (1-2 people, optional depending on severity): approval.
Promotion to production: move the label, atomic. The client caches for 60-300 s, then serves the new version.
Observe: in Langfuse/Phoenix, production metrics and eval scores are segmented by prompt version. If the score drops with v_new, alert.
If there is a serious regression: rollback is moving the label back. A 5-second operation.

Every step is audited, every decision leaves a trail, every rollback is an atomic operation. This is what separates a GenAI system from “demos that worked once” and makes it operable for years.

What we have not covered (upcoming posts)

Automatic prompt optimisation: techniques such as DSPy, TextGrad, PromptBreeder that generate prompt candidates and optimise them against a measurable objective. The extension of versioning where the “developer” can be an optimiser.
Prompt injection and red teaming: integrating versioning with the adversarial evaluation flow. Partially covered in guardrails.
Different versions per tenant: when the same prompt_id needs variants per customer (i18n, branding, domain). Fork + override pattern.

References

Langfuse documentation, Prompt Management: https://langfuse.com/docs/prompts/get-started.
MLflow 3.10 release notes, Prompts in Model Registry (marzo 2026): https://mlflow.org/releases/3.10.
OpenTelemetry, Semantic Conventions for Generative AI (estables desde 1.36 de OTel): https://opentelemetry.io/docs/specs/semconv/gen-ai/.
Google Cloud, Prompt Management: Best Practices for Production LLM Systems (2025).
Chip Huyen, Designing Machine Learning Systems — capítulo sobre model registry y prompt-as-artifact (2ª edición, marzo 2026).
Eugene Yan, Prompt Engineering as Software Engineering (blog, 2025).

The GPU cluster as a platform: turning a shared cluster into a multi-tenant service your teams can consume

Thu, 21 May 2026 07:15:00 +0200

TL;DR

Having an expensive GPU cluster and many different workloads that want to use it is not an infrastructure problem: it is an internal product problem. What separates “we have a cluster” from “we have an inference platform” are four layers the market consolidated in 2026: a gateway layer that centralises authentication, routing and policies (LiteLLM, Portkey, Kong AI Gateway); a GPU isolation model appropriate to the tenants’ profile (MIG hardware isolation for untrusted multi-tenant, MPS for processes from the same team, time-slicing only for dev); a quota and rate limiting system with budgets per tenant/team/project (LiteLLM does this in its core at team/user/api-key level with descriptive 429s); and a multi-tenant observability plane that enables real cost attribution (showback as an intermediate step, chargeback as the destination), per-tenant tracing and differentiated dashboards. Applied to a typical mid-scale GPU cluster (one node with 4-8 H100 SXM and NVLink, a common place to start in production), this translates into concrete decisions: with ~640 GB of aggregate VRAM across 8 GPUs and two typical production models (a large 70B+ model with tensor parallel and a replicated mid-sized model), the cluster serves between tens and low hundreds of simultaneous sessions depending on the mix; GPU isolation is usually resolved with MIG on smaller workloads and per-model dedication on large ones; and the platform’s success metric is effective utilisation, which in typical production sits at 30-40% and where the reasonable optimisation target is to push it to 60-70% without degrading SLAs.

This is the fifth post in the MLOps for LLMs series. It is the most operationally oriented one and cuts across several pipeline stages (Deploy + Observe plus cross-cutting concerns). The “you are here” marker flags the two active stages because the notion of a multi-tenant platform does not live in just one.

You are here: Deploy + Observe (the cluster as a product)

The question that changes the framing

When a platform team buys expensive GPU hardware and starts building inference, the first version is almost always single-purpose: one model, one client, one target latency. It works. When the second team turns up asking for the same resource, that single purpose becomes internal politics: how many replicas do we give them? What do we do if the SLAs clash? Who pays for the tokens of team B’s experiment? And when the third one arrives, what was an SRE project becomes an internal product project.

The distinction is not technical, it is one of framing. A cluster is infrastructure. A platform is a service with customers, contracts and success metrics. The change of framing implies:

Identifiable customers (tenants), not anonymous users.
Contracts (latency SLA, guaranteed throughput, available models), not “whatever we manage”.
Success metrics that are not technical but product ones: adoption, satisfaction, cost per query per tenant, time to the first “hello world”.

This post walks through how that change of framing is carried out. It lands it on a mid-scale cluster (4-8 H100 SXM with NVLink in a single node), the usual configuration when starting with serious LLM inference; but the principles generalise to any topology, from a single node with two GPUs to multi-node clusters with InfiniBand.

The four layers of a multi-tenant inference platform

The canonical architecture that settled in 2026 has four layers that any serious multi-tenant platform implements, from the outside in:

Each layer solves a concrete problem. Let us take them one by one.

Layer 1 — AI Gateway: the single front door

The AI Gateway is the component your tenants see. It is an HTTP/gRPC API compatible with OpenAI (typically /v1/chat/completions, /v1/embeddings, /v1/models) that centralises everything that happens before touching the inference backends.

Why centralise

Without a gateway, tenants connect directly to vLLM or whatever the model is. Every change (rotating an endpoint, adding a model, changing credentials, applying a policy) means notifying every tenant. Every tenant has its own retry logic, its own logging, its own auth model. It is unworkable from the third customer onwards.

With a gateway, the change is made in one place. Tenants have a stable URL and a set of credentials; the rest is the gateway’s problem.

The three dominant options in 2026

LiteLLM is the most popular OSS option, Python-first, deployed as a proxy. It supports 100+ providers (OpenAI, Anthropic, Bedrock, self-hosted vLLM, Ollama, and so on) behind a unified OpenAI-compatible API. It has a native multi-tenant hierarchy with Organizations → Teams → Users → API Keys, each level with its own budget. The Apache 2.0 version covers the basics; RBAC, SSO, audit logs and team-level enforcement require the paid Enterprise version. Deployed on K8s with the official Helm chart.

Portkey is the most mature commercial / SaaS option. A single control plane that enforces budgets, quotas, permissions and compliance. Real-time spending tracking with alerting. RBAC, audit, workspaces and SSO included. The trade-off: dependency on an external service and a per-request pricing model.

Kong AI Gateway is the option for organisations that already have Kong as their API gateway. An AI plug-in on top of the existing Kong gateway, integrated with its plugin, consumer and rate-limit model. If your platform team already runs Kong, it is the lowest friction.

When to choose each one

Situation	Gateway
Pure OSS, self-hosted, Python-first team	LiteLLM
You need RBAC, SSO, audit log out of the box, and have budget	Portkey
You already run Kong as a corporate API gateway	Kong AI Gateway
Greenfield enterprise with strict compliance	Portkey (probably)
Mid-sized OSS-first company without regulated compliance	LiteLLM (typically)

The minimum the gateway has to do

Whichever option you pick, this is what any serious deployment must enforce:

Auth and identity: every request carries an API key resolvable to a tenant + user + team.
Routing by model: the tenant asks for model: "gpt-4o"; the gateway decides whether it goes to OpenAI, to Azure OpenAI, or to your vLLM with Qwen3 32B (a cheaper fallback), according to policy.
Rate limiting: RPS per tenant, TPM (tokens per minute), concurrency limits.
Caching of identical responses: 5-30% of RAG queries are repeats; caching saves latency and cost.
OTel emission: every call produces a span with gen_ai.* semantic conventions and tenant_id as an attribute. Covered in the Evals post and MCP observability.
Failover: if vLLM goes down, the gateway redirects to the OpenAI API. If OpenAI rate-limits, the gateway falls back to Anthropic. Configurable policy.

Example of a multi-tenant LiteLLM configuration

# litellm-config.yaml — simplified example
model_list:
 - model_name: llama-3-70b
 litellm_params:
 model: openai/llama-3-70b
 api_base: http://vllm-llama3-70b.inference/v1
 api_key: os.environ/VLLM_API_KEY

 - model_name: qwen3-32b
 litellm_params:
 model: openai/qwen3-32b
 api_base: http://vllm-qwen3-32b.inference/v1
 api_key: os.environ/VLLM_API_KEY

 - model_name: gpt-4o
 litellm_params:
 model: openai/gpt-4o
 api_key: os.environ/OPENAI_API_KEY

router_settings:
 routing_strategy: usage-based-routing-v2
 fallbacks:
 - llama-3-70b: [qwen3-32b, gpt-4o]  # if vLLM goes down, fall back to the external one

general_settings:
 master_key: os.environ/LITELLM_MASTER_KEY
 database_url: os.environ/DATABASE_URL  # Postgres for budgets/keys

# Hierarchy: Organizations → Teams → Users → API Keys
# Created via the API, not in static YAML

Creating a team with a budget:

curl -X POST http://litellm/team/new \
 -H "Authorization: Bearer ${LITELLM_MASTER_KEY}" \
 -d '{
 "team_alias": "soporte-chat",
 "max_budget": 500, # 500 USD/month
 "budget_duration": "30d",
 "tpm_limit": 100000, # 100K tokens/min
 "rpm_limit": 1000, # 1000 requests/min
 "models": ["llama-3-70b", "qwen3-32b"] # access to these
 }'

And the team’s API key:

curl -X POST http://litellm/key/generate \
 -H "Authorization: Bearer ${LITELLM_MASTER_KEY}" \
 -d '{
 "team_id": "<team-id>",
 "duration": "30d",
 "metadata": {"environment": "production", "app": "support-bot"}
 }'

That API key is what the tenant uses. Every request made with it consumes from the team’s budget. When it runs out, LiteLLM returns HTTP 429 with a description.

Layer 2 — Policy & Quota Plane: what each tenant can do

The gateway is where enforcement happens. Policy is what gets enforced. Five axes of multi-tenant policy:

Technical quotas

TPM (tokens per minute): the hard consumption limit. For a Llama 3 70B at TP=5, ~3000 sustained output tokens/s = 180K aggregate TPM. If you have 10 tenants, assign 18K each as the ceiling.
RPS / RPM: load control, not consumption control. A 4K-token session counts as one request; so does a batch of 100 mini-completions. Useful against abuse.
Concurrency: how many simultaneous active requests per tenant. Important for latency SLAs: 100 RPS with concurrency=50 works out at 2 seconds per request.

Economic budgets

Monthly per tenant: a hard cap in USD.
Daily and hourly: soft caps to avoid a runaway in a single day.
Per project / API key: fine granularity within a single tenant.

LiteLLM has a max_budget field at every level of the hierarchy (organization, team, user, api key). Budgets are inherited and constrained downwards.

Model whitelist and blacklist

Tenants with critical workloads → only stable models (llama-3-70b, gpt-4o). Research tenants → access to experimental models as well.

Priority classes

Not all requests are equal. Three typical classes:

Guaranteed: workloads with an SLA, latency respected even under pressure.
Best-effort: normal workloads without a strict SLA.
Spot: batches that can wait, evictable if a guaranteed one arrives.

The Token Management in Multi-Tenant AI Inference Platforms paper (2026) formalises this with a model of token pools per priority class that has started to be adopted in production. It maintains guaranteed P99 latency for guaranteed workloads even under overload, with selective throttling of spot.

Admission control

Before accepting a request: is there capacity? If not, return 429 immediately instead of queueing and degrading everyone. It is the most underrated operational discipline. A cluster with admission control done properly has predictable latency; without it, catastrophic degradation when the peak arrives.

The typical pattern in 2026

# Conceptual policy for a "soporte-chat" tenant
tenant: soporte-chat
quotas:
 tpm: 50000
 rpm: 500
 max_concurrency: 30
budget:
 monthly_usd: 800
 alert_thresholds: [0.5, 0.8, 0.95] # warn when you get there
models_allowed:
 - llama-3-70b
 - qwen3-32b
priority: guaranteed
fallback_on_overload:
 - qwen3-32b  # if guaranteed fills up, fall back
 - gpt-4o-mini  # last resort, external model

Layer 3 — Isolation Plane: isolating workloads physically

This is the technically densest layer. You have a node with several H100 SXM GPUs interconnected by NVLink. How do you partition them between tenants?

MIG (Multi-Instance GPU) is the strongest isolation. It partitions the GPU into up to 7 instances with physically separate HBM memory and dedicated compute units (SMs). Tenants in different MIG instances cannot touch each other: one workload does not consume memory another needs, one does not degrade another’s throughput. Hardware isolation. Available on A100, H100, B100 and B200.

MPS (Multi-Process Service) is soft. Several processes share the GPU concurrently and NVIDIA distributes SMs according to usage. Good performance if all the processes are yours and you trust them. Worse for multi-tenant setups between customers who do not know each other, because a noisy process can degrade the others.

Time-slicing is the simplest: the GPU is assigned alternately, slot by slot, to different processes. Much worse latency (waits between slots); not recommended for production workloads with an SLA.

The choice for multi-tenant in 2026

According to the enterprise adoption survey: 80% use MIG for untrusted multi-tenant (different customers who do not know each other) and MPS for trusted environments (processes from the same team) where you want to maximise throughput. Time-slicing is only used in dev/staging so that every developer can touch a GPU without the cost of exclusivity.

An important limitation of MIG: it isolates compute and HBM memory, but the PCIe path remains shared. For PCIe-bound workloads (a lot of host↔device traffic), tenants in different MIG instances can still affect each other. For LLM inference the main path is HBM, so this is rarely a problem. But it is worth knowing.

MIG partitions on the H100

An H100 (80GB HBM3) can be partitioned into fixed profiles:

Profile	SM	Memory	Max instances per GPU
1g.10gb	14	10 GB	7
1g.20gb	14	20 GB	4
2g.20gb	28	20 GB	3
3g.40gb	42	40 GB	2
7g.80gb	98	80 GB	1 (the whole GPU)

For a mid-scale cluster with NVLink, MIG has a fundamental problem: when you partition with MIG, NVLink between GPUs is disabled. An H100 in MIG mode does not take part in multi-GPU tensor parallel. If you are going to serve a large model with tensor parallel (Llama 3 70B with TP=4 or TP=8, for instance), those GPUs have to be whole, without MIG.

That defines the architectural decision. There are two main approaches:

Approach A — A shared large model with quotas at the gateway

All the node’s GPUs serve a single large model with tensor parallel spanning the whole node. All tenants share that instance. Isolation happens at the gateway layer (quotas, rate limiting) and the policy layer (priority classes). The cluster’s kernel is one enormous vLLM instance with --max-num-seqs=128 or similar; internally, vLLM shares GPU time between the active requests with continuous batching.

Advantages: you use all the GPUs to the full, NVLink stays active, better KV cache utilisation. Disadvantages: soft isolation. A tenant that saturates does not degrade the others directly (vLLM batches), but it does compete for batch slots. You need serious priority classes.

Approach B — Dedicating GPUs per model / tenant

You split the GPUs into pools dedicated to different models. Examples on an 8-GPU node:

4 GPUs: a large 70B model with TP=4.
2 GPUs: a mid-sized 32B model replicated (2 independent instances) for tenants with a strict SLA.
2 GPUs: miscellaneous workloads (smaller models, experimentation).

Advantages: physical isolation between critical models / tenants. Disadvantages: worse aggregate utilisation; some GPUs idle while others saturate.

Approach C (advanced) — MIG on some GPUs plus dedication for the rest

If you have small workloads (4B, 7B models), you can use MIG on 1-2 GPUs to serve them and dedicate the rest to tensor parallel for the large model. It combines strong isolation for the small workloads with full use of NVLink for the large model.

The operational choice: start with A, move up to C if you need to

In most deployments, Approach A (a shared large model plus quotas) is the right starting point. Utilisation is better, operation is simpler, and the gateway’s soft isolation works for reasonable workloads.

When there is a tenant with a strict SLA that cannot tolerate competing with others, you move to Approach B for that tenant in particular (dedicating GPUs to an instance of the model just for them), keeping the rest of the cluster shared.

Approach C is for when you have 10+ tenants with very heterogeneous profiles.

Isolation at the Kubernetes level

Independently of GPU isolation, pod-level isolation is applied in K8s:

Namespaces per tenant: tenant-soporte, tenant-legal, and so on.
ResourceQuotas and LimitRanges: CPU/memory limits per namespace.
NetworkPolicies: tenant A cannot talk to tenant B’s namespaces.
K8s PriorityClasses: classes with a numeric value defining preemption order if a more critical pod arrives.
PodDisruptionBudgets: how many pods of each deployment can go down simultaneously.

Layer 4 — Observability Plane: seeing what happens per tenant

The fourth layer: observability with a tenant dimension. Without it you cannot do cost attribution, you cannot debug incidents for a single tenant, you cannot show dashboards to stakeholders.

The four mandatory properties

1. tenant_id on every span. The AI gateway resolves the API key and attributes a tenant_id. That ID is propagated via params._meta or OTel headers to every downstream component (vLLM, retrieval, MCP servers, tools). Any span in any system carries that label. It is what lets you reconstruct tenant-specific traces.

2. Metrics labelled per tenant. gen_ai.usage.input_tokens{tenant="soporte-chat"} or equivalents. Prometheus, Grafana, groupable by tenant.

3. Real cost attribution. The sum of tokens × cost/token per tenant gives the cost. For self-hosted vLLM, the cost is per GPU hour plus the proportional share of tokens (you can compute an equivalent cost per 1k tokens).

4. An immutable audit log. Every API key used, every model invoked, every quota change, every budget exceeded. For compliance.

Showback vs chargeback

An important FinOps distinction that gained clarity in 2026:

Showback: visibility without consequence. “Support team, you have consumed 623 USD this month on LLMs”. Information, not an invoice. It lets you spot abuse without penalising anyone before the team understands what is going on.

Chargeback: the cost is charged to the team’s budget. When it runs out, it runs out. It changes behaviour.

The practice that works: 6-18 months on showback while tags are calibrated, misattributions are identified and teams are trained. Then chargeback, once the numbers are credible. Launching chargeback on day 1 when the costs are still dirty creates an immediate political fight; launching showback prepares the ground so that chargeback lands in an orderly way.

Only 14% of organisations have chargeback active according to a recent survey, which indicates this is still mostly showback in real production.

Tools

Kubecost: cost allocation per namespace, deployment and pod on Kubernetes. For the cost of the shared GPU, allocate proportionally to the tokens consumed per tenant.
Finout: a FinOps platform that combines cloud bills and LLM API costs in a unified view with virtual tagging.
Langfuse: already covered. Cost tracking per trace, groupable by user or session metadata.
LiteLLM native tracking: LiteLLM’s master DB keeps running spend per team, user and API key, accessible via the API or the UI.

The minimum multi-tenant dashboard

Any platform should have:

Summary per tenant: monthly spend, current RPS, TPM consumed, % of budget spent, active sessions.
Top users within each tenant (for detecting internal abuse).
p95 latency per tenant: SLA tracking.
429 / 503 errors: how many requests are being rate-limited or rejected because of overload.
Cost trend: the monthly trajectory with a projection.
Drift per tenant (from the post-tracing series): if a tenant starts getting worse results, raise an alert.

Sizing on mid-scale GPU clusters: concrete decisions

Let us get down to hardware. As a reference we take a node with N H100 SXM (between 4 and 8) with NVLink/NVSwitch, 80 GB HBM3 each. That gives between 320 GB and 640 GB of aggregate VRAM. Inter-GPU connectivity is 900 GB/s (NVLink 4) or 600 GB/s (NVLink 3) depending on generation. HBM bandwidth per GPU is 3.35 TB/s.

Default decisions

Start with Approach A: all the node’s GPUs serving a single large 70B model in BF16 with tensor parallel = N. Expected real capacity (computed for a standard 8-GPU HGX node as an example; it scales roughly linearly with N):

Model VRAM (70B BF16): ~140 GB (≈ 17.5 GB/GPU at TP=8).
vLLM overhead + activations VRAM: ~10 GB/GPU.
VRAM free for KV cache: ~52 GB/GPU. On an 8-GPU node that is ~416 GB aggregate; on a 4-GPU one, ~210 GB.
With --kv-cache-dtype=fp8 and a 70B GQA model: ~320 KB/token.
Aggregate cache capacity (8-GPU node): ~1.3M tokens to distribute between simultaneous sessions.

That translates into throughput and concurrency (indicative figures for an 8-GPU node):

Simultaneous sessions	Average context per session	Aggregate throughput (tokens/s)
32	16K	~5000
64	8K	~8000
128	4K	~12000

Typical latencies: TTFT ~150ms at low traffic, TPOT ~15-20 ms/tok. Under high concurrency, TTFT rises to ~500ms if the queue is saturated.

An example tenant scheme

A cluster with 4 tenants and a research pool:

Tenant	TPM cap	RPM cap	Concurrency	Budget	Priority	Models
Support chat	80K	800	50	1500 USD/month	Guaranteed	llama-3-70b, qwen3-32b
Legal RAG	30K	200	15	600 USD/month	Guaranteed	llama-3-70b
Code agent	50K	300	25	1200 USD/month	Best-effort	llama-3-70b, qwen-coder
Data extr. batch	40K	1000	40	400 USD/month	Spot	llama-3-70b, qwen3-32b
Research / notebooks	10K	100	5	200 USD/month	Spot	all

TPM total: 210K. Aggregate cluster capacity: ~180K sustained TPM. It is overcommitted by about 15%, assuming not every tenant hits the ceiling at the same time. That is normal and desirable; if they all do it at once, the priority classes degrade in an orderly way.

When to add hardware

Signals that the node has become too small:

Sustained p95 TTFT > 500 ms during peak hours → the queue is building up.
vllm:num_requests_waiting constantly > 20 → admission control starting to reject.
Sustained GPU utilisation > 80% at critical hours without dropping in quiet hours → there is no headroom.
429 rate on guaranteed tenants > 1% → the platform is breaking its SLA in production.

When several of these hold, the natural next step is to add another HGX node with internal NVLink and stand up a second vLLM instance of the same model. The gateway load-balances between the two instances. Aggregate throughput doubles; latency stays the same.

Common operational traps

A gateway without auth: a backdoor into the cluster

Your vLLM sits on a ClusterIP Service and the main app talks to it. Some tenant discovers the endpoint directly and hits it without going through the gateway. Quotas and costs are silently evaded. Use a strict NetworkPolicy: only the gateway can talk to the vLLM Services; the rest of the cluster cannot.

MIG and NVLink are incompatible

You enable MIG on a GPU thinking you will get isolation plus multi-GPU; you discover that MIG disables NVLink. Any large model with TP becomes unusable. Decide MIG vs NVLink globally per cluster, not per individual GPU.

Quotas pinned to the cluster’s ceiling

You add up the TPM of all the tenants and it comes to exactly the cluster’s capacity. When two tenants peak at the same time, both wait or one gets rejected. An overcommit of 10-20% is healthy (it assumes they do not all peak at once); more than that is dangerous.

No multi-tenant observability from day 1

You launch with quotas and isolation but without tenant_id on spans. Three months later, your CFO asks “how much does the support agent cost versus the legal one?” and you cannot answer. OTel with a mandatory tenant_id from the first version, even if there are no dashboards yet; having the data is worth more than having perfect dashboards with no data.

Showback that never reaches chargeback

You have been on showback for 18 months, the teams know the numbers, nobody changes behaviour. Without the pressure of real chargeback, the incentive dilutes. Set an explicit calendar for the transition to chargeback, with an owner and a deadline.

Non-whitelisted models eating the budget

A team discovers that LiteLLM has gpt-4o configured. They use it without permission. The budget burns on an external API when the idea was to use the cheap self-hosted one. Keep an explicit per-team whitelist of accessible models.

Badly calibrated priority classes

Everyone declares themselves “guaranteed”. At the first peak, there is nothing left to degrade and everything suffers. Priority classes only for critical cases, with justification. The majority should be best-effort.

No failover from the gateway

Your vLLM goes down. The gateway has no fallback configured and returns 503 to every tenant. Configure a fallback to another model, ideally an external one (OpenAI) for guaranteed workloads, even if it costs more per hour. Availability is worth more than cost per hour.

An operational roadmap for getting started

If you start from scratch with an empty GPU node, the minimum order is as follows. Each milestone is a day’s work with slack, not a tight one:

Day 1-2 — Base K8s infrastructure. NVIDIA GPU Operator + nvidia-device-plugin + dcgm-exporter + cluster-default NetworkPolicies. Validation: a basic pod with nvidia.com/gpu: 1 gets scheduled.

Day 3 — vLLM with a large model and tensor parallel across the whole node. The vLLM Production Stack Helm chart (or bare vLLM manifests). Model weights on a shared PVC (CephFS or NFS). Validation: a curl request against the internal Service responds.

Day 4 — AI Gateway: LiteLLM. Helm chart, Postgres for budgets, master key, a first model_list pointing at vLLM. Validation: an OpenAI-compatible request through LiteLLM responds with the same content as vLLM directly.

Day 5 — Basic multi-tenancy. Create teams, API keys, budgets, model whitelists. Test with two teams. Validation: the second team, using the model it does not have whitelisted, gets a 403.

Day 6 — Minimum observability. Prometheus + Grafana scraping vLLM and LiteLLM. A dashboard with TTFT, TPOT, throughput, num_requests_waiting and budget_consumed_per_team. Validation: visible in Grafana with real data.

Day 7-8 — Pilot customer. One real tenant (ideally a controlled internal one) starts using it. Measure real latencies, discover the first operational incidents.

Day 9-10 — Tuning. Adjust --max-num-seqs, --gpu-memory-utilization, priority classes and quotas based on what the pilot taught you.

Day 11-14 — Onboarding the second tenant plus iteration. Repeat. Every new tenant onboarded reveals new cases.

After two weeks you have an operational platform with two real tenants and data to decide whether it is ready for more. The line of progress from here on is horizontal (more tenants) until you saturate; from then on, vertical (more hardware).

What we have not covered (upcoming posts)

Continuous fine-tuning in production (post 6, already decided): LoRA/QLoRA/DPO, dataset curation, eval gates, A/B versioning with real traffic between model versions.
Constitutional AI and alignment at runtime: an option still on the table.
Edge LLMs: when an H100 cluster is too expensive for a specific workload, distilled models running on NPUs or consumer GPUs.
GPU networking deep dive: NCCL, InfiniBand, GPUDirect, RDMA. For multi-node clusters with cross-host tensor parallel.

References

Multi-tenancy and GPU isolation:

Multitenant GPU Infrastructure: 4 Powerful Design Rules — survey of enterprise patterns.
Run Multiple LLMs on One GPU: MIG, Time-Slicing, and MPS Guide (Spheron).
A Practical Guide to GPU Partitioning with MIG (Medium).
GPU Partitioning for AI Workloads: NVIDIA MIG with SUSE Virtualization (KubeCon EU 2026).
Predictable LLM Serving on GPU Clusters (arxiv 2508.20274).
Token Management in Multi-Tenant AI Inference Platforms (arxiv 2603.00356) — paper on priority + admission control.

AI Gateways:

Multi-tenant FinOps:

Cross-references:

Previous posts in series 4: MLOps for LLMs overview, RAG over Kafka, The six-stage pipeline, PostgreSQL + Qdrant.
Relevant posts from the inference series: vLLM on Kubernetes — the multi-GPU HGX node scenario we develop here. LLM operators on K8s — the vLLM Production Stack and OME that the gateway can direct.
Observability: Evals, MCP observability, eBPF + drift.

The six-stage LLMOps pipeline: global architecture and a deep dive into every component

Thu, 21 May 2026 06:30:00 +0200

TL;DR

The first two posts in the series established the LLMOps landscape and went down into the detail of the data pipeline with Kafka. This post does the intermediate zoom: it draws the complete map of the system, a global architecture of a modern LLMOps setup with all the pieces the field has settled on in 2026, and goes into depth on each of the six canonical stages of the pipeline: Data, Tune, Eval, Deploy, Observe, Retrain. For each stage we give the operational sub-tasks, the dominant tools, the design decisions that always show up, and the specific traps that are seen repeatedly in production. And, most important operationally: every stage carries a “you are here” mini-map over the cycle, which will be reused in any later post in the series to place the reader. The idea: that anyone reading a post about fine-tuning, about prompt versioning, about eval gates or about drift detection can look at the mini-map and know immediately which piece of the larger system they are thinking about that day.

This is the third post in the MLOps series specific to LLMs. Previous ones: Landscape 2026 and RAG over Kafka. Here we move from “the what” and “one piece” to the whole map, with detail per stage.

The global architecture: the master map

Before going down into each stage, let us fix the whole map. What follows is the reference drawing of a production LLMOps system in 2026, with every component the field has settled on in its place:

What you see: the six large boxes are the stages; the solid arrows are the flow of the pipeline; the dashed arrow that goes from Retrain to Data is the feedback cycle that turns LLMOps into a living process rather than a project that ends. The grey band at the foot holds the cross-cutting components (observability, prompt versioning, MCP, gateway, schema) that run through every stage and connect to each one.

Three quick readings of the map:

Horizontal, top: the happy path, data → tune → eval. What happens while you prepare the model.
Horizontal, bottom: the service path, deploy → observe → retrain. What happens once the model is alive.
Vertical: the connection between the two floors. Eval gateway feeds Deploy; Observe feeds Retrain; Retrain returns to Data.

From here on, every stage will include a navigation mini-map (“you are here”) to place you in the complete cycle. Let us go through each one.

Stage 1 — Data: ingestion, transport, versioning, indexing

Operational sub-tasks

The Data stage is the most underrated and the one that blocks the most projects. Its sub-tasks:

Ingestion from heterogeneous sources: OLTP databases (Postgres, MySQL), external APIs, file shares, scraping, SaaS systems, application logs, internal messaging.
Change capture (CDC) in streaming if the data is dynamic. Debezium over Kafka, Flink CDC, modern alternatives such as RisingWave that reads the WAL directly.
Transformation (cleansing, dedup, normalisation, PII sanitisation).
Schema management: schema registry, compatible evolution, backward/forward compatibility.
Versioning of training datasets and golden datasets: DVC + lakeFS (unified in November 2025). Covered in detail in the dedicated data versioning post.
Indexing for RAG: chunking, embeddings, writing to vector stores. Covered in depth in the Kafka post.
Materialisation into analytical tables: Tableflow → Iceberg/Delta, for BI consumption and low-latency queries.

Dominant tools

Sub-task	Tools 2026
CDC	Debezium, Flink CDC, RisingWave
Transport	Kafka (Confluent Cloud, Redpanda, pure Apache)
Schema Registry	Confluent Schema Registry, Apicurio
Stream processing	Apache Flink, RisingWave, Kafka Streams
Data versioning	DVC + lakeFS
Vector stores	Milvus, Qdrant, Weaviate, pgvector, LanceDB
Materialised tables	Tableflow → Iceberg/Delta
Batch ETL/ELT (where it applies)	dbt + Snowflake/Databricks

Design decisions

The three decisions that always show up:

Batch vs streaming: the more dynamic the data, the more streaming. For static corpora (manuals that never change) a nightly batch is enough; for transactional data the agent needs to see minute by minute, streaming from day 1.

Embedding model: changing the embedding model invalidates every indexed vector. An architectural decision: pin the model and have an explicit migration plan (the dual-index pattern seen in the Kafka post).

Vector store: pgvector if you already run Postgres and you are below 10M vectors; Qdrant if you want mid-scale simplicity; Milvus if you need billions; Weaviate if you value native hybrid search.

Traps

Hardcoding connections to the source (with no abstraction): when the database changes (version, host, schema), you break the whole pipeline. An adapter layer from day 1.
No schema registry: topics start breaking silently.
Full reindexing whenever something changes: it costs hours or days. Design the dual-index pattern from the start.
Unsanitised PII: the RAG is serving sensitive data without meaning to. Anonymisation in the pipeline, not at consumption time.

Stage 2 — Tune: preparing the model for your case

Operational sub-tasks

Base model selection: Llama, Qwen, Mistral, Gemma, DeepSeek according to licence, size, quality in your domain.
Dataset preparation: train/val/test split, format (chat templates, JSONL), augmentation where it applies.
Adapter configuration: LoRA rank, target modules, alpha; QLoRA if you want to train on a consumer GPU; full fine-tune only if you have the budget.
Training loop: HuggingFace Transformers + PEFT + TRL as the canonical stack; Axolotl or llama-factory as convenience wrappers; Unsloth if you want 2-4× more speed on consumer GPUs.
Hyperparameter sweep: W&B Sweeps, Optuna, Ray Tune.
Checkpointing and resumability: save every N steps, resume from failure.
Promotion: the adapter is promoted to the registry after passing the next stage (Eval).

The three modes of Tune

The detail of the table we saw in the landscape:

Supervised fine-tuning (SFT) with LoRA/QLoRA. You collect (prompt, ideal-response) pairs and apply SFT with cross-entropy loss. The simplest option. The rule of thumb: 300-3,000 well-curated examples are usually more useful than 50,000 noisy ones.

DPO (Direct Preference Optimization) and RLAIF. Instead of “ideal-response”, you collect (prompt, good_response, bad_response) triples and train the model to prefer the good one. More stable than classic RLHF, same objective. It is what most teams use when they go beyond SFT.

Agent training (RFT / Reinforcement Fine-Tuning, pure RLHF). For cases where the model needs to learn multistep trajectories: when to choose tool A over B, when to ask for confirmation, how to decompose a large task. Far more expensive and complex. OpenAI’s work with RFT set the pattern in 2024-2025; in 2026 it is coming out of the experimental phase.

RAG as an alternative to Tune: although conceptually it is another stage (it lives in Data + Deploy), functionally it competes with fine-tuning for many cases. The 2026 verdict: hybrid is the default (60% of deployments), fine-tune for behaviour plus RAG for volatile knowledge.

Tools

Aspect	Tools 2026
Base framework	HuggingFace Transformers, PEFT, TRL
Convenience wrappers	Axolotl, llama-factory
Consumer speed	Unsloth (2-4× speedup on RTX GPUs)
Distributed training	DeepSpeed, FSDP, NeMo Framework
Experiment tracking	MLflow, W&B, ClearML
Adapter registry	Private HuggingFace Hub, MLflow registry
Hyperparameter	W&B Sweeps, Optuna, Ray Tune

Traps

Catastrophic forgetting: very aggressive SFT destroys the model’s general capabilities. Keep a small % of the original dataset or use regularisation.
Overfitting to the golden dataset: the model learns to memorise the eval set. Keep a holdout test set that nobody on the team looks at until the final release.
Train/serve skew: prompts in training with a different format from production. The same chat template in both.
LoRA rank too high: it looks like it improves metrics but it inflates the adapter with no real benefit. Start with r=8 or r=16; raise it only if there is evidence.

Stage 3 — Eval: validating before promoting

Operational sub-tasks

Covered in depth in Evals: the layer after tracing. A structured summary for the pipeline:

Golden dataset curation: 100-500 examples as a minimum, actively maintained with incident cases.
Evaluators: heuristic (regex, length), semantic (embeddings), LLM-as-judge (G-Eval), human (golden labels).
Running in CI: block the merge if critical metrics drop by more than X%.
Running on the platform: over sampled production traffic, persist results, detect long-term regression.
Judge calibration: 85-90% agreement with humans before accepting the judge as productive.
Eval gates: explicit thresholds per metric (faithfulness > 0.85, relevancy > 0.80, and so on).

Tools

CI gates: DeepEval (Apache 2.0, pytest-style), Promptfoo (MIT, CLI), Ragas (RAG-specific), Inspect AI (safety/capability).
Platform: Langfuse (MIT, complete suite), LangSmith (LangChain), Phoenix (ELv2, OTel), Braintrust.
Judges: GPT-4 (expensive but the reference), Claude 3.5 Sonnet, Prometheus (OSS, 0.897 correlation), JudgeLM.

Traps

Aged golden dataset: if it is not updated, it stops reflecting production.
Contaminated judge: the judge knows the dataset (it appeared in its training).
Insufficient sample size: fewer than 50 examples makes differences look like noise.
Runaway costs: G-Eval with GPT-4 over many cases costs thousands of USD a month.
Forgetting the segment: an average of 0.85 can hide a 0.55 in German.

Stage 4 — Deploy: putting the model into production

Operational sub-tasks

Covered in depth in vLLM on Kubernetes and LLM operators on K8s. A summary for the pipeline:

Runtime selection: vLLM (default), SGLang (agents with high prefix caching), TensorRT-LLM (pure latency), llama.cpp (edge).
Operator selection: vLLM Production Stack, KServe, OME (LMSYS), NVIDIA Dynamo, llm-d (CNCF).
Serving configuration: --tensor-parallel-size, --kv-cache-dtype=fp8, --enable-prefix-caching, --enable-chunked-prefill, --gpu-memory-utilization=0.92.
Routing between models: LiteLLM as the multi-provider abstraction.
Release strategy: canary (1% → 10% → 100%), blue-green (all or nothing with fast rollback), shadow (eval in parallel without affecting users).
Autoscaling with LLM metrics: KEDA + Prometheus over vllm:num_requests_waiting or equivalent.
Gateway / Inference Extension: Gateway API Inference Extension once it is GA.

Dominant tools

Serving engines: vLLM, SGLang, TensorRT-LLM, llama.cpp, MLX.
Operators: OME, vLLM Production Stack, NVIDIA Dynamo, llm-d, KServe.
Routing: LiteLLM (100+ providers), OpenRouter (managed), LangChain Router.
GPU primitives: NVIDIA GPU Operator, LeaderWorkerSet (LWS) for multi-pod tensor parallel, KEDA for autoscaling.

Traps

Naïve rolling update that cuts sessions: maxUnavailable: 0, maxSurge: 1 and terminationGracePeriodSeconds: 120+.
A short readiness probe that kills pods while they load: startupProbe with failureThreshold: 60.
HPA on CPU% with no LLM metrics: vLLM batches internally, one replica serves dozens. KEDA on queue depth.
Unquantised KV cache: --kv-cache-dtype=fp8 is almost always worth it.
Tensor parallel on GPUs without NVLink: all-reduce saturates PCIe and throughput collapses.

Stage 5 — Observe: seeing what happens in production

Operational sub-tasks

This is the stage we have covered most deeply in previous series: the whole eBPF series (4 posts) and the whole post-tracing series (4 posts) deal with sub-tasks of Observe. A structured summary:

Tracing: OpenLLMetry/Traceloop, Langfuse, Phoenix, LangSmith. Spans with OTel GenAI semantic conventions (gen_ai.*, mcp.*).
Metrics: Prometheus + Grafana. TTFT, TPOT, throughput, queue depth, KV cache usage, cost per tool.
Active guardrails (not just eval): NeMo Guardrails with 5 types of rails, multimodal Llama Guard 4, Llama Prompt Guard 2 (86M/22M), LLM Guard.
eBPF observability (zero-instrumentation): Hubble (network), Tetragon (process/syscall), AgentSight (LLM agent with SSL uprobes + stdiocap MCP).
eBPF in the local engine (inference): ProfInfer-style with uprobes in llama.cpp / vLLM / libcudart.
Drift detection: Evidently AI, NannyML, WhyLabs. KS, PSI, MMD over embeddings.
MCP observability: OpenTelemetry GenAI MCP semantic conventions, trace propagation via params._meta, a centralised MCP Gateway.

The four mandatory metrics

Out of everything covered, the four any minimal dashboard must have:

TTFT p50/p95 (time to first token) — what the user perceives.
TPOT p50/p95 (time per output token) — streaming speed.
Throughput (aggregate tokens per second) — capacity planning.
Queue depth (vllm:num_requests_waiting) — the leading indicator.

To these you add, per domain:

For RAG: faithfulness rolling mean, retrieval hit rate.
For agents: tool call accuracy, multi-step task completion.
For multi-tenant: cost per tenant, p95 latency per tenant.

Traps

Cardinality in Prometheus: metrics carrying every K8s label blow up.
Tracing with no sampling: storage grows out of control.
Guardrails permanently in monitoring mode: they never reach enforce.
Drift with no alerts: you spot drift on the dashboard once a month; meanwhile the problem has been running for weeks.
OTel with no propagation: MCP, Tetragon and AgentSight spans left disconnected.

Stage 6 — Retrain: closing the loop

Operational sub-tasks

This is the stage most neglected in GenAI projects. Closing the loop turns LLMOps into a living practice; not closing it leaves it as a project that ages.

Explicit feedback: thumbs up/down in the UI, annotations by power users, forms for “what went wrong”.
Implicit feedback: anomalous latency, abandonment rate, user retries, aborted sessions.
Incident triaging: classify incidents by root cause (model issue, retrieval issue, prompt issue, infra issue).
Dataset enrichment: add to the golden dataset the cases where the system failed, with the correct answer labelled by a human.
Retrain cadence: quarterly by default, incident-driven when a problematic pattern goes over a threshold.
Promotion: the new model/adapter goes through the Tune → Eval → Deploy stages, with eval gates that compare against the model in production.

The two cadences

Scheduled retrain (quarterly or half-yearly): an established process. It lets you plan capacity, budget and risk. The default.

Incident-driven retrain: when a serious incident (detected drift, a failing segment, a prompt injection attack) crosses a threshold, a mini-cycle fires. More expensive but necessary for critical cases.

Dominant tools

Annotation and feedback collection: Langfuse (built-in UI), Argilla (OSS), Label Studio.
Dataset enrichment: pipelines in Airflow or Argo Workflows.
Triaging: Langfuse dashboards with filters on traces with a low eval.
Promoting candidates: MLflow model registry stages.

Traps

An open loop: production does not inform the dataset; the model never improves.
Human feedback gets lost: thumbs down with no structured capture channel.
Undefined cadence: “we retrain when it is needed” → nobody ever retrains.
No holdout test set: the golden dataset is enriched with the same cases used to evaluate; eval measures memorisation.
Promotion with no gates: the new model enters production without passing the checks the previous models passed.

The complete cycle: how the stages fit together

Now that we have seen each stage separately, the key insight is how they hook into each other. Five emergent properties of the cycle:

1. Data is the raw material of every stage. Tune reads from the golden dataset. Eval reads from the eval dataset. Deploy reads from the RAG (vector store). Observe produces new data. Retrain creates new datasets. The Kafka log is the gospel of the whole system (post 2 in the series).

2. Eval is the bidirectional gatekeeper. Before Deploy: it blocks the release if the model regresses. After Observe: it feeds Retrain by identifying the worst-scoring cases. The quality of the eval determines the quality of the whole cycle.

3. Observe feeds Retrain and Eval simultaneously. Traces produce metrics for Observe; problematic traces are annotated and go to the dataset; the new cases enrich the eval golden set. Observe is the operational source of truth.

4. The cross-cutting components (the grey band on the map) are not a stage, they are an infrastructure. OpenTelemetry, prompt versioning, MCP gateway, model gateway, schema registry. Badly configured, every stage suffers separately. Well configured, the stages integrate without friction.

5. The cycle is not strictly sequential, it is concurrent. At any given moment the system has: requests being served (Deploy + Observe), a new version in training (Tune), continuous eval in CI (Eval), data arriving from CDC (Data), incident analysis (Retrain). Every stage is alive at once.

Cross-stage traps: things that break the whole system

There are errors that do not belong to a stage but to the interfaces between stages. The most common ones:

Train/serve skew

The exact format of the prompt in training is different from the one in production. The result: the model trained to answer <|im_start|>user\n...\n<|im_end|> receives User: ...\nAssistant: in production and performs worse. Solution: extract the chat template into a shared library used by the Tune pipeline and the Deploy one.

An eval that does not reflect production

Your golden dataset is careful questions; production is real questions with typos, mixed languages, and so on. Eval passes at 95%, production performs at 70%. Solution: continuously enrich the golden set with real samples.

Drift with no response pipeline

You spot drift on the Observe dashboard; nobody has a defined workflow for what to do. Solution: every drift alert must have a clear runbook: investigate, classify, act (retrain, adjust the prompt, widen the retrieval).

Schema break cascade

You change the schema at the OLTP source; Debezium reflects it; the Flink job breaks; the embedded topic stops updating; the vector store ages; the RAG answers over old data. Three stages affected by one change in Data. Solution: mandatory backward-compatible schema evolution, contracts between producers and consumers.

No observability of the pipeline itself

The LLMOps pipeline is a complex system. If it has no observability of its own (how long training takes, how many jobs fail, how many re-embeddings run), debugging failures is an exercise in spelunking. Solution: OTel over the pipeline itself, not just over the LLM calls.

Invisible vendor lock-in

Pipelines written against LangChain, prompts stuck in LangSmith, embeddings in Pinecone, the model in OpenAI. Migrating is a months-long project. Solution: LiteLLM and OpenLLMetry abstractions, vendor-neutral from the start.

What comes in the following posts

Post 4 — PostgreSQL + Qdrant in the ingestion stage — the first post that applies the “you are here” pattern over the Data stage. Synchronisation patterns (outbox + CDC), microservices architecture, deployment manifests.
Upcoming posts — still to be decided: the cluster as a multi-tenant platform, Constitutional AI / runtime alignment, continuous fine-tuning in depth, edge LLMs.
In any later post in this or future series, the “you are here” mini-map will tell you which stage of the cycle the topic fits into. If you read a post about quantization, you will know you are in Deploy. If you read one about evaluator ensembles, you will know you are in Eval. If you read one about RAG over Iceberg, you will know you are in Data.
If you want to see the whole pipeline in action following a single real request, the synthesis post Anatomy of an LLM request in production does exactly that: it rewinds a request back to the data that trained it 90 days earlier and follows it forward to the feedback that will reappear in the next Retrain cycle, crossing the six stages and the two cross-cutting components in one coherent story.
If what interests you is comparing how each stage is built in open source against the hyperscalers, the post The parallel catalogue: the six LLMOps stages in OSS and in AWS / GCP / Azure makes the vertical cut: for every stage, which tools the blog’s reference OSS stack uses and what the cloud equivalents are, with summary tables, gap identification and the multi-tenant chatbot ported to an AWS stack as a concrete example.
If you want the OSS toolbox piece by piece, the post The OSS catalogue for LLMOps in six stages: entry by entry does the zoom in: ~150 words of description per core tool (what it does, how it differs from its alternatives, licence and governance, typical gotcha), a decision matrix per stage, a diagram of the connected OSS stack and a master table of licences and EE / SaaS offerings.
If what you are missing is the common vocabulary that runs through the six stages, the post Ontologies and knowledge graphs in LLMOps walks the six stages from the perspective of shared formal nomenclature: how TBox + ABox + SHACL + SKOS change the operation of Data, Train/Adapt, Eval, Deploy, Observe and Govern, the GraphRAG 2026 landscape (Microsoft GraphRAG v2, LightRAG, HippoRAG 2, KAG/OpenSPG), vertical ontologies actually deployed (FIBO, SNOMED CT, schema.org, ENS, EU AI Act) and an open source stack with its licensing caveats (Neo4j Community GPLv3, KuzuDB upstream archived in October 2025).

References

Foundations:

Per stage (entries from the blog series):

Data: RAG over Kafka — technical architecture and the dedicated post on data versioning with DVC and lakeFS — the four artefacts to version separately (training, RAG corpus, golden eval, enriched retrain), schema contracts, end-to-end lineage from dataset to trace, and why a golden set without a strict holdout measures memorisation.
Tune: partly covered in Landscape 2026; a deeper treatment in post 4 if continuous fine-tuning is chosen.
Eval: Evals: the layer after tracing.
Deploy: vLLM on Kubernetes and LLM operators on K8s.
Observe: the whole eBPF series and the whole post-tracing series.
Retrain: the dedicated post on how to close the loop — feedback capture (explicit + implicit), triage by root cause, dataset enrichment with human annotation (Argilla / Label Studio), scheduled vs incident-driven cadences, governed promotion with eval gates.

Cross-cutting components:

Prompt versioning: the dedicated post with Langfuse and MLflow Prompts — the three-primitive pattern (immutable version, mutable label, cache), eval gates on promotion, and per-request traceability.
MCP: deep MCP observability.
Drift detection: eBPF + drift detection.
Local inference: PagedAttention deep dive, KV cache.

Frameworks and tools referenced:

MLflow, W&B, Kubeflow, ZenML, BentoML, Metaflow.
HuggingFace PEFT, TRL, Axolotl, Unsloth.
DVC + lakeFS.
Langfuse, Evidently AI, Phoenix.
vLLM Production Stack, KServe, OME, NVIDIA Dynamo, llm-d.

RAG on Kafka: a technical reference architecture for streaming datalakes, with fresh embeddings and vector stores that are always up to date

Thu, 21 May 2026 06:00:00 +0200

TL;DR

The piece that blocks enterprise GenAI projects most in 2026 is not the model, nor even the guardrails: it is data ingestion for RAG. Companies hold valuable information in OLTP databases, in operational logs, in SaaS systems, and all of it is quietly changing every second. Batch RAG systems that reindex every night arrive late (the model’s answer is backed by a snapshot from 18 hours ago) and open the door to operational hallucinations even when the retriever is perfect. The dominant answer in production in 2026 is to build the RAG piece on top of Kafka as the source of truth: an immutable log, massive throughput, managed schema evolution, and a mature stream processing ecosystem (Flink, Kafka Streams, RisingWave) that makes it possible to transform and embed events as they happen, delivering them to vector stores (Milvus, Qdrant, Weaviate, pgvector) in milliseconds. The canonical pattern: source → CDC with Debezium → Kafka topics → Flink SQL with an embedding UDF → sink connector to a vector store → serving with vLLM or equivalent. The 2026 developments that change the game: Confluent Tableflow turns Kafka topics into Iceberg/Delta tables automatically (read from Snowflake/Databricks/Trino with no ETL, 30-50 % lower TCO); native Flink SQL brings openai_embedding() and vector search integrated with Cosmos DB and Amazon S3 Vectors; Confluent’s official MCP server lets AI agents query Kafka/Flink/Tableflow in natural language. This post develops the end-to-end architecture with manifests, Flink SQL code and concrete numbers.

This is the second post in the MLOps series specific to LLMs. The first (2026 landscape) set out the framework. Here we go down to the most operational piece of the stack: how a real enterprise system connects to an LLM agent while keeping RAG fresh without falling into explosive complexity.

The analogy: Kafka as the “single source of truth”

Anyone who has spent time in distributed systems has seen the pattern again and again: an immutable, append-only, replicated, time-ordered log has become the canonical primitive for rebuilding complex systems. DBAs know it as the write-ahead log (PostgreSQL WAL, MySQL binlog). Event system developers know it as event sourcing. Data architects know it as the Kappa architecture. Kafka is the massive, distributed, mature implementation of that primitive: a log that lives on disk, partitioned to scale, replicated for durability, retained by time or size, readable from any point in history.

When you think about RAG, this is exactly what you need. A well-designed RAG system has two critical questions: how do you keep the index fresh? And how do you rebuild the index when something breaks? Kafka answers both naturally: fresh because every change at the source is published as an event to the log and the pipeline processes it in milliseconds; rebuildable because the whole log is there: you drop the vector store, you have the Kafka topic from offset 0, and you build the index back exactly as it was.

There is a second layer to the analogy as well. For a modern GenAI architecture, Kafka plays the role of the WAL of the entire system. Just as the Postgres WAL is the gospel of the database’s state (if you lose the DB but keep the WAL, you can rebuild it), the Kafka log is the gospel of the state of the business as a whole: orders, users, transactions, documents. Connecting your AI agent to Kafka means connecting it to the real pulse of the system, not to stale snapshots.

The static RAG problem

Before presenting the architecture, it is worth pinning down what concrete problem we are solving. The antipattern that trips up most GenAI projects:

A team builds RAG on a static dataset: it dumps Confluence documents, product PDFs, database snapshots.
It embeds the lot with a nightly cron that regenerates the index every 24 hours.
It ships the product.
Day 2: a user asks about a change that happened two hours ago. RAG does not have it; the model answers about the old version.
The team adds fragile logic: “if the query mentions a recent date, escalate to a human agent”.
Day 30: the dataset has moved so much that half the RAG is out of date. The team decides to refactor and migrate to streaming.

It is the repeated story of so many projects that the ecosystem has learned the lesson: streaming from day 1, even when the volume is low. The operational complexity of a well-designed streaming pipeline is constant; the complexity of migrating from batch to streaming on a live project is enormous.

From Lambda to Kappa to Streaming RAG

Three architectures in historical order:

Lambda (the classic big-data design from 2014): two parallel pipelines, one batch for accuracy and one streaming for freshness. The query combines both. It works, but it means maintaining two pipelines.

Kappa (Jay Kreps, 2014, mainstream since 2020): a single streaming pipeline. Batch is a special case of streaming (reprocessing from the beginning). It simplifies a great deal.

Streaming RAG (emerging in 2025-2026): a specific variant of Kappa where the pipeline’s output is embeddings indexed in a vector store that the LLM queries at runtime. The Kafka log is the source of truth, the vector store is a queryable projection.

The mental conversion: think of the vector store as the materialised view of the Kafka log. If the view is corrupted, you rebuild it from the log. If you want a new view (a different embedding model, a different chunking strategy), you create another consumer of the log and build a second view in parallel.

The reference architecture

On to the diagram. I am going to present the canonical architecture that stabilised in 2026, showing where each component fits:

[OLTP DB (Postgres)] [Other sources]
│ │
│ WAL via logical decoding │
▼ ▼
┌──────────────────────────────────────────────────────────┐
│ Debezium / Kafka Connect (Sources) │
└──────────────────────────────────────────────────────────┘
│
▼ produces events
┌──────────────────────────────────────────────────────────┐
│ Kafka cluster │
│ ┌───────────────────────────────────────────────────┐ │
│ │ topic: orders.raw (3 partitions, RF=3) │ │
│ │ topic: users.raw (3 partitions, RF=3) │ │
│ │ topic: documents.raw (6 partitions, RF=3) │ │
│ └───────────────────────────────────────────────────┘ │
│ + Schema Registry (Avro/Protobuf) │
└──────────────────────────────────────────────────────────┘
│
▼ consumes and transforms
┌──────────────────────────────────────────────────────────┐
│ Flink SQL streaming jobs │
│ - chunking text │
│ - calls to the embedding model (UDF) │
│ - enrichment with metadata │
│ - sink to a curated topic: documents.embedded │
└──────────────────────────────────────────────────────────┘
│
┌───────────┼────────────────────┐
▼ ▼ ▼
[Vector store] [Tableflow] [Iceberg/Delta]
Milvus/Qdrant auto-convert for analytics
/pgvector/ topics →
Weaviate tables
│
▼ queried at runtime
┌──────────────────────────────────────────────────────────┐
│ LLM serving (vLLM / SGLang) + Retriever │
│ - receives the agent's query │
│ - searches top-K in the vector store │
│ - builds prompt + context │
│ - generates an answer with citations │
└──────────────────────────────────────────────────────────┘

The five layers you can see (source, ingestion (CDC), transport (Kafka), processing (Flink), storage (vectors + tables)) are the ones that structure any serious RAG on a datalake in 2026. Let us go through each one.

Layer 1 — Sources: your OLTP as the starting point

The typical source is an OLTP database (Postgres, MySQL, SQL Server). It is where the live state of the business lives. The technique for extracting changes in real time is Change Data Capture (CDC): reading the database’s transaction log (PostgreSQL WAL, MySQL binlog) and turning each commit into a Kafka event.

The OSS standard is Debezium. It supports Postgres, MySQL, SQL Server, MongoDB, Oracle, Cassandra and others. The typical deployment is a Kafka Connect cluster with Debezium connectors.

An example Debezium configuration for PostgreSQL:

{
 "name": "postgres-orders-connector",
 "config": {
 "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
 "tasks.max": "1",
 "database.hostname": "postgres.prod.internal",
 "database.port": "5432",
 "database.user": "debezium",
 "database.password": "${secret:postgres-creds}",
 "database.dbname": "ecommerce",
 "database.server.name": "ecommerce-prod",
 "table.include.list": "public.orders,public.users,public.products",
 "publication.autocreate.mode": "filtered",
 "slot.name": "debezium_slot",
 "plugin.name": "pgoutput",
 "topic.prefix": "ecommerce",
 "key.converter": "io.confluent.connect.avro.AvroConverter",
 "value.converter": "io.confluent.connect.avro.AvroConverter",
 "key.converter.schema.registry.url": "http://schema-registry:8081",
 "value.converter.schema.registry.url": "http://schema-registry:8081"
 }
}

For each commit in the database, this produces an Avro event on the corresponding topic (ecommerce.public.orders, ecommerce.public.users, and so on) carrying the change: type (INSERT/UPDATE/DELETE), values before and after, commit timestamp, position in the WAL.

A simpler alternative for 2026: RisingWave can read the Postgres WAL directly, with no Debezium or Kafka Connect in between. When the case is CDC only, with no other sources, it is operationally simpler. For architectures with multiple sources (CDC + APIs + scrapers + logs), Debezium remains the standard piece.

Layer 2 — Kafka as transport and persistence

The Kafka cluster is where all events land. Key operational decisions:

Topics: raw vs curated

The convention that settled in 2026:

*.raw: the raw event exactly as it arrived. Untransformed CDC, unparsed application logs.
*.cleaned: after dedup, schema validation, type normalisation.
*.enriched: after adding metadata (geolocation, cross identifiers, and so on).
*.embedded: the event with its vector embedding already computed.

Multi-stage topics allow debugging per layer and partial reprocessing: if you change the embedding model, discarding *.embedded and rebuilding from *.enriched costs hours; rebuilding from *.raw costs days.

Schema Registry

Without a schema registry, topics break silently when someone changes the schema at the source. Confluent Schema Registry or the OSS Apicurio are the dominant options.

Common formats:

Avro: versioned schema, strict evolution rules. The historical default.
Protobuf: compatible with gRPC stacks, good performance.
JSON Schema: textual, debuggable by eye, less efficient.

For RAG on Kafka we recommend Avro by default. Schema evolution matters because source tables change over time, and an unversioned schema breaks downstream consumers.

Partitions, replication and retention

Operational decisions for RAG topics:

Partitions: typically 3-12. More partitions means more parallelism in the Flink consumer, but more overhead. The rule of thumb: partitions = expected peak events/s ÷ 1000.
Replication factor: 3 as a minimum in production. Replication protects against broker failure; with RAG, the cost of losing a topic can be weeks of re-embedding.
Retention: for topics that feed RAG, use long retention or compaction by key. If document doc-42 changes 100 times, compaction keeps only the last state per key, leaving a smaller, rebuildable log. For data that is never updated (historical logs), retention by time (90 days, 1 year).

Cross-cluster replication

For multi-region or multi-cloud deployments, MirrorMaker 2 or Cluster Linking (Confluent) replicate topics between Kafka clusters. RAG can then query the local cluster without crossing regions.

Layer 3 — Flink as the streaming processor

Apache Flink is the dominant stream processing piece in 2026. Apache 2.0, a mature distribution, a broad ecosystem. The main alternative is Kafka Streams (simpler, Java-only); RisingWave is the emerging option for pure SQL cases.

What Flink adds to Kafka:

Stateful streaming: time-based aggregations, joins between streams, sessions.
Exactly-once semantics: with checkpoint coordination.
Watermarks: correct handling of out-of-order events.
UDFs in Python/Java: including calls to LLM models.

Flink SQL: the most operational piece

Flink SQL is the most usable part of Flink for data engineers who are not streaming experts. Here is a realistic RAG pipeline example:

-- 1. Define the source: Kafka topic with CDC events from documents
CREATE TABLE documents_raw (
 doc_id STRING,
 title STRING,
 body STRING,
 category STRING,
 updated_at TIMESTAMP_LTZ(3),
 PRIMARY KEY (doc_id) NOT ENFORCED
) WITH (
 'connector' = 'upsert-kafka',
 'topic' = 'ecommerce.public.documents',
 'properties.bootstrap.servers' = 'kafka:9092',
 'key.format' = 'avro-confluent',
 'value.format' = 'avro-confluent',
 'value.fields-include' = 'EXCEPT_KEY'
);

-- 2. Define the sink: vector store via an intermediate Kafka topic
CREATE TABLE documents_embedded (
 doc_id STRING,
 chunk_id INT,
 title STRING,
 chunk_text STRING,
 category STRING,
 embedding ARRAY<FLOAT>,
 embedded_at TIMESTAMP_LTZ(3),
 PRIMARY KEY (doc_id, chunk_id) NOT ENFORCED
) WITH (
 'connector' = 'upsert-kafka',
 'topic' = 'rag.documents.embedded',
 'properties.bootstrap.servers' = 'kafka:9092',
 'key.format' = 'json',
 'value.format' = 'json'
);

-- 3. UDF for chunking (defined in Python or Java)
-- CREATE TEMPORARY FUNCTION chunk_text AS 'com.example.ChunkingUDF';

-- 4. Pipeline: chunk, embed, write to the sink
INSERT INTO documents_embedded
SELECT
 doc_id,
 chunk_idx AS chunk_id,
 title,
 chunk AS chunk_text,
 category,
 OPENAI_EMBEDDING(chunk,
 'text-embedding-3-small') AS embedding,
 CURRENT_TIMESTAMP AS embedded_at
FROM documents_raw
CROSS JOIN UNNEST(chunk_text(body, 512, 64))
 WITH ORDINALITY AS t(chunk, chunk_idx);

What happens here, line by line:

The documents_raw table reads the CDC topic in upsert-kafka mode (each new event on the same key replaces the previous one). This correctly reflects the semantics of “this is the latest version of doc 42”.
The documents_embedded table will be the intermediate topic where Flink writes the embedded chunks.
The chunk_text UDF (defined in Python or Java) splits each doc into 512-token chunks with an overlap of 64.
The INSERT INTO query runs continuously: every new event in documents_raw is chunked, each chunk is embedded with OPENAI_EMBEDDING (a built-in Flink SQL function in Confluent Cloud 2026), and written to the embedded topic.

OPENAI_EMBEDDING can be swapped for a custom function that calls a self-hosted model (vLLM with an encoder), SentenceTransformers, or a managed service. The syntax is the same; you change the provider.

Watermarks and late events

For cases where an event can arrive late (for example, the Postgres WAL falls behind because of a network blip), Flink lets you define watermarks:

CREATE TABLE documents_raw (
 doc_id STRING,
 title STRING,
 body STRING,
 updated_at TIMESTAMP_LTZ(3),
 WATERMARK FOR updated_at AS updated_at - INTERVAL '5' MINUTE
) WITH (...)

This tells Flink “assume no event arrives more than 5 minutes late relative to the event timestamp”. For temporal joins and aggregations, Flink uses the watermark to decide when to “close” a window.

Layer 4 — Sinks to vector stores

The last step is indexing the embeddings in a vector store. Three patterns in 2026:

Pattern A — direct Kafka Connect sink

Each vector store has its official connector:

Milvus: official Zilliz sink connector. Supports named/unnamed dense/sparse vectors.
Qdrant: official sink connector. Supports dense, sparse, multi-vector.
pgvector: has no dedicated connector, but the JDBC Sink Connector is used with custom SQL.
Weaviate: community connector.
LanceDB: community connector.

An example Milvus sink configuration:

{
 "name": "milvus-rag-embeddings-sink",
 "config": {
 "connector.class": "com.milvus.io.kafka.MilvusSinkConnector",
 "tasks.max": "3",
 "topics": "rag.documents.embedded",
 "milvus.host": "milvus.prod.internal",
 "milvus.port": "19530",
 "milvus.collection.name": "documents",
 "milvus.collection.dim": "1536",
 "milvus.collection.partition": "default",
 "key.converter": "org.apache.kafka.connect.storage.StringConverter",
 "value.converter": "org.apache.kafka.connect.json.JsonConverter",
 "value.converter.schemas.enable": false
 }
}

Three parallel tasks (tasks.max: 3) consume the embedded topic and write to the Milvus collection. Latency from “event in Kafka” to “indexable vector in Milvus” is typically <5 seconds.

Pattern B — pgvector with a direct CDC pipe

For teams that already live in PostgreSQL, pgvector is the lowest-friction option. The pattern: the same source Postgres cluster holds a second DB for embeddings with the pgvector extension enabled; the Flink pipeline writes to it directly over JDBC.

-- On the Postgres cluster with pgvector enabled
CREATE EXTENSION IF NOT EXISTS vector;

CREATE TABLE document_embeddings (
 doc_id TEXT,
 chunk_id INT,
 chunk_text TEXT,
 category TEXT,
 embedding vector(1536),
 embedded_at TIMESTAMP,
 PRIMARY KEY (doc_id, chunk_id)
);

CREATE INDEX ON document_embeddings
 USING hnsw (embedding vector_cosine_ops)
 WITH (m = 16, ef_construction = 64);

Advantages: the same DBA operates everything, transactionality across tables, trivial joins with relational metadata. Limitation: beyond 10M vectors, pgvector performance starts to give way compared with dedicated systems.

Pattern C — Confluent Tableflow → Iceberg + Flink SQL vector search

This is the 2026 development that changes the mechanics. Confluent Tableflow materialises Kafka topics automatically as Apache Iceberg or Delta Lake tables. Features:

No ETL pipeline: you do not write Flink/Spark jobs to move Kafka into a table. Tableflow does it.
Automatic schema evolution: changes to the topic schema are reflected in the table.
Unified catalogue: the table appears in Glue, Unity Catalog, Snowflake, Databricks. Any analytical engine queries it without copying data.
Native CDC: handles inserts, updates and deletes correctly.
30-50 % lower TCO according to the figures Confluent publishes against traditional pipelines.

And since 2026, Tableflow + Flink SQL offer native vector search integrated with Cosmos DB and Amazon S3 Vectors. The RAG query can be made directly in Flink SQL:

SELECT doc_id, chunk_text, category
FROM documents_embedded
WHERE VECTOR_SEARCH(embedding,
 OPENAI_EMBEDDING('user query', 'text-embedding-3-small'),
 top_k => 10) > 0.7
ORDER BY VECTOR_SEARCH_SCORE DESC;

This unifies layers that used to be separate (vector store + analytics). For many cases, it removes the need to maintain a dedicated vector store.

Confluent’s official MCP server

One piece added in 2026 deserves a mention: Confluent has published an official MCP server that exposes Kafka, Flink and Tableflow as tools accessible to AI agents over MCP. Any MCP client (Claude Desktop, Cursor, your own agents) can:

List topics, read recent messages, publish to topics.
Run Flink SQL queries in natural language (“give me the orders from the last 24 hours worth more than 1000 €”).
Query Tableflow Iceberg tables.
Manage Kafka Connect connectors.

This closes the circle: as well as reading data from the datalake through RAG (with vector search), your AI agent can write data to the log (through MCP) and trigger transformations (through Flink SQL in natural language). It is the deepest point of fusion between LLM ops and data ops this year.

A connection with the earlier series: this MCP server emits traces with the OpenTelemetry GenAI MCP semantic conventions we covered in the MCP observability post. The spans show up in Langfuse, Phoenix or your OTel backend with the right cardinality. Zero instrumentation code.

Vector stores: a 2026 comparison

The five dominant options:

Vector store	Licence	Operation	Where it fits
pgvector	Postgres ext, OSS	Your DBA	<10M vectors, Postgres-heavy team
Qdrant	Apache 2.0	Self-host or managed	Mid-scale, performance focus
Milvus	Apache 2.0	Self-host or Zilliz Cloud	Large-scale, scalability focus
Weaviate	BSD-3	Self-host or managed	Native hybrid search, semantically rich
LanceDB	Apache 2.0	Embedded or serverless	Small-medium, simplicity

The choice depends on:

Scale: pgvector falls short beyond 10M vectors. Milvus and Qdrant scale to billions.
Hybrid search: Weaviate brings lexical + vector natively. Others support it but less integrated.
Operation: pgvector if you already run Postgres. Qdrant if you want simplicity. Milvus if you need maximum scale.
Cloud managed: Zilliz Cloud for Milvus, Qdrant Cloud for Qdrant, Pinecone if you want pure SaaS (with no OSS behind it).

Freshness vs accuracy: the operational trade-off

A critical decision any RAG system on Kafka must answer: when is a new document considered “live” in the index?

Three options:

Synchronous streaming: the event reaches Kafka, Flink embeds it, the sink writes it to the vector store, and only then is it considered live. Typical latency: 1-5 seconds. The best freshness. But if the embedding model fails or the vector store is slow, events pile up in the topic.

Asynchronous streaming with a baseline: the event is considered live immediately; a background process embeds it when it can. In the meantime, queries asking for that document do not find it. Typical latency: 5-60 seconds. Acceptable for most applications.

Micro-batch: processing happens in mini-batches every 1-5 minutes. Less efficient than continuous streaming but more stable under variable load. Latency: 1-5 minutes.

The decision depends on the product’s SLA. For customer support chatbots, 5-60 seconds is acceptable. For systems that react to critical events (financial prices, alarms), synchronous streaming is necessary.

Schema evolution and re-embedding

When the embedding model changes (you move from text-embedding-3-small to text-embedding-3-large, or from OpenAI to Cohere), the vectors already in the index are incompatible: different dimensions, different semantic spaces. The distance between an old vector and a new one means nothing.

The standard pattern for handling this: dual-index during the migration.

T0: the active index is V1 (embedding model A).
T1: a parallel pipeline starts writing to a V2 index (embedding model B), consuming the topic from offset 0 (reprocessing the whole log).
T2: V2 has caught up with the present.
T3: you switch the retriever over to V2.
T4: a week later, you discard V1.

The Kafka log makes this pattern feasible because it is immutable and reproducible. Without the log, this pattern becomes a weeks-long data migration project.

Operational pitfalls

Topics without adequate retention

Configuring topics with 7-day retention on the grounds that “I already have the vector store” means losing the ability to rebuild if the vector store fails. Long retention (90+ days) or compaction by key for topics that feed RAG.

Heavy CDC during peak load

Debezium reading the WAL at peak hours can hit the source database’s performance. A dedicated read replica for Debezium, not the production primary. Or use logical replication restricted to just the tables you need.

Embedding cost run-away

OPENAI_EMBEDDING on every event of a topic with millions of messages a day comes to thousands of USD per month. Strategies: filter before embedding (only embed what adds value); deduplicate by content hash; use self-hosted open-source embedding models (BGE, E5, GTE) when the cloud cost is prohibitive.

Slow re-embedding due to limited throughput

Recomputing 10M embeddings with the OpenAI API at 3000 req/min takes 55 hours. If you wait for an incident to re-embed, that is two days without service. Embedding throughput is an explicit capacity planning item; reserve capacity or keep an offline job ready to start.

Downstream schema breaks

A change to the raw topic’s schema breaks Flink jobs downstream. Schema Registry with BACKWARD compatibility made mandatory; never ALLOW_ALL. And test schema evolution in CI.

A vector store with no backup

Your vector store holds 50M vectors. It is the only copy (the topics expired). A failure wipes it. Vector stores must be backed up like any primary persistence layer. For Milvus/Qdrant: periodic snapshots. For pgvector: plain pg_dump.

Multi-region without cross-cluster replication

Your RAG serves users in the US and the EU. The vector store is in US-east. Latency from the EU = 100 ms+ per query. MirrorMaker or Cluster Linking to replicate topics and vector stores in both regions.

What we have not covered

Hybrid search in production: combining BM25/lexical + vector + reranker. A topic for its own post.
Multimodal RAG: indexing images, audio and video alongside text. Multimodal embeddings (CLIP, Imagebind), a specific architecture.
GraphRAG: using structured knowledge (knowledge graphs) as well as vector retrieval. Microsoft GraphRAG, LlamaIndex KnowledgeGraphQueryEngine.
RAG with multi-tenant ACLs: filtering by permissions at runtime. The pattern uses metadata in the vector store + server-side filters.
Query rewriting with an LLM: using a first LLM to expand the query before retrieval (HyDE, multi-query, step-back prompting).

References

Kafka and stream processing:

Apache Kafka y Debezium.
Confluent Schema Registry y Apicurio Registry.
Apache Flink y Flink SQL docs.
RisingWave — alternativa SQL streaming con embedding built-in.

Vector store connectors:

Tableflow and 2026 architecture:

Streaming RAG:

Cross-references:

Previous post: MLOps specific to LLMs in 2026: the landscape.
Data series: PostgreSQL + Qdrant at the ingestion stage, RAG corpus curation: the active librarian, Reranker and hybrid retrieval — the four pieces that close the Data block: streaming, ingest, curation and retrieval.
Post-tracing series: Evals, Guardrails, MCP observability, eBPF + drift.
eBPF series: eBPF from zero to Cilium, Tetragon, Hubble, AgentSight.

MLOps specific to LLMs in 2026: the landscape of three modalities, six stages and ten tools that make them work

Thu, 21 May 2026 05:30:00 +0200

TL;DR

This is the blog’s fourth series and it is called MLOps specific to LLMs. It takes the operational craft of traditional MLOps (reproducible pipelines, model registries, dataset versioning, eval gates, controlled deployments) and redraws it for a world where the model is probabilistic, the outputs are subjective, the dependencies include external vendors that update weights without warning, and the “application” is not a model but an orchestration of models, embeddings, retrievers, guardrails and routers. Gartner predicts that more than 50% of enterprise GenAI deployments will fail before the end of 2026, and the main cause is not the model: it is that deterministic software assumptions were applied to probabilistic systems. This post opens the series with the frame: the seven structural differences between LLMOps and classic MLOps; the six-stage pipeline (data → tune → eval → deploy → observe → retrain); the three modalities of preparing a model (continuous fine-tuning, RAG over datalakes, agent training) with their decision matrix, where 60% of 2025-2026 deployments use hybrid because each modality solves a different problem: “fine-tune for behaviour, RAG for volatile knowledge”; and the 2026 tool landscape, which already forms reasonably stable layers: MLflow 3.10 (March 2026) as a GenAI-aware registry, W&B Weave and ZenML for tracing and pipelines, Kubeflow + KServe vLLM 0.8.1+ for serving, BentoML for flexibility, DVC + lakeFS (joined since November 2025) for data, Langfuse for prompts and observability. The three following posts will go down into the detail of the most critical pieces.

This is the opening of series 4: MLOps for LLMs. It continues the tradition of the previous series: LLM inference (the first), eBPF (the second) and post-tracing (the third). Here we enter the discipline that ties all the pieces together: how an LLM system is operated in production over months, not just deployed once.

The analogy: the SRE’s craft redrawn

Anyone who has worked for years as an SRE or a platform engineer recognises the classic pillars: reproducibility (same code + same data + same config = same result), observability (what happens can be measured), safe rollback (if something goes wrong, I go back within minutes), gradual rollout (the new thing goes to 1% before it goes to 100%). These pillars are not negotiable. The question is whether they hold up when the central component is an LLM.

The answer is: same pillars, radically different mechanics. Reproducibility: versioning code and data is no longer enough; you have to version prompts, retrieval configurations, snapshots of the vendor’s model (which change without warning). Observability: error and latency metrics are no longer enough; you have to measure subjective quality via LLM-as-judge and embedding drift. Rollback: dropping the binary’s version is no longer enough; you have to keep the old model cached because loading a new one takes minutes. Gradual rollout: a % of traffic is no longer enough; you have to decide what % of what kind of queries by segment.

It is the same craft, practised with partly new tools and reflexes. MLOps specific to LLMs, or “LLMOps”, as the field has christened itself, is the discipline that codifies those reflexes.

The seven structural differences between LLMOps and traditional MLOps

Before going down to the pipeline, let us fix the differences that make this territory new rather than a mere continuation. Each one has concrete practical consequences.

1. Non-deterministic outputs

Traditional MLOps: the model receives structured input and returns a bounded, reproducible prediction. Same input → same output. Unit tests work.

LLMOps: same input → different output every time (through sampling, through temperature, through the order of tools invoked, through the retrieval context that changed). The idea of a “unit test” breaks.

Operational consequence: tests over properties (was the tone maintained?, does it mention the source?, does it respect the JSON schema?), not over equality. Statistical evals over a distribution, not over samples.

2. Behaviour metrics, not statistical accuracy

Traditional MLOps: F1, accuracy, AUC, RMSE. Metrics with a clear number.

LLMOps: subjective rubric scores (G-Eval, faithfulness, helpfulness, toxicity), judge LLMs, human feedback. The “number” depends on who judges.

Operational consequence: platforms have to treat evals as versioned artifacts, not just “model v3 scored 0.87”, but “model v3 evaluated with the judge claude-3-5-sonnet-20251022 over the dataset gold-rag-v7 with the prompt judge-v2 scored 0.87”. Versioning the judge is as important as versioning the model being evaluated.

3. The model is an external dependency, not an internal asset

Traditional MLOps: you train the model, it lives in your registry, it does not change until you change it.

LLMOps: the base model belongs to Anthropic, OpenAI, Google, Meta. They change it on you without warning. The claude-3-5-sonnet version that answered well yesterday answers somewhat differently today.

Operational consequence: drift detection becomes far more critical (previous post). Pinning to specific snapshots (claude-3-5-sonnet-20251022) when the vendor allows it. For high-commitment apps, self-hosting the base model to guarantee reproducibility.

4. The application is an orchestration, not a model

Traditional MLOps: an app calls a model and consumes its output.

LLMOps 2026: an app connects foundation model + LoRA adapters + retrievers + vector stores + guardrails + routers + tool servers (MCP) + evaluators, all composing the final behaviour. Any component can degrade the result.

Operational consequence: cross-component debugging requires distributed tracing with OTel (covered in previous posts). The registry does not only store “the model” but the composition: which prompt version + which adapter + which vector store + which retriever config.

5. Cost per inference, not per training

Traditional MLOps: the high cost is training; serving is cheap. You optimise training.

LLMOps: the high cost is serving (every token costs, every call to the vendor is paid for, the GPUs doing the serving are switched on 24/7). You optimise inference.

Operational consequence: cost accountability per tenant, per agent, per tool. Metrics such as gen_ai.usage.input_tokens aggregated at customer and product level. Model decisions based on cost per query, not only on quality.

6. GPU-heavy infra with specific primitives

Traditional MLOps: CPU + some GPU for training. Standard Kubernetes.

LLMOps: Hopper/Blackwell SXM GPUs, NVLink/NVSwitch, tensor parallel, paged attention, KV cache. Infra that only fits into Kubernetes with primitives such as LeaderWorkerSet, GPU Operator, KEDA with LLM metrics (covered in vLLM on Kubernetes).

Operational consequence: the orchestration stack includes specialised operators (OME, vLLM Production Stack, NVIDIA Dynamo, llm-d, see LLM K8s Operators) that traditional MLOps does not contemplate.

7. RLHF and human feedback as a first-class citizen

Traditional MLOps: human feedback means labelling data before training.

LLMOps: human feedback lives inside the model in production, whether through RLHF of the foundation models (Anthropic, OpenAI), through RLAIF, through DPO, or through explicit user feedback that is fed back into fine-tuning.

Operational consequence: bidirectional production → training pipelines. Datasets grow with real incidents. Model decisions are taken with continuous feedback, not in a training project every N months.

Why Gartner predicts 50%+ failures

Gartner published that more than 50% of enterprise GenAI deployments will fail before 2026. The causes are not technical ones about the model but about the system:

Hallucinated outputs from poor grounding: badly designed RAG, poor retrieval, insufficient context.
Data architectures that are not ready: companies have data in silos, with no stable schemas, with no controlled freshness. Connecting an LLM to this data without a serious pipeline produces erratic answers.
Lack of structured workflows for prompt-driven systems: teams that treat prompts as code in hardcoded strings, with no versioning, no tests, no gates.

The conclusion the field draws: LLMOps is not optional. Companies that deploy GenAI without operational discipline fall into one of the three failure modes. Those that apply it (MLflow/W&B for tracking, DVC/lakeFS for data, Langfuse for prompts and evals, KServe or vLLM Production Stack for serving, drift detection in production) are the ones that keep the system working six months after the first release.

The six-stage LLMOps pipeline

On to the pipeline. The six stages that any serious LLM system goes through, in order:

[1. Data] → [2. Tune] → [3. Eval] → [4. Deploy] → [5. Observe] → [6. Retrain]
│
└─→ back to 1

Each stage is an operational domain of its own, with its own tools and traps:

Stage 1 — Data

What happens: ingestion, cleaning, curation, versioning, indexing of the corpus. It is where real projects suffer most, because companies have data in heterogeneous silos.

Typical sub-tasks: extraction from source (CDC over Kafka, batch from data lakes, scraping), cleaning (PII removal, dedup, format), curation (labelling for fine-tuning, golden datasets for eval), versioning (DVC + lakeFS), indexing (embeddings + vector store for RAG).

Traps: schema drift at the source, undetected PII, poor dedup that injects redundancy into training, a vector store that is not updated.

Stage 2 — Tune

What happens: preparing the model for your use case. Three modalities (we go deeper into them shortly): fine-tuning, RAG, agent training.

Typical sub-tasks: base model selection, adapter preparation (LoRA, QLoRA), training loop with continuous eval, hyperparameter sweep (Optuna, W&B Sweeps), saving the checkpoint.

Traps: catastrophic forgetting if the fine-tuning is very aggressive, overfitting to the golden dataset, no independent validation set.

Stage 3 — Eval

What happens: validating that the model + adapters + RAG configuration is acceptable before promoting. Covered in Evals.

Typical sub-tasks: running an eval framework (DeepEval, Promptfoo, Ragas) against the golden dataset, judge LLM evaluations, human review over a sample, gates with thresholds.

Traps: a golden dataset that ages, an uncalibrated judge, evals that pass in CI but fail in production because of distribution shift.

Stage 4 — Deploy

What happens: going from “the model evaluated well” to “the model serves real traffic”. Covered in LLM K8s Operators.

Typical sub-tasks: serving with vLLM/SGLang/TRT-LLM, runtime configuration, gradual rollout (canary, shadow, blue-green), routing between models (LiteLLM, OpenRouter, LangChain routers).

Traps: a naive rolling update that cuts sessions, autoscaling by CPU% that does not respond to LLM metrics (covered), a new model that performs worse in production than in eval.

Stage 5 — Observe

What happens: seeing what is happening in real time. Covered in the whole post-tracing series and the eBPF series.

Typical sub-tasks: tracing (Langfuse, LangSmith, Phoenix, OpenLLMetry), metrics (TTFT, TPOT, queue depth, cost per query), active guardrails (NeMo, Llama Guard), drift detection (Evidently, NannyML, WhyLabs).

Traps: cardinality explosion in metrics, batch evals with no tail-sampling over real traces, drift that is ignored until an incident makes it real.

Stage 6 — Retrain

What happens: closing the loop. Production feedback (incidents, worst-scoring cases, detected drift) generates new data to go back to stage 1.

Typical sub-tasks: extraction of problematic logs, human labelling of the sample, incorporation into the golden dataset, re-fine-tuning if applicable, decision on a new release.

Traps: an “open” loop where production never informs the dataset, human feedback that gets lost, lack of a clear retrain cadence.

The three modalities of “preparing the model”

Stage 2 (Tune) is where there is most confusion. In 2026 three modalities coexist, each solving a different problem:

Fine-tuning

What it does: modifying the model’s weights (or those of a LoRA/QLoRA adapter on top) so that it learns specific behaviour patterns: tone, output structure, idiomatic decisions of the domain.

When: when your main failure is behavioural inconsistency between calls. The model sometimes answers formally, sometimes not; sometimes it structures the JSON, sometimes not; sometimes it follows the company’s conventions, sometimes it invents them. Fine-tuning stabilises it.

When NOT: when what you need is up-to-date knowledge. Fine-tuning fixes knowledge in frozen weights; the day after the fine-tuning, the model knows nothing new.

RAG (Retrieval-Augmented Generation)

What it does: leaving the model intact and, on each call, retrieving fresh context from a knowledge base (vector store + lexical search, typically) and passing it to the model so that it answers on that basis.

When: when the knowledge you need is dynamic or very large. Documentation that changes, a product catalogue that is updated, an internal knowledge base that grows.

When NOT: when the problem is behavioural (RAG does not teach the model how to behave, it only gives it information). Or when retrieval is so noisy that the context arriving is worse than nothing.

Agent training

What it does: going beyond conventional fine-tuning with Reinforcement Learning techniques. RFT (OpenAI’s Reinforcement Fine-Tuning), classic RLHF, RLAIF (with AI feedback), DPO (Direct Preference Optimization) over datasets of (good, bad) pairs.

When: when the model needs to learn complex multistep trajectories: when to choose each tool, how to decompose a task, when to ask the user for confirmation. It is what is turning Claude, Gemini and GPT into agents capable of long tasks.

When NOT: when your case is simple chat or RAG. It is overkill, expensive and complicated for problems that the previous modalities solve.

Decision matrix

Observed problem	Modality
Inconsistent answers in tone/structure	Fine-tuning
The model makes things up (hallucinates)	RAG
Out-of-date knowledge (>1 year)	RAG
The model chooses tools badly	Agent training (RLAIF/RFT)
Mixed behaviour + knowledge	Hybrid (fine-tune + RAG)
Multi-step trajectory fails	Agent training
A specific language/regional style	Fine-tuning

The 2026 verdict: hybrid is the default

Multiple reports agree that in 2025-2026, around 60% of production projects use hybrid: fine-tuning for behaviour + RAG for knowledge. The key insight:

Fine-tune for behaviour (brand voice, decision protocol, output structure); use RAG for volatile knowledge you need the model to cite. Do not force a single tool to do both jobs.

A practical observation: the biggest quality improvements of 2025-2026 come from better reranking in RAG (cross-encoders), not from better embeddings. Rerankers add 15-35% of quality with little complexity.

On cost: combined fine-tuning + RAG tends to be 30-50% cheaper than pure RAG with frontier models at high volume, because the fine-tuned model can be smaller and cheaper while keeping equivalent quality.

The 2026 tool landscape

On to the concrete pieces, grouped by function. The field has matured enough for each piece to have 2-3 reasonable options and a couple of leaders.

Experiment tracking and model registry

MLflow remains the de facto standard, now with LLM-specific traction. MLflow 3 was published in June 2025; version 3.10.1 (March 2026) added:

A GenAI Overview dashboard with ready-made metrics for LLM apps.
Multi-workspace support for large teams.
Cost tracking in traces (gen_ai.usage.* aggregated per experiment).
MemAlign: a new specific eval algorithm.
Integrated native OpenTelemetry tracing.
First-class support for LangChain, LlamaIndex, AutoGen as frameworks.

MLflow treats prompts and agents as first-class citizens alongside classic models. It is the biggest change relative to MLflow 2.x.

Weights & Biases (W&B) with its LLM-specific product Weave offers tracing + eval + debug with a very polished UI. More commercial, less self-host friendly, but excellent UX.

ZenML is the piece that most cleanly integrates “classic MLOps + emerging LLMOps” into a single framework. Its automatic artifact versioning captures prompt templates, retrieval chunks and agent conversation histories with no extra work. Open-source. The most complete unification option that exists.

Dataset versioning

DVC remains the OSS standard. It extends Git to large files and pipelines. Important news from November 2025: lakeFS acquired DVC, consolidating the two OSS data versioning projects under one organisation. The combined roadmap is oriented towards LLM training and RAG datalakes specifically.

Typical pattern: Git for code + DVC for data/models + MLflow or W&B for experiment tracking + registry. Few teams use just one; the combination is what covers the cycle. Detailed in the dedicated post on data versioning for LLMOps — the four artifacts to version separately, schema contracts, end-to-end lineage, and a golden eval set with a strict holdout.

Prompt versioning and observability

Covered in depth in the AgentSight post where we go deeper into Langfuse as the OSS reference. Summary here:

Langfuse: MIT, self-host, prompt management with v1/v2/v3 versioning + labels + cache + linkage with traces.
LangSmith: if your stack is LangChain.
Arize Phoenix: ELv2, OTel-native.

Pipeline orchestration

For the LLMOps pipeline steps, the dominant options:

Kubeflow Pipelines: the K8s-native standard. KServe (the serving part of Kubeflow) has its vLLM runtime upgraded to v0.8.1+ with support for reasoning models, tool calling, embeddings, reranking, Llama 4 and Qwen 3.
ZenML: already mentioned; also a pipeline orchestrator.
Metaflow (Netflix-originated): Python-first pipelines, less LLM-specific but workable.
Argo Workflows: a pure K8s OSS alternative.
Flyte: Kubernetes-native, OSS.

Serving

Covered in depth in vLLM on Kubernetes and LLM K8s Operators. Summary:

vLLM Production Stack: a curated Helm chart.
KServe vLLM runtime: K8s-native, vLLM 0.8.1+ with full agentic support.
BentoML: flexible serving, popular in startups for its simplicity.
NVIDIA Dynamo: the successor to Triton.
llm-d: CNCF Sandbox.
OME: an LMSYS operator with native SGLang.

Evals and guardrails

Covered in Evals and Guardrails. Ultra-short summary:

Evals CI: DeepEval, Promptfoo, Ragas.
Evals platform: Langfuse, LangSmith, Phoenix, Braintrust.
Guardrails: NeMo Guardrails, Llama Guard 4, Llama Prompt Guard 2, LLM Guard, Lakera.

Drift detection and observability

Covered in the closing eBPF post. Summary:

Drift: Evidently AI, NannyML, WhyLabs.
Tracing: Langfuse, OpenLLMetry, Phoenix.
eBPF: AgentSight, Hubble, Tetragon, ProfInfer.

The typical 2026 stack table

Stage	Dominant piece	Alternatives
Data ingestion + versioning	DVC + lakeFS (unified Nov 2025)	Pachyderm, Quilt
Vector store / RAG index	Milvus, Qdrant, pgvector, Weaviate	LanceDB, Pinecone, Chroma
Experiment tracking	MLflow 3.10	W&B Weave, Neptune
Pipeline orchestration	Kubeflow + Argo	ZenML, Metaflow, Flyte
Model registry	MLflow registry	W&B Models, KServe ModelMesh
Prompt versioning	Langfuse	LangSmith, MLflow Prompts
Serving	vLLM Production Stack	KServe, BentoML, Dynamo, llm-d, OME
Evals CI	DeepEval, Ragas	Promptfoo, OpenAI Evals
Evals platform	Langfuse, Phoenix	LangSmith, Braintrust
Guardrails	NeMo + Llama Guard	LLM Guard, Lakera
Tracing	OpenLLMetry + Langfuse	Phoenix, LangSmith
Drift detection	Evidently AI	NannyML, WhyLabs
eBPF observability	AgentSight + Tetragon + Hubble	(new territory, few alternatives)

13 pieces. No org uses them all; any serious org uses at least six. This is the current LLMOps stack.

The operational reality: nobody uses a single tool

Multiple comparisons agree on one thing: the teams that win combine. Recurring patterns:

ZenML to orchestrate + MLflow for tracking + KServe for serving: the most popular OSS stack in companies coming from classic MLOps.
Kubeflow + W&B + BentoML: for teams with a research focus.
Langfuse + DeepEval + Phoenix + LiteLLM: for pure-LLM teams with no classic MLOps background.
MLflow + DVC + Argo + KServe: an idiomatic cloud-native stack without additional LLM-specifics (with its limitations).

The choice depends on the team’s background, on the licensing model they can afford, on the level of self-hosting they need, and on which frictions blocked them most in previous projects. There is no “one right answer”; there is a stable meta-pattern of layers worth covering.

Common operational traps

Treating the prompt as text in code

Hardcoding prompts in strings in the repo. Changing them requires a PR + redeploy. Result: teams that do not iterate on prompts because every change costs hours of pipeline. Solution: externalised prompt management (Langfuse, MLflow Prompts) with versioning, labels, hot-reload.

Skipping dataset versioning

“DVC is complicated, we will add it later”. Result: two months later, nobody knows which dataset trained which model. Impossible to reproduce incidents. Solution: DVC + lakeFS from day 1, even if only with a small subset.

Mixing layers in the same pipeline

Teams that put ingestion, fine-tuning, eval and deploy into a single giant pipeline. When something fails, the whole pipeline fails. Solution: independent pipelines per stage, with versioned artifacts as the interfaces between them.

Tracking with no structure

Logging everything to stdout and “we will look at it in CloudWatch later”. Result: impossible to correlate, compare, debug. Solution: OTel from day 1 with gen_ai.* semantic conventions.

Evals that block nothing

You have evals, you run them, you look at them, but they do not prevent the deploy if they drop. Eventually it drops gradually and nobody notices. Solution: eval gates in CI/CD that block the merge if critical metrics regress by more than X%.

No retrain cadence

You launch v1 and never go back to the model. Six months later, drift has degraded it but the team is on other projects. Solution: a formal retrain cadence (monthly, quarterly) tied to the production incident queue.

Invisible vendor lock-in

You start with the OpenAI API + LangSmith + Pinecone. When you want to self-host, you discover that migrating is a 3-month project. Solution: abstraction layers (LiteLLM, OpenLLMetry) and vendor-neutrality from the start.

What is coming in the following posts of the series

Post 2 — RAG over datalakes with Kafka: end-to-end technical architecture — the most hands-on one. Kafka as source-of-truth, Flink CDC, embedding pipelines, continuous indexing in Milvus/Qdrant, a complete example with real numbers and manifests.
Post 3 — The six-stage LLMOps pipeline: global architecture — the master map of the complete system with a reusable “you are here” SVG for the following posts. A deep dive into each of the six stages (Data, Tune, Eval, Deploy, Observe, Retrain).
Post 4 — PostgreSQL + Qdrant in the ingestion stage — synchronisation patterns (dual-write, outbox + CDC, event-driven), a complete microservices architecture, a Qdrant cluster manifest.
Upcoming posts — still to be decided: the cluster as a multi-tenant platform, Constitutional AI / alignment runtime, continuous fine-tuning in depth, edge LLMs.
Synthesis post — Anatomy of an LLM request in production, May 2026 — the complete journey of a real request through the six stages and the two cross-cutting components, with cross-links to each dedicated post. It works as a mental map of the blog and as an integrator’s guide.
Parallel OSS vs hyperscalers catalogue — The parallel catalogue: the six LLMOps stages in OSS and in AWS / GCP / Azure — for each stage of the pipeline, which tools the blog’s reference OSS stack uses and what the cloud equivalents are, with summary tables, identification of real gaps and the multi-tenant chatbot ported to an AWS stack component by component. A neutral editorial stance.
OSS catalogue entry by entry — The OSS catalogue for LLMOps in six stages: entry by entry — the zoom in on the open source side: ~150 words of description per core tool (vLLM, Langfuse, DVC, Qdrant, Airflow, NeMo Guardrails, Presidio…), licence and governance, a decision matrix per stage, a diagram of the connected OSS stack and a master table of licences and EE / SaaS offerings.

References

LLMOps vs MLOps:

Tools:

MLflow — registry + tracking + serving.
Weights & Biases Weave — LLM tracing.
ZenML — pipeline orchestration MLOps + LLMOps.
Kubeflow — K8s-native MLOps.
KServe — model serving K8s.
BentoML — flexible serving.
Metaflow — Netflix’s pipelines.
DVC — dataset versioning.
lakeFS — enterprise data versioning, acquired DVC in Nov 2025.

2026 comparisons:

RAG vs Fine-Tuning:

Cross-references (the three previous series):

LLM inference series: KV cache, vLLM on K8s, PagedAttention, LLM K8s Operators.
eBPF series: eBPF from zero to Cilium, Tetragon, Hubble, AgentSight.
Post-tracing series: Evals, Guardrails, MCP observability, eBPF + drift.

MCP from the inside and its deep observability: the LSP of AI agents and how to see everything with OpenTelemetry

Wed, 20 May 2026 06:00:00 +0200

TL;DR

Model Context Protocol (MCP) is the standard Anthropic published at the end of 2024 and which by 2026 has become the dominant protocol for connecting AI agents to external tools and data. Its value, the reason the whole industry adopted it in under 18 months, is that it solves a combinatorial problem: before MCP, integrating M AI apps with N tools required M×N ad-hoc integrations; with MCP, M + N. It is the same move the Language Server Protocol made in 2016 for code editors. The architecture is three well-defined roles, Host (the AI app), Client (the connection, one per server) and Server (the piece that exposes capabilities); the primitives are six, three on the server side (Tools, Resources, Prompts) and three on the client side (Sampling, Roots, Elicitation); the protocol is JSON-RPC over two transports, stdio for local processes and Streamable HTTP for remote. The operational challenge appears when there are 10-20 MCP servers running simultaneously, each with several tools, connected to an agent that chains multistep calls: observing what happens, where things fail, how much each tool costs, which tenant invokes what becomes critical. The ecosystem’s answer in 2026: the new OpenTelemetry GenAI semantic conventions for MCP (already stable), trace context propagation via params._meta (because JSON-RPC does not bring it natively), FastMCP with built-in OTel instrumentation, MCP Gateways as a centralised layer (Traefik Hub, MintMCP, OpenObserve), and MCP Inspector for interactive debugging. This article walks the architecture from the outside in, puts each concept in its exact place, and goes down to the detail of observability: traces, RED metrics, real use cases and pitfalls.

This is the third post in the post-tracing series. Previous posts: Evals and Guardrails. Here we go down to the protocol that connects agents to tools, and how to see it in production.

The master analogy (in three versions)

MCP is a communication protocol. Like any protocol, it is best understood with the right analogy. I am going to give you three, because each one lights up a different facet and the combination leaves you understanding it better than any technical definition.

Version 1 — The USB-C of AI apps (the official one)

This is the analogy Anthropic adopted when presenting it. Before USB-C, every electronic device had its own connector. Your phone had microUSB or Lightning, your laptop a proprietary power port, your headphones a 3.5mm jack, your external disk USB-A at one end and mini-USB at the other. The result: three boxes full of specific cables that got lost, none of them good for two things, and buying a new device meant buying new accessories.

USB-C changed that. A single physical connector that many protocols cross: data (USB 3, USB 4, Thunderbolt), video (DisplayPort), power (Power Delivery), audio. You connect anything to anything and it works; the protocols negotiate above.

MCP plays the same role for AI apps. Before MCP, every application that wanted to integrate tools with an LLM, Claude Desktop, Cursor, Continue, in-house custom agents, invented its own way of doing it. Every tool vendor had to write N different integrations, one per app. The result: massive fragmentation, a lot of duplicated code, integrations that broke when an app changed its internal API.

With MCP, the connector is one: any app that speaks MCP can use any MCP tool. Just as your USB-C talks to printers, monitors and disks without the printer “knowing” whether the cable is connected to a Mac or to a Linux box.

Version 2 — The LSP of code editors (the most technically precise)

This is my favourite because the analogy is structurally identical, not just metaphorical.

Until 2016, if you wanted your code editor to support a new language, Rust, Go, TypeScript, somebody had to write a specific plugin for your particular editor. VSCode had its Rust plugin, IntelliJ a different one, Vim another, Emacs another. Every decent feature (go-to-definition, autocompletion, refactoring) was an implementation duplicated N times. M editors × N languages = M·N integrations.

In 2016 Microsoft proposed the Language Server Protocol (LSP): each language implements a single “language server” (a process that understands that language); each editor implements a single LSP client; when you work with Rust code in VSCode, VSCode launches rust-analyzer as a subprocess and talks LSP to it over stdio. Any LSP editor + any LSP server = it works. M + N.

MCP is literally this pattern, transferred from “editor + language server” to “AI app + tool provider”. And it shares even the technical detail: both pass JSON-RPC over stdio (among other transports). When Anthropic designed MCP, they looked at LSP. Anyone coming from the world of editors and IDEs will find MCP familiar.

Version 3 — The operating system driver (the operational one)

Finally, an analogy that helps you understand what a specific MCP server does.

An operating system does not know directly how to talk to your particular HP LaserJet printer. What it knows is a generic interface: “print document”, “query status”, “cancel job”. The printer driver is the piece that translates that generic interface into the proprietary commands of your specific printer.

An MCP server does exactly the same:

Your AI agent knows a generic interface: invoke a tool with a defined schema, read a resource by URI, ask for a prompt template by name.
The MCP server is the driver: it translates those generic operations into the concrete APIs of the underlying system, your PostgreSQL database, your filesystem, your GitHub API, your Stripe.

This leaves the AI agent free from knowing how it authenticates with GitHub, what exact SQL PostgreSQL uses, what endpoints Stripe has. It speaks MCP; the server takes care of the details.

With the three analogies combined: MCP is the layer between the LLM and the world, a standard USB-C implemented like LSP in JSON-RPC, with each server acting as a driver for one concrete underlying system.

What concrete problem MCP solves

Before going down to the architecture, it is worth pinning down the specific problem MCP solves, because without that many design decisions look arbitrary.

The problem is the quadratic cost of integrations.

Imagine you have M applications that use LLMs (Claude Desktop, Cursor, Continue, ChatGPT Desktop, your own custom agent, and so on) and N external tools those LLMs could use (filesystem, GitHub, Slack, PostgreSQL, Jira, Notion, and so on). Without a standard:

Each (application, tool) pair requires a specific integration.
Every time the application changes its internal API, N integrations have to be updated.
Every time the tool changes its API, M have to be updated.
For your new tool to be adopted, you have to write M integrations.
For your new application to support the ecosystem, you have to write N.

The real result in 2023-2024: massive fragmentation. OpenAI’s function calling was not compatible with Anthropic’s tool use; every framework (LangChain, LlamaIndex, dspy) had its own wrapper; Claude Desktop plugins did not work in Cursor; and so on.

MCP breaks the quadratic. Each application implements the protocol once; each tool implements the protocol once; any pair works. M + N.

It is exactly what happened with USB-C, with LSP, with SQL (before there were proprietary APIs per database), with POSIX (before there were proprietary APIs per operating system). The pattern repeats because it always solves the same kind of problem.

The architecture: three roles, clearly placed

We are going to pin down where each thing lives, because mixing up the roles is the number one source of confusion in MCP.

Three roles. Let us pin down what each one does and where it physically lives.

Host: the AI application

The Host is the application the user opens. Claude Desktop, Cursor, Continue, ChatGPT Desktop, a custom agent your team builds, a VSCode extension. What the user perceives as “the product”.

The Host is responsible for:

Deciding which MCP servers to connect (configured by the user in a file or via the UI).
Launching or connecting to each MCP server.
Creating one MCP Client per server (it is 1:1, they are not shared).
Embedding the LLM (or calling it via API) that takes the decisions about which tools to use.
Mediating the user’s authorisation for sensitive actions (showing the human “the agent wants to run tool X, do you allow it?”).

Important: the LLM lives inside the Host, not in the servers. The servers are dumb; they execute operations when asked. The reasoning (“should I call this tool now?”) lives in the host’s LLM.

Client: the connection, one per server

An MCP Client is a specific connection between the Host and a Server. If your Host has 5 MCP servers configured, it has 5 clients, not one shared. Each client:

Maintains its socket or stdio pipe with the server.
Negotiates capabilities in the initial handshake (which protocol version, which primitives both support).
Serialises JSON-RPC requests to the server and deserialises responses.
Is the point where the Host invokes operations on the server.

The 1:1 client-server separation matters because it lets each server have its own session state, its specific permissions and its independent authenticated context. There is no multiplexing in the client.

Server: the piece that exposes capabilities

The MCP Server is the piece that implements the tool-provider side of the protocol. It receives JSON-RPC from the client, processes it, executes the action against the underlying system and returns a response.

There are two flavours physically:

Local server: it starts as a subprocess of the Host and communicates over stdio. Its lifecycle is the Host’s (when you close Claude Desktop, the local servers die). Typical model: your Host launches node filesystem-mcp-server.js as a child.
Remote server: it runs as an independent service, reachable over HTTP. Multi-tenant, authenticated, scalable. Typical model: a company publishes https://mcp.acme.com/v1 and many hosts connect to it.

This difference has enormous consequences for observability (we will come back to it shortly).

Summary of where each thing lives

Component	Lives in	How many there are	Speaks what with whom
Host	User’s machine	1 (the open app)	UI with the user; launches clients
LLM	Embedded in the Host (or cloud API)	1 (the main one)	Reasons; asks for tools
Client	Host	1 per server	JSON-RPC with its server
Local server	Subprocess of the Host	1 per local integration	stdio with its client
Remote server	External service	1 per service	HTTP/SSE with its clients
Underlying system	External	Depends	API/DB/FS, not MCP

If you get confused in a discussion, come back to this table. The number one source of errors in MCP is saying “the server” when you mean “the host”.

The two layers of the protocol

MCP separates the data layer and the transport layer. This separation is what lets the protocol work over local stdio and over remote HTTP without changing anything in the primitives.

Data Layer: JSON-RPC with MCP extensions

The data layer defines the vocabulary of the messages. It is JSON-RPC 2.0. Each message is a JSON with jsonrpc: "2.0", a method (eg tools/call, resources/read), params, and an id to correlate request with response.

On top of JSON-RPC, MCP adds:

Lifecycle: the initial handshake (initialize, initialized) that negotiates capabilities.
The primitives (next section): tools/*, resources/*, prompts/*, sampling/* and so on.
Notifications: messages without a response (eg notifications/cancelled to abort a tool in progress).
Meta-information: the params._meta field carries cross-cutting metadata by convention (trace context, request IDs).

Transport Layer: how the messages move

The transport layer defines how the JSON-RPC messages travel. Two official transports:

stdio: the client launches the server as a subprocess and they communicate over its stdin/stdout/stderr with JSON-RPC. One message per line, separated by a newline. No network, no TLS handshake, no auth (trust is inherited from the operating system itself: if you launch the subprocess, you trust it). Minimal latency (~100 μs round-trip), maximum bandwidth (memcpy, not a socket).

Use case: local servers that live on the same machine as the host. Most of the MCP servers you see in public directories are stdio.

Streamable HTTP: the client sends a POST to an HTTP endpoint on the server; the server responds with JSON, optionally opening a Server-Sent Events stream to send asynchronous notifications or long responses. Auth by bearer token, API key or custom headers.

Introduced in the November 2025 spec, it replaces the pure SSE transport of earlier versions, which had bidirectionality limitations. Use case: remote servers serving many simultaneous clients, with authentication and multi-tenancy.

Important: the primitives are the same on both transports. A tools/call is identical over stdio and over HTTP. The transport is accidental, not fundamental.

The six primitives: placed in the architecture

Here is the meat. There are six primitives in MCP. They are often confused because several look like they do similar things. The key classification: three live on the server side (the server exposes, the client consumes) and three on the client side (the client exposes, the server consumes).

Server-side: what the server gives the host

Tools are actions the server exposes. Each tool has a schema (typed parameters, description) and an implementation. When the host’s LLM decides to invoke a tool, the client sends tools/call to the server, which executes it and returns a result.

Example: the github-mcp server exposes create_issue(repo, title, body). The host’s LLM decides “I am going to create an issue”, calls this tool, github-mcp talks to the GitHub API, returns the issue ID to the LLM.
Architectural place: the server exposes them, the LLM consumes them.

Resources are contextual data the server exposes, addressable by URI. They are not actions; they are content reads. A resource has a URI (file:///path/to/doc.md, postgres://table/users), metadata and an endpoint to read content.

Example: the filesystem-mcp server exposes the files in the authorised directories as resources. The LLM asks for resources/read with the URI file:///docs/api.md and gets the text.
Architectural place: the server exposes them, the host reads them (and optionally passes them to the LLM as context).

Key difference between Tools and Resources: Tools are verbs (they execute, they modify state, they have side effects); Resources are nouns (they exist, they are read, they are idempotent). If you have something that is “search for text in files” → probably a Tool (an action). If it is “this specific file” → a Resource. The distinction matters for auditing and permissions: tools require more control.

Prompts are parameterised prompt templates the server exposes. The user or the host can invoke them to inject a conversational pattern into the model.

Example: a code-review-mcp server exposes a prompt review_diff(diff_text, style="strict") that returns a complete, well-written prompt asking the LLM to review code.
Architectural place: the server exposes them, the user or the host invokes them, the LLM receives them as input.

Prompts are the least used of the three primitives; many servers do not even implement them. But they let a team publish good prompts as a reusable library, separate from the agent.

Client-side: what the host gives the server

This is where MCP differs from protocols like HTTP REST: the server can also ask things of the host, it is not a one-way street. Three primitives travel in that direction.

Sampling: the server asks the host to run a generation with its LLM. That is, the server borrows the host’s LLM in order to reason.

Example: the search-mcp server receives a query from the agent, searches its corpus, finds 50 results and needs to summarise them before returning. Instead of having its own LLM, it sends a sampling/createMessage to the client; the host passes this to its LLM, runs the generation with the user’s permissions, and returns the summary to the server.
Architectural place: the server asks for it, the host (with its LLM and the user’s authorisation) fulfils it.
Why it matters: the user controls which model is used, what cost is paid, what permissions apply. The server does not need its own OpenAI API key.

Roots: the host tells the server where to look. Roots are URIs (directories, repositories, namespaces) the host authorises the server to explore.

Example: your Claude Desktop starts filesystem-mcp with roots [file:///Users/me/projects]. The server knows it must only operate inside that folder, not in /etc/passwd.
Architectural place: the host declares them in the handshake, the server respects them.

Elicitation: the server asks the host for additional information from the human user via a structured UI.

Example: the stripe-mcp server is about to process a refund of €5000. Before executing, it sends an elicitation/createMessage to the client; the host shows the user “Confirm this €5000 refund” with a button; when the user confirms, it returns OK to the server, which then proceeds.
Architectural place: the server asks, the host shows the user, the user decides, the answer goes back to the server.
It is the key primitive for human-in-the-loop on sensitive actions.

Visualising the flow of the six primitives

 HOST SERVER
│ │
Server-side ─────┼─────────────────────────────────────┤
│ │
tools/list ──────┼─────── asks what tools exist ──────▶│
│◀────── returns the list ────────────│
│ │
tools/call ──────┼────────── run this tool ───────────▶│
│◀────── result ──────────────────────│
│ │
resources/read ──┼────────── read this URI ───────────▶│
│◀────── content ─────────────────────│
│ │
prompts/get ─────┼─────── give me this prompt ────────▶│
│◀────── compiled prompt ─────────────│
│ │
Client-side ─────┼─────────────────────────────────────┤
│ │
sampling ────────│◀────── I need a generation ─────────│
│── use my LLM ───┐ │
│── returns ──────▼──────────────────▶│
│ │
roots ───────────┼───── declared in the handshake ────▶│
│ │
elicitation ─────│◀────── ask the user X ──────────────│
│── shows UI ────┐ │
│── confirms ────▼───────────────────▶│

JSON-RPC in action: a concrete example

To make the theory concrete, here is a real MCP conversation between a client and the filesystem-mcp server:

// 1. Initial handshake (client → server)
{
"jsonrpc": "2.0", "id": 1, "method": "initialize",
"params": {
"protocolVersion": "2026-03-01",
"capabilities": {
"sampling": {}, // this client supports sampling
"roots": { "listChanged": true }
},
"clientInfo": { "name": "ClaudeDesktop", "version": "1.2.0" }
}
}
// 2. The server responds with its capabilities
{
"jsonrpc": "2.0", "id": 1, "result": {
"protocolVersion": "2026-03-01",
"capabilities": {
"tools": { "listChanged": true },
"resources": { "subscribe": true, "listChanged": true },
"prompts": {}
},
"serverInfo": { "name": "filesystem-mcp", "version": "0.5.2" }
}
}
// 3. The client asks for the list of tools
{
"jsonrpc": "2.0", "id": 2, "method": "tools/list"
}
// 4. The server returns its tools with a schema
{
"jsonrpc": "2.0", "id": 2, "result": {
"tools": [
{
"name": "read_file",
"description": "Read a file from the filesystem",
"inputSchema": {
"type": "object",
"properties": { "path": { "type": "string" } },
"required": ["path"]
}
},
{ "name": "write_file", "description": "...", "inputSchema": {} },
{ "name": "list_directory", "description": "...", "inputSchema": {} }
]
}
}
// 5. The LLM decides to call read_file; the client sends tools/call
{
"jsonrpc": "2.0", "id": 3, "method": "tools/call",
"params": {
"name": "read_file",
"arguments": { "path": "/Users/me/projects/notes.md" },
"_meta": { // ← the extension where trace context will go
"traceparent": "00-abc123...-def456-01"
}
}
}
// 6. The server returns the file content
{
"jsonrpc": "2.0", "id": 3, "result": {
"content": [
{ "type": "text", "text": "# My notes\n\n..." }
]
}
}

The important thing to note: params._meta. That is the bag where MCP conventionally passes cross-cutting metadata, including trace context. We will come back to it shortly.

The observability problem: why traditional tracing is not enough

So much for theory. Let us go down to the operational problem: in a 2026 production cluster, a typical agent has 5-15 MCP servers connected simultaneously, each with 5-20 tools, and each conversation with the agent can generate dozens of chained tool calls. Without observability, debugging incidents is impossible.

Why generic tracing (Hubble, OTel without MCP conventions) is not enough:

Stdio is not visible on the network. Local servers talk over OS pipes. Your Hubble or your Datadog APM see nothing; there are no packets to capture. AgentSight (seen in the previous post in the eBPF series) with stdiocap captures it but gives you the raw JSON-RPC, without semantic context (which tool it is, which resource, which prompt).

Generic HTTP does not understand MCP either. If you trace the HTTP to a remote MCP server without MCP conventions, you see a POST to /v1 with an opaque JSON-RPC body. You lose “which tool was invoked”, “what arguments”, “was it elicitation or sampling”. RED metrics per endpoint are no use to you; you need RED per tool.

JSON-RPC does not propagate trace context natively. Unlike HTTP (the W3C traceparent header) or gRPC (metadata), JSON-RPC has no standard field for trace context. If you do not propagate it, every call to the server starts a new trace disconnected from the agent’s trace.

Multistep multi-server is very hard to follow. A single user conversation can translate into: 1) a call to github-mcp get_pr; 2) a call to filesystem-mcp read_file for several files; 3) a call to the main LLM with all the context; 4) a call to postgres-mcp query; 5) a call to slack-mcp send_message. Without propagated trace context, those are five disconnected traces. With propagation, it is one tree.

The solution: OpenTelemetry semantic conventions for MCP, already stable in 2026.

OpenTelemetry semantic conventions for MCP

The GenAI MCP semantic conventions are the set of standardised attributes for MCP-related spans and metrics. They were published as part of the GenAI subgroup of the OpenTelemetry SIG and are the first part of the GenAI semantic conventions to reach stable.

Why MCP-specific semantic conventions

Before they existed, teams instrumented MCP with the generic RPC semantic conventions (the ones you would use for gRPC or XML-RPC). It half worked. The MCP-specific conventions add:

Attributes to identify which primitive was executed (mcp.method.name = "tools/call").
Attributes to identify which specific tool/resource/prompt was touched (mcp.tool.name, mcp.resource.uri, mcp.prompt.name).
Attributes for the bidirectional flow (sampling/elicitation requests from the server to the client).
Attributes for the handshake (mcp.protocol.version, mcp.client.name, mcp.server.name).
Standardised RED metrics per tool (mcp.tool.call.duration, mcp.tool.call.errors).

The canonical attributes

The attributes any MCP-aware instrumentation should emit:

Attribute	Meaning	Example
`mcp.method.name`	JSON-RPC method	`"tools/call"`
`mcp.tool.name`	Name of the tool	`"read_file"`
`mcp.resource.uri`	URI of the resource	`"file:///docs/api.md"`
`mcp.prompt.name`	Name of the prompt	`"code_review"`
`mcp.session.id`	MCP session ID	`"sess-abc123"`
`mcp.protocol.version`	Protocol version	`"2026-03-01"`
`mcp.client.name`	Client identity	`"ClaudeDesktop/1.2.0"`
`mcp.server.name`	Server identity	`"filesystem-mcp/0.5.2"`
`mcp.transport`	Transport used	`"stdio"` or `"http"`
`mcp.error.code`	JSON-RPC error code	`-32602` (Invalid params)
`gen_ai.usage.input_tokens`	Tokens consumed (if sampling)	`1240`
`gen_ai.usage.output_tokens`	Tokens generated (if sampling)	`512`

The last two come from the generic GenAI semantic conventions and apply when the MCP call involves sampling (the server using the client’s LLM).

RED metrics per tool

Beyond spans, the semantic conventions define three core metrics:

mcp.tool.call.duration (histogram): the latency of each invocation.
mcp.tool.call.count (counter): the total number of invocations.
mcp.tool.call.errors (counter): errors per tool.

Labelled with mcp.tool.name, mcp.server.name, mcp.client.name. Pivotable in Grafana to answer “which tool is the slowest”, “which tool fails most”, “which client loads which server hardest”.

Trace context propagation: the `params._meta` trick

JSON-RPC has no headers like HTTP, so MCP cannot use the W3C traceparent header directly. The solution the ecosystem has converged on: propagate trace context in params._meta.

When the MCP client sends a tools/call, its OTel instrumentation does:

import json
from opentelemetry.propagate import inject

carrier = {}
inject(carrier) # fills it with traceparent/tracestate from the active span

params = {
 "name": "read_file",
 "arguments": {"path": "/notes.md"},
 "_meta": carrier, # ← propagates trace context
}

When the server receives it, it does the symmetrical thing:

from opentelemetry.propagate import extract

ctx = extract(request.params.get("_meta", {}))
with tracer.start_as_current_span("tools/call", context=ctx):
 # this span is a child of the client's
 return execute_tool(request.params)

The result: the server’s span is a child of the client’s span in the trace tree. When you look at the trace in Tempo or Phoenix, you see the whole chain: user → host → client → server → execution → response → client → host → response to the user.

This requires both ends to instrument consistently. If the server does not extract the context, you see disconnected spans but at least you have traceability on the client side.

Instrumentation patterns

There are three routes to instrumenting MCP, in increasing order of effort:

1. FastMCP with built-in OpenTelemetry

FastMCP is one of the most used Python frameworks for building MCP servers. It ships built-in OpenTelemetry instrumentation: every tool, resource template and prompt operation generates spans automatically with the correct MCP conventions.

from fastmcp import FastMCP
from opentelemetry.sdk.trace.export import OTLPSpanExporter

mcp = FastMCP("my-server", otel_endpoint="https://otel-collector:4318")

@mcp.tool()
def search_docs(query: str) -> str:
 """Search the corpus for matching documents."""
 # this automatically generates a span with
 # mcp.tool.name=search_docs, mcp.method.name=tools/call, etc.
 return run_search(query)

Zero instrumentation code. Spans with the correct conventions. It is the recommended pattern if you are starting an MCP server in Python from scratch.

2. Manual OpenTelemetry SDK

For servers that already exist or are in other languages (TypeScript, Go), the option is to instrument manually with the standard OTel SDK and emit the conventional MCP attributes:

from opentelemetry import trace
tracer = trace.get_tracer(__name__)

async def handle_tools_call(req: JSONRPCRequest):
 ctx = extract_trace_context(req)
 with tracer.start_as_current_span("mcp.tools.call", context=ctx) as span:
 span.set_attribute("mcp.method.name", "tools/call")
 span.set_attribute("mcp.tool.name", req.params["name"])
 span.set_attribute("mcp.server.name", "filesystem-mcp")
 try:
 result = await execute_tool(req.params)
 return result
 except Exception as e:
 span.set_attribute("mcp.error.code", -32603)
 span.record_exception(e)
 raise

More boilerplate but it works with any existing server.

3. MCP Inspector for interactive debugging

MCP Inspector (official) is a tool for interactive protocol-level debugging. It launches a local proxy (port 6277) between your client and the server, and opens a web UI (port 6274) where you see every JSON-RPC message back and forth in real time.

It is not production observability, it is development and debugging. But it is irreplaceable during the bring-up of a new server: you see exactly which requests arrive, which responses are returned, which errors occur. It saves hours of ad-hoc logging.

MCP Gateways: the centralised piece for enterprise

When your organisation has many agents connecting to many MCP servers, managing the connection matrix becomes operationally serious. The natural question, “can there be a proxy in front of all the MCP servers that centralises auth, rate limiting, logging and observability?”, already has an answer: MCP Gateways.

An MCP Gateway is a proxy that:

Accepts MCP connections from the hosts/agents.
Routes them to the corresponding backend MCP servers.
Applies centralised authentication and authorisation (which agent can call which tool).
Applies rate limiting per agent, per tool, per tenant.
Observes: it emits OTel metrics for every operation passing through.
Propagates the identity of the agent to the backend server (with several models: token forwarding, token exchange, impersonation).

The options that have established themselves in 2026:

Traefik Hub MCP Gateway — from the Traefik team. Declarative configuration, native integration with Traefik’s Kubernetes/Helm ecosystem.
MintMCP — a gateway focused on observability and multi-tenancy. SaaS and self-host.
OpenObserve MCP Gateway — integrated with the OpenObserve observability platform.

For small deployments (one team, few agents) a Gateway can be overkill. For enterprise (dozens of agents, dozens of servers, regulated compliance), it is practically mandatory.

Real use cases for MCP observability

Let us land this with five cases where properly instrumented MCP observability delivers immediate value:

1. Audit per tool, per tenant, per agent

Question: “who ran the delete_repo tool last month?”. Without MCP observability, impossible. With OTel conventions + identity propagation: a query in your trace backend filtering by mcp.tool.name="delete_repo", grouping by mcp.client.name or by the user_id propagated in _meta. Compliance happy.

2. Cost per tool and per tenant

Question: “how much does each tool cost?”. If the tools invoke external APIs (Stripe, OpenAI sampling) or consume significant resources (a GPU for an inference tool), knowing their aggregated cost matters. With mcp.tool.call.duration + gen_ai.usage.* aggregated by tool and tenant, you build cost accountability dashboards without instrumenting anything extra.

3. Debugging multistep chains that fail

Question: “the agent failed to complete this task, where did it go wrong?”. The propagated trace connects: the user’s span → the LLM’s span with its CoT → the spans of each tool invoked → the final LLM span. If the chain broke at the third tool, in Tempo you see the red span with the specific error message. Reproducing the failure is trivial.

4. Tool latency and degradation

Question: “which tool is degrading?”. RED metrics per tool in Grafana show p95/p99 latency over time. When a tool starts climbing from 200ms to 800ms (because the underlying service is collapsing), you see it before the users complain.

5. Detecting loops and agentic anomalies

Question: “is any agent stuck in a loop?”. If an agent calls tools/call read_file 80 times in 30 seconds for the same path, something is clearly wrong. An alert on mcp.tool.call.count grouped by (session_id, tool_name) detects this. Combined with loop detection at the reasoning level, it closes the circle.

Operational pitfalls

Lack of identity propagation

Your Gateway authenticates the agent, but passes requests to the backend without propagating identity. The result: the backend’s logs say “service-account” for everything, and it is impossible to audit who invoked what. Choose a propagation strategy early: token forwarding (simple, exposes tokens to the backend), token exchange (more secure), or impersonation with cross-logging.

Stdio servers that do not show up in your APM

This is the field’s number 1 pitfall. Your Cursor agent uses filesystem-mcp over stdio; you see nothing in Datadog because there is no network traffic. Solution: instrument the stdio server with an OTel SDK that exports over OTLP to your collector (via gRPC or HTTP; the OTel collector can receive even though the server talks stdio to its client). Or use AgentSight stdiocap to capture the raw JSON-RPC and process it offline.

Multiple protocol versions in production

Different clients use different MCP versions simultaneously. Your metrics dashboard mixes apples and oranges. ALWAYS label with mcp.protocol.version and filter/group by it.

`_meta` lost when passing through a proxy

Your Gateway accepts the client’s request, rewrites it for the backend, and forgets to copy params._meta. The result: the trace is broken at the Gateway, two disconnected traces. Make sure your Gateway preserves or re-injects trace context at each hop.

Trace volume with chatty servers

Some MCP servers emit many small operations (filesystem listings, partial reads). Without sampling, they fill your backend with useless traces. Apply tail-based sampling that keeps complete sessions or keeps only traces with errors or high latency.

Cardinality in metrics

mcp.tool.call.duration with mcp.session.id as a label blows up cardinality. Do not include unique per-session IDs in labels; keep cardinality under control with labels that take few discrete values (tool name, server name, client name, error code).

Confusing client and server spans

When you look at the tree, distinguish: the client sees the total latency from its perspective (network included); the server sees only its own work. If you look only at the server span to debug the latency perceived by the user, you miss the RTT. Use both.

What we have not covered

Experimental WebSocket MCP transport: an alternative to Streamable HTTP, not yet standard.
MCP servers in cloud-native deployments with sidecars: an emerging pattern of deploying MCP servers as pod sidecars.
MCP federation: composing several servers as one (similar to GraphQL federation).
eBPF + MCP: how AgentSight’s stdiocap and Cilium’s hooks complement native instrumentation.
MCP testing and contract tests: how to validate that your server complies with the spec.

References

Specification and concepts:

Model Context Protocol — sitio oficial — entrada canónica.
MCP architecture overview.
Transports — MCP docs.
MCP Inspector (GitHub) — debugging interactivo.

OpenTelemetry GenAI MCP:

Semantic conventions for Model Context Protocol — OpenTelemetry — referencia normativa.
Adding OpenTelemetry Trace Support to MCP (Discussion #269) — historia de la propuesta.
How to Instrument MCP Servers with OpenTelemetry (OneUptime).
How to trace MCP server tool calls with OpenTelemetry and Elastic APM.
MCP Observability with OpenTelemetry (SigNoz).
Distributed tracing for agentic workflows (Red Hat Developer).
OpenTelemetry for AI Agents in MCP Workflows (MintMCP).

Frameworks and gateways:

FastMCP OpenTelemetry — instrumentación built-in.
Traefik Hub MCP Gateway — gateway de Traefik.
MintMCP — gateway con foco en observabilidad.
OpenObserve MCP Gateway guide.
What is an MCP Gateway (DEV Community).
OpenTelemetry MCP Server (Traceloop) — el patrón inverso: usar MCP para que agentes consulten traces OTel.

Cross-references:

Post anterior: Guardrails y safety.
AgentSight y el nuevo tracing de LLMs — donde se introdujo stdiocap para capturar stdio de servidores MCP locales.
Evals: la capa después del tracing.

AgentSight and the new LLM tracing: zero-instrumentation with eBPF against Langfuse, LangSmith, Phoenix and company

Tue, 19 May 2026 18:00:00 +0200

TL;DR

Observing an LLM agent in production in 2026 splits into two approaches with opposite philosophies. The instrumented one, dominant until 2025, lives in tools such as Langfuse, LangSmith, Arize Phoenix, Helicone, OpenLLMetry/Traceloop or Pydantic Logfire: you install an SDK, decorate your calls, emit spans with the OpenTelemetry GenAI convention (gen_ai.request.model, gen_ai.usage.input_tokens, and so on) and export them to a backend. Enormous depth when you control the code; zero visibility when the agent is an opaque binary you run without instrumenting. The zero-instrumentation one, which AgentSight popularised in the second half of 2025, turns the perspective 180º: it puts eBPF hooks on the uprobes of the SSL/TLS libraries and captures the plaintext of every HTTPS request before encryption, without touching the app’s code, with less than 3% overhead and the guarantee of being tamper-proof (the agent cannot falsify what is seen in the kernel). Combined with BPF capture of stdio for local MCP servers, AgentSight gives you complete observability of any agent, including closed binaries such as Claude Code, Gemini CLI or Cursor, on a Kubernetes cluster. The two families are not enemies: the 2026 reference stack combines both (instrumented for your own apps with LangChain, eBPF for opaque binaries and tamper-proof compliance) over OpenTelemetry GenAI semantic conventions as the common vocabulary the ecosystem is stabilising this year.

This is the fourth and last post of the eBPF series. Part 1: eBPF from zero to Cilium. Part 2: Tetragon: runtime security. Part 3: Hubble: network observability. Here we close the circle with the semantic dimension: what an AI agent does, not only what network it opens or what syscalls it emits.

The analogy: traditional APM vs network sniffer

Anyone who has operated enterprise applications knows the two tribes of monitoring. The APM tribe (New Relic, AppDynamics, Datadog APM): you install an agent or an SDK in each application, mark spans, collect traces with enormous depth inside each process, down to lines of code, SQL queries, Java methods. The wire-level tribe (network sniffers, SolarWinds NPM-style tools, NetFlow): it does not touch the application; it observes the cable, sees protocols, latencies, retransmissions, identifies problems the app does not know it has.

Each one sees different things and both are useful. Anyone who has lived through a serious incident where APM said “all green” while users suffered knows that wire-level would have caught the problem (a saturated middlebox, a badly configured MTU, a TCP timeout). Anyone who has tried to debug a memory leak with sniffers knows it was impossible without APM.

LLM agent observability in 2026 is exactly at this point. The APM-style has been in place for a couple of years: Langfuse, LangSmith, Phoenix, OpenLLMetry. Enormous depth, requires instrumenting the app. The wire-level with eBPF has just arrived: AgentSight is the first project to take it to production. Less depth inside the agent, but it sees any agent without touching anything and it is tamper-proof. Both are useful. The industry is in full coexistence.

Why observing LLM agents is different

Before getting into tools, it is worth pausing on what makes LLM agents specific as subjects of observability:

Non-determinism. The same input can produce different outputs. Reproducing an incident requires capturing exactly the conversation, the model, the parameters and, ideally, the seed. An aggregate “p95 latency” metric falls short; what you need is replay of the individual trace.

A chain of external invocations. A typical agent calls LLM → tools (tool calling) → MCP servers → other APIs → back to the LLM. A chat session can generate dozens of chained calls that have to be correlated by trace_id to understand the decision.

Cost linear in tokens. Every call is paid for in tokens. Without tracing input/output tokens per request, you cannot assign cost to a tenant or a team, nor detect loops that eat your budget in an hour.

Semantic risk. Prompt injection (a user input containing instructions to manipulate the model), jailbreaks, secret leakage through tool calls. It is a class of problem that does not appear in traditional applications, and observability must see it.

Opaque binaries. In 2026, many teams deploy third-party agents such as Claude Code, the Cursor agent, Aider, Gemini CLI or Codex CLI as internal tools. They are not your own applications; they are closed binaries that call the vendor’s API. Instrumenting them is impossible. Observing them requires something else.

Multi-agent and orchestration. More and more architectures have agents that invoke other agents (planner → executor → critic). Observability must understand the topology, not just the individual span.

With these five points in mind, the tools we are about to see differ mainly in which parts of the problem they cover well and which parts they leave blind.

The instrumented approach: how it works

The model is direct and familiar:

Your code calls the LLM or the tools using an official library: openai, anthropic, langchain, llama_index, dspy.
You install a tracer SDK (Langfuse, LangSmith, OpenLLMetry, Logfire) that wraps or monkey-patches those libraries.
Each call emits an OpenTelemetry span with standardised attributes: model used, input/output tokens, latency, parameters, messages, tool invoked, result.
The spans are exported over OTLP to a backend that shows them as a tree of traces.

# Typical example with OpenLLMetry + any SDK
from traceloop.sdk import Traceloop
from openai import OpenAI

Traceloop.init(app_name="my-agent", api_endpoint="https://otel-collector:4318")

client = OpenAI()
# this call automatically emits a span with
# gen_ai.request.model, gen_ai.usage.input_tokens, etc.
resp = client.chat.completions.create(
 model="gpt-4.1",
 messages=[{"role": "user", "content": "..."}]
)

What you see afterwards: a dashboard with each conversation as a trace, each call as a span, the complete prompts and completions (if you opt in), the computed cost, latencies per span, errors flagged.

OpenTelemetry GenAI semantic conventions: the common vocabulary

The fragmentation of the field is being mitigated with OpenTelemetry GenAI Semantic Conventions. It is the CNCF’s effort to get all tools to emit spans with the same attribute names:

gen_ai.system — the provider (openai, anthropic, vertex_ai, and so on).
gen_ai.request.model — the model requested (gpt-4.1, claude-3-5-sonnet).
gen_ai.response.model — the model actually used (sometimes it differs, e.g. fallbacks).
gen_ai.usage.input_tokens and gen_ai.usage.output_tokens — counters.
gen_ai.request.temperature, gen_ai.request.top_p, and so on — parameters.
gen_ai.response.finish_reasons — why it ended (stop, length, content_filter).
gen_ai.operation.name — the type of operation (chat, embedding, completion).

In early 2026, the client spans moved from experimental to stable. The rest (server spans, multi-agent events) is still in development. The operational meaning: if your SDK emits these attributes, any backend that understands OTel GenAI can consume them. Moving from Langfuse to Phoenix to Helicone does not imply re-instrumenting, only changing the exporter.

The SIG is actively developing conventions for multi-agent systems: agent teams, tasks, actions, memory, artifact tracking. This is what is missing for complex agent architectures to have a common vocabulary. In 2026 it is experimental; stabilisation is expected at the end of the year or the start of 2027.

Instrumented tools: the 2026 landscape

Tool	Licence	Self-host	Focus	Where it shines
Langfuse	MIT	Yes	LLM observability + evals + prompt mgmt	Best OSS balance, complete suite
LangSmith	Commercial	No	Native LangChain/LangGraph	If you use LangChain, zero-config integration
Arize Phoenix	ELv2 (OSS)	Yes	OTel-native, strong RAG	Vector DBs, retrieval, embeddings
Helicone	Commercial + OSS lite	Yes (lite)	Simple proxy	Setup in minutes, OpenAI-only
OpenLLMetry / Traceloop	Apache 2.0	Yes	OTel SDK for LLMs	Vendor-neutral, exports to any OTel backend
Pydantic Logfire	Commercial	No	App + LLM unified	If you use Pydantic AI, native integration
Weights & Biases Weave	Commercial	Limited	Experimentation + production	If you already use W&B for training
Laminar / Braintrust	Commercial	No / Yes	Evals + tracing	More recent, focused on evaluation

Deep dive: Langfuse

It is worth pausing on Langfuse because in 2026 it is the default choice among the open-source options and the one most teams have adopted this year. It is a YC W23 project, MIT licence, and keeps a sustained release cadence with serious architectural changes between versions.

Four declared pillars: observability (tracing), evaluations, prompt management, playground/datasets. Each one separately has complete commercial products behind it; Langfuse integrates them into a single platform with a single backend.

SDK v4: OTEL-native, not a replacement

The big recent operational change is the SDK v4, a thin layer over the official OpenTelemetry client. The choice is deliberate: instead of maintaining a client of its own that would fall behind the OTel primitives, Langfuse uses the standard SDK and enriches the spans with LLM-specific attributes and helpers. The consequence: any code already instrumented with vanilla OpenTelemetry (@opentelemetry/sdk-node, opentelemetry-sdk in Python) can export to Langfuse without major changes, and the other way round, if tomorrow you want to migrate from Langfuse to another OTel backend, the spans are portable.

In Python the idiomatic decorator is @observe:

from langfuse import observe, get_client

langfuse = get_client()

@observe()
def buscar_documentos(query: str):
 # any internal call is traced as well
 return vector_store.similarity_search(query)

@observe(as_type="generation")
def llamar_llm(prompt: str):
 # marked as "generation" so it appears with LLM metadata
 return openai_client.chat.completions.create(...)

@observe()
def pipeline_rag(pregunta: str):
 docs = buscar_documentos(pregunta)
 return llamar_llm(build_prompt(pregunta, docs))

The call tree is captured automatically: the trace shows pipeline_rag as the root span, with buscar_documentos and llamar_llm as children, nested. Without writing a single with tracer.start_as_current_span(...) by hand.

In TypeScript the equivalent is modular: you install @langfuse/tracing, @langfuse/otel and @opentelemetry/sdk-node, and you can use TS decorators, context managers or manual spans, the three models interoperate. The consequence: third-party libraries that emit OTel spans (openai, @anthropic-ai/sdk, Vercel AI SDK instrumentations) show up in Langfuse with no extra work.

Self-host architecture: designed for serious production

The Langfuse backend architecture has two explicit decisions that set its self-host deployment apart:

Persistence in S3/Blob Storage first. When a tracing event arrives, it is persisted in object storage before touching the database. Only when the later processing confirms OK is it inserted into Postgres/Clickhouse. If the DB goes down temporarily, the events are not lost; they stay in S3 waiting to be reprocessed. For production where losing the traces of an incident is equivalent to losing evidence, this is load-bearing.
Long migrations as background jobs. Schema upgrades that on other platforms mean a downtime window run in the background on Langfuse while the application keeps serving. Upgrade downtime drops drastically.

The officially supported deployment modes:

Docker Compose: for development and POCs. One command, everything up.
VM: a single node, containers, no orchestration. For small environments.
Kubernetes with Helm: the recommended mode for production. Official chart maintained. Supports external Postgres, external Clickhouse, external S3, HPA.

The external dependencies in a typical production setup: Postgres (metadata, prompts, configuration), Clickhouse (tracing events, high-cardinality queries), S3 or a compatible blob store (pending events), Redis (the queue between components). Yes, that is several pieces; it is what holds up the durability and the scale.

Prompt management as a first-class citizen

What sets Langfuse apart from platforms focused only on tracing is that the prompts live in Langfuse, not in the application repo or in spreadsheets. Each prompt has:

Name and version (v1, v2, v3…). Changing the prompt does not require redeploying the app: the app asks the SDK for the prompt, and the SDK caches it and refreshes when there is a new version.
Typed variables: {{user_input}}, {{context}}. Rendering with validation.
Tags and labels: by environment (production, staging), by team, by experiment.
Client and server cache: the SDK caches locally with a configurable TTL, avoiding a roundtrip to Langfuse on every call.
Linkage with traces: each trace records which exact version of which prompt was used. Investigating “this answer came out wrong” leads to prompt version Y, not to “some version of the prompt at some point”.

from langfuse import get_client

langfuse = get_client()

prompt = langfuse.get_prompt("rag-system-prompt", version=3)
# or by label: langfuse.get_prompt("rag-system-prompt", label="production")

compiled = prompt.compile(context=docs_text, user_input=question)
# 'compiled' is the final string, ready to send to the LLM

For teams iterating on prompts daily, this is what avoids the chaos of “which version of the prompt is actually running in production right now”.

Evaluations: four combinable evaluation models

Langfuse covers the four patterns of answer evaluation:

LLM-as-a-judge: you configure a model (typically GPT-4 or Claude) with a rubric and it evaluates each answer. Result: a numeric score (0-1) and a justification. Applicable to automatic tracing (all answers) or batch (a dataset selection).
User feedback: the app lets the user mark an answer as good/bad. The feedback is associated with the trace and the prompt version, which lets you see which versions have a worse rate.
Manual labeling: a UI where human labelers score answers. Useful for golden datasets and for evaluating the judge.
Custom evaluators through the API/SDK: your own evals (a unit test, a business metric) report a score through the API. They integrate with CI.

Combined, they give regression testing of the prompt: you change from v3 to v4, evaluate the golden dataset with LLM-as-judge, compare; if v4 gets worse in any of the segments, the merge fails.

Integrations

Langfuse does not compete with OpenLLMetry, LangChain or LiteLLM: it integrates them. The ones that are tested and documented:

OpenTelemetry: any OTel instrumentation emits to Langfuse over OTLP.
LangChain and LangGraph: a native callback that captures the whole chain.
LlamaIndex: native callback.
OpenAI SDK (Python and TS): a wrapper that adds tracing automatically.
LiteLLM: integration as a callback, which covers 100+ providers through LiteLLM.
OpenLLMetry / Traceloop: they emit to Langfuse like any OTel backend.
MLflow: through an OTel exporter from MLflow to Langfuse.
Vercel AI SDK: native instrumentation.

The strategy is clear: Langfuse is a backend, not an SDK. Your team chooses how it instruments; Langfuse accepts any path. The operational consequence: moving from Langfuse to another OTel backend tomorrow is viable.

When Langfuse is not the answer

So as not to present it as a silver bullet:

If you only use LangChain and have no resources to self-host: LangSmith will give you a smoother integration (it is the same team).
If your only need is a proxy with cost tracking and no evals: Helicone is simpler.
If you want an integrated commercial vendor solution: Datadog LLM Observability, New Relic AI Monitoring or Dynatrace AI are Enterprise alternatives with 24/7 support.
If your load is pure batch massive inference with no agents: you probably do not need semantic tracing; Prometheus + Grafana with OTel metrics is enough.

For everything else, your own apps with serious tracing, multi-tenant with quotas, teams iterating on prompts daily, RAG with continuous evaluation, Langfuse is the safe bet.

Quick choice summary:

LangChain → LangSmith (zero effort, automatic instrumentation).
Your own multi-framework applications with OSS → Langfuse (MIT, self-host, complete).
RAG with vector stores → Arize Phoenix (better retrieval visibility).
Simple proxy, low budget → Helicone.
Strict vendor neutrality → OpenLLMetry/Traceloop.
Pydantic AI → Logfire (same team).

Strengths and weaknesses of the instrumented model

Strengths:

Enormous depth: nested spans with all the context (chain steps, retrieval, embeddings, tool calls).
Semantic vocabulary: the SDK knows the domain (LLM, vector store, agent).
Maturity: three years of evolution, a rich ecosystem, ready-made dashboards.
Built-in evals: the top platforms combine tracing with evaluation (judge LLM, datasets, regression).

Weaknesses:

Requires control of the code: if you cannot instrument, it does not work.
Trust in the app: if the app reports badly or has a bug, so does the trace. It is not tamper-proof.
Coupling to the SDK: version changes in a library can break the instrumentation.
Uneven coverage: Python SDKs are mature; Go, Rust and JS are younger.

The zero-instrumentation approach: AgentSight

AgentSight is the project from the eunomia-bpf group that champions the opposite approach. Its paper on arxiv (2508.02736), presented at the Workshop on Practical Adoption Challenges of ML for Systems, formalises the proposal. The premise is direct:

Instead of instrumenting the agent, observe it at the system boundary.

And “system boundary” means the kernel boundary: the last point before a piece of data leaves the process towards the network or the filesystem. There, with eBPF, things are seen as they are, without the application being able to cooperate in hiding them.

Architecture: three planes

AgentSight assembles three layers:

Plane 1 — SSL/TLS uprobes. eBPF can attach programs to functions in userspace libraries (uprobes). The target functions are the encryption ones: SSL_write, SSL_read from OpenSSL/BoringSSL, and their equivalents in Rustls. AgentSight puts hooks on them that capture the arguments: the plaintext buffer the app passes to be encrypted, just before TLS processes it. On reception it does the symmetric thing: a hook after SSL_read with the freshly decrypted plaintext. Result: AgentSight sees the complete content of any HTTPS request the app makes without needing a man-in-the-middle, certificates or traffic decryption. The payload is plaintext because it was captured before being encrypted.

This works because uprobes are cheap (around 100 ns per invocation) and because apps use common TLS libraries. The few apps that implement their own TLS (rare in production) escape this hook; for those you need a different kprobe or manual instrumentation.

Plane 2 — Kernel events. In parallel, AgentSight observes relevant syscalls through tracepoints: execve (which processes start), connect/accept (network), read/write with file descriptors (filesystem and stdio), unlink, clone. Any action by the agent that has an effect outside the process passes through here. This covers, among other things, shell commands executed by the agent: if a Claude Code agent decides to run rm -rf to “clean up the project”, the execve is seen even if the LLM API does not report it.

Plane 3 — Correlation engine. The two previous planes produce streams of asynchronous events. AgentSight has a userspace component that correlates them causally cross-process: an outgoing HTTP request with bash -c rm -rf can be correlated with the previous LLM response that suggested it, through PIDs, timings and heuristics. The paper mentions the optional use of a secondary LLM (Anthropic Claude, for instance) that analyses the sequence of events and produces semantic alerts: “the agent responded with a tool call that was not in the whitelist”, “the reasoning chain has gone 47 iterations without converging”.

`stdiocap`: capturing the stdio of local MCP servers

One specific piece that deserves its own mention is stdiocap, a separate BPF tool included in the repo. The Model Context Protocol (MCP), popularised by Anthropic in 2024 and mainstream in 2025-2026, has two transport modes: HTTP/SSE (network) and stdio (between the client and the server it starts as a subprocess). Local MCP servers, the ones that run on the same machine and are started by the client as children over pipes, communicate through stdin/stdout/stderr with JSON-RPC.

stdiocap hooks read/write/dup on the stdin/stdout/stderr file descriptors of a target process and records all the JSON-RPC traffic between the MCP client and server. It is the same idea as the SSL capture but for stdio: you observe the conversation without either the client or the server knowing. Typical use case: seeing which tools of the filesystem-mcp MCP server a Claude Code agent has invoked in the last hour, what arguments it passed, what errors it received. Impossible with classic instrumentation (MCP servers are usually third-party binaries).

Guarantees: tamper-proof, kernel safety, <3% overhead

Three properties make AgentSight interesting for production:

Tamper-proof: the observation happens in the kernel (uprobes, syscalls). A malicious or compromised application cannot falsify what is seen. Compare with instrumentation: if the agent decides not to emit the span for its action, it does not appear in Langfuse. Here it has no choice.
Kernel safety: eBPF formally verifies that programs terminate and respect bounds checks. It cannot crash the kernel. The same as in the rest of the eBPF series.
<3% CPU overhead measured on real agent workloads (per the paper). The number compares favourably with SDK instrumentation, which typically adds 5-10% in intensive applications.

What it detects out of the box

The paper and the documentation highlight three classes of detection:

Real-time prompt injection: the correlation engine can apply rules or a detection model over the plaintext captured by the SSL uprobes. If the prompt contains suspicious patterns, “ignore all previous instructions”, a system prompt embedded in a user input, instructions to exfiltrate data, it raises an alert.
Reasoning loops that burn resources: agents that enter infinite loops calling tools without progressing. Detectable because the causal chain does not converge to a “final answer” and the tokens pile up. The correlation engine flags them.
Bottlenecks in multi-agent setups: when several agents coordinate, AgentSight sees the communication matrix between all of them and can detect agents that block while waiting, deadlocks, excessive fan-out.

The clash and the coexistence

The two families look like competitors, but in reality they see different things and complement each other in production.

What only the instrumented one sees

Internal agent variables that never reach the wire: the intermediate state of a LangChain chain, the values before passing them to a tool, how a prompt is built from a template with internal vars.
Deep semantic spans: retrieval > embed > vector_search > rerank > format_context > prompt_template > llm. AgentSight sees only the final call to the LLM; the path taken to build it is invisible.
Evaluations: scoring of answers, judge LLMs, quality regression. This lives only in instrumented platforms.

What only eBPF sees

Opaque binaries: Claude Code, Cursor, Gemini CLI, third-party agents. You do not have the code; you cannot instrument them. Only eBPF sees them.
System-level actions: the agent decides to run git push --force or kubectl delete. The action is seen in the execve. The agent’s instrumentation may not report it (especially if it was a command the agent generated as output without going through an explicit “tool”).
Tamper-proof audit: for regulatory compliance (HIPAA, SOC2, NIS2), having observation the app cannot dodge has formal value. eBPF provides it.
Local MCP servers over stdio: invisible to classic instrumentation unless each server emits its own spans (rare).

What both see, complementarily

Prompts and completions: the instrumented approach emits them with rich metadata; eBPF captures them off the wire. A perfect cross-check for spotting discrepancies.
Calls to external APIs: APM flags it; eBPF confirms it at kernel level.
Latency: APM per span; eBPF measures RTT at TCP level and network connectivity.

Decision matrix

Case	Instrumented	eBPF (AgentSight)
Your own app with LangChain	Yes, first	Optional
Your own multi-framework app	Yes	Optional
Third-party binary (Claude Code, Cursor)	Does not work	Yes, the only path
Tamper-proof regulatory compliance	Insufficient	Yes, required
Zero-trust multi-tenant	Insufficient	Yes, required
Local MCP servers (stdio)	Difficult	Yes, with stdiocap
Answer quality evaluation	Yes, required	No (out of scope)
Depth of the internal chain	Yes, required	No (a black box for AgentSight)
Reasoning loop detection	Possible with plumbing	Yes, built in
Real-time prompt injection	Possible (post-processed)	Yes, in stream

The natural conclusion: for your own apps, instrumented; for opaque binaries or compliance, eBPF; for everything that matters, both.

2026 reference architecture

Four recipes that cover the bulk of real cases:

Setup A — Your own application with LangChain or similar

Needs: depth, evals, a team comfortable with SDKs.

Langfuse self-host or LangSmith cloud as the backend.
OpenLLMetry SDK or LangSmith SDK instrumenting the code.
OpenTelemetry Collector between the app and the backend for routing flexibility (to Langfuse + Tempo + Loki, for instance).
Hubble for the network layer in the cluster (inter-pod latency, drop attribution).

Setup B — Productionising an opaque binary (Claude Code, Gemini CLI)

Needs: observe without touching, audit, control cost.

AgentSight deployed as a DaemonSet over the cluster (or standalone on the node).
Grafana with dashboards fed by AgentSight’s metrics.
An OTLP exporter from AgentSight to an OTel backend (Tempo, Jaeger). The spans will use the GenAI semantic conventions once they are fully standardised.
Tetragon optionally for policy over what the agent is allowed to run (Sigkill if it tries rm -rf or similar).

Setup C — Zero-trust multi-tenant platform

Needs: agents from different clients running on the same cluster, mandatory audit, nobody trusts anybody.

AgentSight as the tamper-proof audit layer. Compliance requires it.
Langfuse multi-tenant for the clients that do instrument.
Tetragon with TracingPolicyNamespaced per tenant (different policies per namespace).
Hubble with persistent flow logs for forensics.
Cilium NetworkPolicy to isolate tenants from each other on the network.

Setup D — A local MCP server on a workstation

Needs: seeing what an agent does with a stdio MCP server.

AgentSight stdiocap pointed at the PID of the client or the server.
Complete JSON-RPC capture to a file or to an OTLP endpoint.
Visualisation: Grafana, or simply jq over the log.

A real use case: if you are integrating an MCP server of your own and want to see what tool calls a Claude Code or Cursor agent makes to your server, stdiocap is the cleanest way. You need to modify neither client nor server.

Operational traps

Sensitive data in prompts (instrumented)

By default, Langfuse, LangSmith and similar tools capture the complete content of prompts and completions. If your app processes PII, secrets or medical data, that goes to your observability backend. Configuring redaction or content opt-out before going to production is mandatory. OTel GenAI has specific flags (OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT=false) to avoid it.

Sensitive data in prompts (AgentSight)

The same problem, worse: AgentSight captures literally what goes to the wire, in plaintext. If the agent talked to api.openai.com with a prompt containing sensitive data, AgentSight has that plaintext. You have to encrypt or redact before storing.

Pinned certificates or non-standard TLS

Some high-security apps do certificate pinning or use unconventional TLS implementations (Go’s crypto/tls, custom BoringSSL). In those cases, the uprobes on libssl do not cover them. AgentSight detects when it cannot observe and reports the gap; you still have to add specific hooks for the alternative SDK.

Token volume and storage

An application with medium traffic can generate millions of tokens a day. If you store them all in Langfuse or Phoenix with long retention, the database grows fast. Strategies: aggressive sampling, short retention for normal sessions and long retention only for errors/anomalies, content redaction and keeping only metadata.

Tracing with sampling and consistency

To cut cost, many installations sample: only 1 in every N traces is persisted. Be careful with inconsistent sampling: a trace can carry several spans across multiple services, and if the sampling decision is taken per span, you end up with incomplete traces. OTel has head sampling (in the SDK, at the start), which is consistent, and tail sampling (in the collector, at the end), which allows finer rules. For LLM, tail sampling is ideal: sample everything, discard only the “normal” traces and keep the ones with errors, high latency or high cost.

Multi-agent and trace propagation

When agent A calls agent B, you have to propagate the trace context (W3C Trace Context headers) so it shows up as a single tree. If you do not, you see two disconnected traces. Modern platforms do it automatically with inject/extract, but if your transport between agents is custom (over Redis pub/sub, over a DB), you have to propagate by hand.

The cost of uprobes on critical libraries

Hooking libssl adds around 100 ns per invocation. On extreme TLS traffic loads (tens of thousands of connections/s per core), that adds up. AgentSight keeps it under 3% on typical agent workloads (which are chatty but not networking-intensive). If your use were sniffing all the HTTPS on the node, it could hurt more.

What we have not covered (next series)

Evals: the next layer after tracing. Phoenix, Langfuse, LangSmith and company offer answer evaluation (judge LLM, datasets, regression). It is a world of its own.
Guardrails and safety: NeMo Guardrails, Llama Guard, Llama Prompt Guard, evaluators specific to prompt injection and jailbreaks.
Deep MCP server observability: how the OpenTelemetry GenAI conventions are being extended to MCP servers for trace-aware tools.
eBPF + on-device inference: when the LLM runs locally through vLLM or llama.cpp, the uprobes can see the output token queue BEFORE it goes to the client. New territory.
Statistical analysis of agent flows: detecting drift, outliers, patterns that indicate degradation.

Closing the eBPF series

This series of four articles has travelled eBPF from first principles to the 2026 frontier:

eBPF from zero to Cilium — what eBPF is, networking hooks, how Cilium skips the TCP/IP stack, BGP Control Plane v2.
Tetragon: runtime security — observability and enforcement of processes in the kernel.
Hubble: network observability — L3-L7 flow logs and the frontier with AI agents.
This one — AgentSight, LLM tracing, instrumented vs zero-instrumentation.

If you have got this far you have the map to sit down with a platform, security or AI team in 2026 and recognise what each piece does, what problem it solves and where to start. That whole stack, Cilium for CNI and BGP, Tetragon for runtime security, Hubble for network observability, AgentSight for AI agents, sharing eBPF as a common substrate, Cloud Native governance and OpenTelemetry vocabulary. It is the clean architecture the industry asked for a decade ago and that finally exists.

References

AgentSight:

AgentSight GitHub (eunomia-bpf) — the project.
AgentSight: System-Level Observability for AI Agents Using eBPF (arxiv 2508.02736) — the formal paper.
AgentSight ACM workshop publication.
AgentSight blog post (eunomia.dev) — accessible description.

OpenTelemetry GenAI semantic conventions:

Instrumented platforms:

Langfuse — MIT, self-host + cloud.
LangSmith — LangChain team.
Arize Phoenix — OSS, OTel-native.
Helicone — simple proxy.
OpenLLMetry (Traceloop) — Apache 2.0, OTel SDK.
Pydantic Logfire — AI observability.

2026 comparisons:

Cross-references from the series:

eBPF from zero to Cilium.
Tetragon: runtime security.
Hubble: network observability.
LLM inference series: KV cache, vLLM on Kubernetes, PagedAttention, LLM Operators on K8s.

Llmops on lo0 — Tech Blog

Anatomy of an LLM request in production, May 2026: a tour of the six stages following a single request

TL;DR

You are here: every stage at once

The analogy: forensic analysis of a request

The scenario: a multi-tenant support chatbot for regulated customers

Step 0 — The frame: the answer the user sees

Rewinding backwards: what was already in place before the request

t = −90 days — The previous Retrain stage closes the earlier cycle

t = −60 days — The Data stage: the enriched dataset is versioned and enters circulation

t = −45 days — The Tune stage: the customer_support_v7 adapter is trained

t = −38 days — The Eval stage: the v7 adapter goes through eval gates

t = −31 days — The Deploy stage: the v7 adapter enters serving

Moving forward: the user’s request crosses the system

t = 0 ms — Entry through the gateway

t = 8 ms — Pulling the versioned prompt

t = 12 ms — RAG retrieval

t = 40 ms — Building the final payload

t = 45 ms — Prefill

t = 580 ms — Decode (streaming)

t = 4,200 ms — Complete answer, root span closed

In parallel: Observe is watching

The feedback: the loop closes

Insertion into feedback_signals

Triage by root cause

The next cycle picks it up

What goes in each trace: identity and traceability

Synthesis diagram: how the pieces fit

The on-premise stack applied

Ten bridges between stages where the system breaks

How to walk through the blog

What we have not covered (yet)

See also

References

Real ClickHouse capacity in Langfuse v4: 2,782 bytes per observation, and why almost half of that is not data

TL;DR

You are here: the sizing you cannot do by reading

The analogy: the archive and the index card

The benchmark

What one observation takes up

Where the bytes go

The split by observation type

The indexes, which are almost half

The materialized view as an amplifier

What happens on deletion

Projection

What this benchmark does not measure

Checklist

Traps and things that are not what they look like

Closing

The series: the ten articles

See also

Sources

deepagents on a cluster of your own: the SDK is free, the server is not, and what you have to build in between

TL;DR

You are here: the harness arrives after the model

The analogy: the scaffolding and the crane

What deepagents actually is

What is no longer there

Where state lives

The backends

What grows in the database

The store, which does have cleanup

The isolation that does not exist

The model, and the trap you need to know about

The tools, and the MCP that is not where it seems

The subagents

Code execution

Observability, and the package it goes through

The server: here is where the free part ends

What you have to build

How it fits with activity logging

Checklist

Traps and things that are not what they look like

Closing

See also

Sources

Making the model prefer our tools: the priority that exists in no layer, the two primitives that do, and the `_meta` that crosses the gateway

TL;DR

You are here: the question that arrives after the third server

The `priority` that exists and is no use

The test: the `_meta` crosses the gateway