AI on lo0 — Tech Blog

Anatomy of an LLM request in production, May 2026: a tour of the six stages following a single request

Fri, 22 May 2026 16:00:00 +0200

TL;DR

Across several series the blog has laid out the pieces that hold up an LLM system in production: the Data stage (dataset versioning, ingestion and vector stores, RAG over Kafka), the Tune stage (continuous fine-tuning), the Eval stage (evals as the layer after tracing, guardrails and safety), the Deploy stage (KV cache, PagedAttention, disaggregated serving, multi-tenant GPU cluster, vLLM on Kubernetes, LLM operators on K8s), the Observe stage (tracing with AgentSight, MCP observability, eBPF + drift), the Retrain stage (closing the feedback → dataset → adapter loop), and the cross-cutting components (prompt versioning and data versioning). What is missing is joining them up: seeing a single request cross every piece in order, in one coherent story. That is what this post does. We take a specific request from a multi-tenant support chatbot, rewind it back to the data that trained the adapter serving it today, follow it forward through serving, watch it reach the feedback store when the user marks a thumbs-down, and leave it as the seed of the next quarterly retrain cycle. The route works as a mental map and as an integrator’s guide: the system does not hold up if a single one of the seven pieces (six stages + two cross-cutting) is broken or absent. The practical lesson of the tour is not a new one. It is that everything is connected, that local measurements lie when taken in isolation, and that the real cost of not operating one stage well is paid by another stage further down the line.

You are here: every stage at once

Unlike previous posts, where the mini-map marked a single active box, this one covers the whole pipeline. It is the only post on the blog that activates the six stages and the two cross-cutting components simultaneously, because we follow a real request that crosses all of them.

The analogy: forensic analysis of a request

When an air accident happens, the forensic analysis does not limit itself to the last seconds of the flight. The investigation team rewinds to the maintenance of the previous six months, the manufacturer’s protocols, the pilot’s record, the weather briefing, the controller’s decisions, the incident history on the same model. The conclusion is rarely “the wing broke”; it is “the wing broke because an inspection protocol written a certain way did not detect microcracks that the 2014 calculation model did not consider critical and that were critical beyond a certain fatigue cycle”.

When an LLM request in production fails or succeeds, there is also a long causal chain behind it. The answer the user sees is the last frame; what determined it starts months earlier and branches across six operational stages. If you only look at the last frame, you attribute the result to the model. If you look at the whole chain, you see that the model is one of twelve factors and rarely the most important.

This post does that forensic analysis, but the other way round: instead of starting from a failure and rewinding, we start from a specific request that works and break down what had to happen for it to work, and what will happen to it afterwards. It is a guided tour, not an incident diagnosis. But the mental discipline is the same: no stage is autonomous, and understanding the system means understanding the bridges between stages, not just the boxes.

The scenario: a multi-tenant support chatbot for regulated customers

For the tour we use a concrete realistic scenario, representative enough that the observations carry over to most serious deployments in May 2026. It is an LLM customer support SaaS product, sold to several corporate customers (multi-tenant) in regulated sectors (banking, insurance, health). The product:

Accepts natural-language questions through a chat embedded in the customer’s website.
Retrieves relevant fragments from the customer’s internal knowledge base (product documents, contractual terms, FAQs).
Answers citing the fragment used, in the user’s language, with a tone adapted to the brand.
If the question requires an action (cancel, change plan, open a case), it invokes a tool from the catalogue exposed via MCP, with user confirmation.
It is contractually obliged not to invent customer data, to respect the tenant’s style guide, and to meet ENS / NIS2 compliance on data in the EU.

The infrastructure is on-premise over Kubernetes (RKE2 with the Cilium CNI, in the pattern covered in Cilium BGP and CNI networking). The GPU plane is a generic 4×H100 SXM cluster (80 GB each, NVLink) for serving and production training, plus an RTX 4090 (24 GB) for development, offline evals and ad-hoc re-embeddings.

The request we are going to follow:

User (end customer, Spanish, premium plan with an insurer): “How do I cancel my premium subscription? I have been paying for three months and I have not used half of the cover.”

At first sight, a simple question. Let us see what is behind it.

Step 0 — The frame: the answer the user sees

At 11:47:32 on a Tuesday, the user sends the message. 4.2 seconds later, this appears in the UI:

To cancel your premium subscription, follow these steps: (1) Sign in to the customer portal with your ID number and password. (2) Go to My policies → Premium Life 2024. (3) Press Request cancellation. (4) Select the reason (in your case, “not using the product”) and confirm. You will receive a confirmation email within 24-48 h.

Before you cancel, let me remind you that your plan includes a free annual medical check-up and 24/7 travel assistance, cover that is lost on cancellation. If you like, I can help you review your use of the cover over the past three months to see whether switching to a more tailored plan suits you better than cancelling.

Source: [Cancellation policy, section 4.2 of the 2024 General Terms]

After reading it, the user marks a thumbs-down and writes in the form: “the answer is correct but the tone is too salesy; I only wanted to know how to cancel”.

That sequence, question, answer, thumbs-down with structured feedback, is the last visible frame. Let us go backwards to understand what had to happen for the answer to come out that way.

Rewinding backwards: what was already in place before the request

Before the user typed, the system already had a model loaded in serving, an active prompt labelled production, an up-to-date vector index, a versioned dataset from the last fine-tuning, and a golden eval set that validated the promotion. Each of those artefacts got there through a process. We go through four jumps backwards.

t = −90 days — The previous Retrain stage closes the earlier cycle

Three months ago, during a quarterly Retrain cycle, two things happened. The first: the support team reviewed the feedback accumulated over the previous six months and saw a pattern, the model answering premium users in an excessively formal tone, with users reporting that it “feels robotic”. The second: a one-off incident (a customer cancelling over an answer perceived as curt) triggered an incident-driven mini-cycle.

The process, covered in detail in the Retrain post, followed five sub-processes:

Feedback capture — explicit thumbs-down plus implicit feedback (abandonments, retries) accumulated in a feedback_signals table in Postgres, all of them with a trace_id that allows rewinding to the exact context.
Triage by root cause — the “curt tone” incident cluster was categorised as a prompt issue (it was not the model answering badly, it was the system prompt asking for an overly formal register). One sub-cluster was a model issue (in some cases the model dug its heels in even with a warmer prompt).
Dataset enrichment — the team manually annotated 280 cases where the model was too curt, labelled with the reference answer (“how it should have answered”). Double annotation on the critical 20%; cases with a quality score below 4 were left out.
Cadence decision — the incident was treated as incident-driven; the rest of the quarterly Retrain followed the calendar.
Promotion — the new customer_support_v7 adapter went through eval gates against customer_support_v6, a 5% canary for a week, and was promoted when the golden set metrics showed a stable improvement in the “tone / clarity” segment with no regressions elsewhere.

The result: the adapter active in production when the user sent the Step 0 request is customer_support_v7, trained on the enriched dataset enriched_retrain_2026_q1 version 3, with double lineage back to the original incident.

t = −60 days — The Data stage: the enriched dataset is versioned and enters circulation

Immediately after Retrain, the Data stage of the LLMOps pipeline does its work. Three critical operations, covered in detail in the data versioning post:

Immutable versioning of the enriched dataset with DVC, sha256 hash propagated to the registry. The identifier (enriched_retrain_2026_q1, v3, sha256:9af...) becomes the luggage tag that will travel through the coming stages.
Schema contract validated by CI: every row meets the JSON Schema of the entry the trainer expects (example_id, input.user_query, input.retrieved_context, expected_output, rubric, segment, difficulty). A validation fails in CI if any row breaks the contract.
Holdout segregation check: the normalised sha256 hash of each input is compared against every hash in the active golden eval set (customer_support_golden_v12). Zero overlaps = the dataset does not contaminate the eval. Had there been a single one, CI would have blocked the merge.

In parallel, the RAG corpus (product manuals, FAQs, general terms of the insurer tenant) is kept alive. The ingestion pipeline keeps capturing changes from the customer’s CMS: a new section of the cancellation policy was modified in February and reindexed in Qdrant. As the post about RAG over Kafka explains, the corpus is not retrained with every change: only the delta is re-embedded, and lakeFS keeps a branch of the embeddings bucket with the new version. The branch is merged into main when recall@10 over a representative set of queries stays above the threshold (0.78 in this system).

t = −45 days — The Tune stage: the customer_support_v7 adapter is trained

Three weeks after closing the dataset, training of the new LoRA adapter starts. As the continuous fine-tuning post details, the production pattern in 2026 avoids retraining the base model, which is expensive, slow and irreversible, and favours a LoRA adapter over a stable base model (in this system, Llama 3 70B-instruct quantised to INT8 for serving). The training:

Runs over 4 of the H100s (NVLink, tensor parallel) for ~6 hours.
Uses transformers + PEFT + bitsandbytes, monitored by MLflow.
Every step records the dataset_id, dataset_version and dataset_hash as an input artifact in MLflow.
The output, a customer_support_v7.safetensors file of ~280 MB with the LoRA weights, is uploaded to MinIO with its own hash, and MLflow records model_id, model_version, parent_dataset.

At this point the lineage chain is closed for this leg:

enriched_retrain_2026_q1, v3, sha256:9af...
│
▼
mlflow run train, run_id: 0xa721...
│
▼
customer_support_v7, sha256:5c1...

t = −38 days — The Eval stage: the v7 adapter goes through eval gates

The freshly trained adapter is not promoted. It goes through an eval suite covered in detail in the post about evals. The golden eval set, customer_support_golden_v12, 850 human-curated examples with an inter-annotator kappa of 0.81, is run against two models: the candidate v7 adapter and the v6 currently in production. The metrics:

Metric	v6 (prod)	v7 (cand.)	Threshold
Faithfulness to the RAG fragment	0.87	0.89	≥ 0.82
Toxicity (low is good)	0.012	0.011	≤ 0.02
“Warm but professional” tone (judge LLM)	0.71	0.84	≥ 0.78
Format compliance (structured markdown)	0.94	0.93	≥ 0.90
Helpful-but-not-pushy (judge LLM)	0.66	0.79	≥ 0.75
Latency p95 (ms)	2,840	2,910	≤ 3,500

To this is added the guardrails and safety suite covered in the guardrails post: jailbreak resistance, PII leakage detection, prompt injection over MCP tools. The v7 improves on safety in two metrics and ties on the rest.

The v7 enters a 5% traffic canary for 7 days, under close monitoring. At the end of the canary, the online metrics confirm what the offline ones anticipated: an improvement in tone and helpfulness, equivalent latency, no new failure modes. Promotion approved. The v7 moves to the production label.

t = −31 days — The Deploy stage: the v7 adapter enters serving

The customer_support_v7 adapter is promoted to the serving cluster. Three pieces covered in separate posts come into play.

vLLM as the inference engine. The engine lives on Kubernetes, deployed via a dedicated Operator, as the post about LLM operators and the post about vLLM on K8s explain. The operator is responsible for detecting the new adapter in the registry, hot-loading it without restarting the engine (a native vLLM capability with --enable-lora), and directing traffic based on the label.

Disaggregated serving. As the post about disaggregated serving details, the system separates prefill (compute-intensive, throughput-bound) and decode (memory-intensive, latency-bound) into different GPU pools. The user’s request, when it arrives, prefills in a specialised pod and decodes in another, communicating over NVLink plus a shared KV cache fabric.

Multi-tenant GPU cluster. The H100 cluster serves several tenants, not just the insurer from Step 0. As the post about the multi-tenant cluster explains, isolation is realised on four planes: the Kubernetes namespace, ACLs over adapters (only the tenant’s namespace loads its LoRAs), partitioning of the KV cache per tenant (one tenant cannot read another’s cached prefixes), and a tokens-per-minute quota enforced at the gateway.

Synchronised prompt registry. The product’s system_prompt lives in Langfuse with the production label. The active version is customer_support_system_prompt, version 12. The gateway reads the prompt from Langfuse on the request path (with a few seconds of cache so it does not hammer the registry). Detailed in the prompt versioning post.

The result at t = −31 days: the combination (adapter v7, prompt v12, golden v12) is active and served. The system is ready for the request that will arrive 31 days later.

Moving forward: the user’s request crosses the system

Back to Step 0: 11:47:32 on a Tuesday. The user presses Enter. We go in real time, in milliseconds.

t = 0 ms — Entry through the gateway

The user’s browser POSTs to chat.aseguradora-ejemplo.com/api/chat. The traffic crosses the edge load balancer and enters the SaaS product’s API gateway. The gateway:

Authenticates the user’s JWT (an end customer of the insurer tenant).
Extracts the tenant_id and checks that its tokens-per-minute quota is not exhausted.
Resolves which model_id, adapter_id and prompt_id correspond to this tenant and product. In this case: llama-3-70b-int8 + customer_support_v7 + prompt label production.
Builds a unique trace_id (W3C TraceContext, propagable to OTel) and starts a root span.

At 8 ms, the gateway passes the request to the prefill pool.

t = 8 ms — Pulling the versioned prompt

Before serving, the OpenAI-compatible client the engine uses internally pulls the active system prompt. As the post about prompt versioning details, the pattern is:

prompt = prompt_registry.pull(
 name="customer_support_system_prompt",
 label="production", # now pointing at v12
)
# A local 30 s cache reduces the round-trip to 0.1 % of requests

The OTel span of the prompt pull carries the attributes gen_ai.prompt.id = customer_support_system_prompt, gen_ai.prompt.version = 12, gen_ai.prompt.label = production. They are propagated to all children.

t = 12 ms — RAG retrieval

The system needs context from the tenant’s knowledge base. It runs:

query_embedding = encoder.encode(user_query)
chunks = qdrant.search(
 collection=f"tenant_{tenant_id}_kb_v3",
 vector=query_embedding,
 limit=4,
 score_threshold=0.72,
)
reranked = reranker.rerank(user_query, chunks, top_k=2)

At 38 ms, the reranker returns two fragments: one from the Cancellation policy, section 4.2 and another from Premium plan benefits, section 2.1. As the post about PostgreSQL + Qdrant details, the tenant’s corpus is kept isolated by collection and ACL: no tenant can read another’s chunks.

t = 40 ms — Building the final payload

The engine composes:

[system_prompt v12]
+ [retrieved context: 2 chunks]
+ [short session history: 1 previous turn]
+ [user query]

Total: ~1,850 tokens of context. The OTel span records gen_ai.request.input_tokens = 1850, gen_ai.request.model = llama-3-70b-int8, gen_ai.request.adapter = customer_support_v7.

t = 45 ms — Prefill

The payload enters the prefill pool. The GPU processes the 1,850 tokens in a single parallel pass, computing for each token its K and V vectors (attention key and value). Those vectors are materialised as the KV cache, covered in detail in the KV cache fundamentals post. The resulting cache takes up ~120 MB of VRAM in INT8.

Here a key optimisation appears: the v12 system prompt is cached in the prefill pool (prefix caching, covered in the post about PagedAttention). Since the system prompt is the same for this tenant, the first ~500 tokens of the context are not recomputed: they are read from the prefix cache. That reduces the effective prefill from 1,850 tokens to ~1,350 tokens, saving ~270 ms of compute.

At 580 ms (effective prefill), the TTFT (time to first token) is ready. The first token leaves for the decode pool.

t = 580 ms — Decode (streaming)

The decode pool receives the prefilled KV cache and starts generation token by token. As the post about disaggregated serving details, the prefill/decode separation is what lets a multi-tenant system keep TPS stable: the decode pool is sized to sustain thousands of sessions decoding in parallel at a low cost per token, while the prefill pool is sized for bursts of short TTFT.

Generation at ~80 tokens per second. The answer will be ~290 tokens. Total decode time: ~3.6 s. Streaming: the user starts seeing words from t = 580 ms.

While decode advances, the engine emits child spans on every iteration with gen_ai.response.tokens_generated, gen_ai.response.cache_hit_ratio, gen_ai.response.cumulative_latency. The post about AgentSight and the post about MCP observability with OTel cover the detailed instrumentation of this layer.

t = 4,200 ms — Complete answer, root span closed

Generation finishes. The engine closes the root span with gen_ai.response.completion_tokens = 290, gen_ai.response.finish_reason = stop, gen_ai.response.total_latency_ms = 4200. The user sees the final answer. The session is ready for a next turn or for the user to click thumbs-up/thumbs-down.

By this point, every active stage has taken part:

Data (pre-existing): the indexed RAG corpus, the dataset that trained the adapter, the golden set that validated it.
Tune (pre-existing): the v7 adapter trained 45 days ago.
Eval (pre-existing): the gates that approved the promotion.
Deploy (at this very moment): vLLM + disaggregated + KV cache + multi-tenant.
Observe (at this very moment): the OTel spans emitted to Langfuse + Tempo, the metrics to Prometheus.
Retrain (about to activate): the feedback the user will leave in 15 seconds.

In parallel: Observe is watching

While the request happens, several pieces of Observe run in parallel and leave a structured trail.

OTel tracing. Every span (gateway, prompt pull, retrieval, prefill, decode) travels to Langfuse and to an OTel collector that forwards them to a backend (Tempo / Jaeger). The unique trace_id links all the spans. As the post about tracing with AgentSight details, end-to-end propagation is the main enabler of post-incident debugging: without it, you cannot reconstruct what happened three weeks later.

Runtime metrics. The engine emits Prometheus metrics per interval: gpu_utilization, kv_cache_usage, tokens_per_second, queue_depth, prefill_latency_p95, decode_latency_p95. The metrics are not tied to a trace; they are aggregated by tenant and service.

Online LLM-as-judge. A configurable percentage of answers (2% in this system) is also run through a judge LLM in the background, which scores the answer against a simple rubric (correct / partial / incorrect + a tone score). The judge does not block the answer to the user; it feeds the dashboard.

Statistical drift. In parallel, a slower pipeline computes drift over the distribution of inputs and outputs. As the post about eBPF + drift explains, low-level monitoring (latency, error rate per endpoint) is complemented with statistical drift detection (KS test, embedding distance) that spots when “something is wrong” before a thumbs-down confirms it.

Safety and guardrails monitor. The guardrails post describes the layer that watches for jailbreak attempts, PII leakage and prompt injection via MCP tools. In this case, none of them fires.

All these pieces operate continuously, not per request. But this particular request left its trail in every one of them.

The feedback: the loop closes

Fifteen seconds after reading the answer, the user marks a thumbs-down and writes in the form: “the answer is correct but the tone is too salesy; I only wanted to know how to cancel”. That apparently trivial gesture triggers an important sequence.

Insertion into feedback_signals

As the post about Retrain details, the thumbs-down is persisted as a structured row in a Postgres table:

INSERT INTO feedback_signals (
 signal_id, trace_id, request_id, signal_type, signal_value,
 prompt_id, prompt_version, model, user_segment, occurred_at
) VALUES (
 gen_random_uuid(),
 '4f5...', -- the trace_id from Step 0
 'r-22a...', -- request_id
 'thumbs',
 '{"vote":"down","reason":"too pushy","text":"I only wanted to know how to cancel"}',
 'customer_support_system_prompt',
 12,
 'llama-3-70b-int8+customer_support_v7',
 'premium-es',
 '2026-05-19T11:47:51+02:00'
);

With this, the row is linked by trace_id to everything that happened: prompt v12, retrieved context, complete output, latency metrics, judge score (0.82 in this case, considered good by the judge, though the human disagrees).

Triage by root cause

The MLE team runs triage the next morning. Combining heuristic rules, LLM-as-classifier and human review:

The signal is not a model issue: the model answered correctly to the prompt it received.
It is not a retrieval issue: the retrieved chunks were the right ones.
It is not an infra issue: latency was normal.
It is a prompt issue: the v12 system prompt instructs the model to “offer alternatives before processing destructive actions”. That instruction generates the “salesy tone” in some contexts.

The incident accumulates with others from the month in the “salesy tone” cluster. When the cluster crosses a threshold (typically 30-50 incidents of the same type or a percentage of the total), it will enter an incident-driven mini-cycle or wait for the quarterly Retrain, depending on the size.

The next cycle picks it up

Three months later, in the next quarterly Retrain, this feedback is one of many that will motivate two changes:

A new prompt version v13 with an adjusted instruction: “offer alternatives only if the user does not express a clear intention to cancel”.
A possible reinforcement of the adapter with cases of a more direct tone for premium-es. If the cluster justifies it.

The v13 will go into its own eval gate. The golden set will grow with cases where the correct tone is “direct, not salesy”. The v8 of the adapter (if it arrives) will retrain over the enriched dataset enriched_retrain_2026_q2 that already contains this annotated case.

The loop closes. The Step 0 request has contributed to the version of the system that will serve another user three months later.

What goes in each trace: identity and traceability

If the reader looks at the seven identifiers that are everywhere in this route, they see the network of identities that makes all of the above possible. It is the identity infrastructure of an LLM system in production:

trace_id 4f5... (unique per request)
request_id r-22a... (idem)
prompt_id customer_support_system_prompt
prompt_version 12
prompt_label production
dataset_id enriched_retrain_2026_q1
dataset_version v3 (sha256:9af...)
model_id llama-3-70b-int8
adapter_id customer_support_v7 (sha256:5c1...)
deployment_id d-prod-7b
schema_version 3.2
tenant_id aseguradora-ejemplo
user_segment premium-es
golden_set_id customer_support_golden_v12

If a single piece of that set is missing or does not propagate, the chain breaks. The next incident investigated will land in “we cannot rewind to the origin because the system did not record it”. That is why the cross-cutting components, prompt versioning and data versioning, are not luxuries: they are the connection without which the other six stages operate blind.

Synthesis diagram: how the pieces fit

 ┌─────────────────────────────────────────┐
│ User (end customer, B2C) │
└─────────────────┬───────────────────────┘
│ chat msg + JWT
▼
┌─────────────────────────────────────────┐
│ Edge LB + WAF + Cilium CNI │
└─────────────────┬───────────────────────┘
│ HTTPS, internal mTLS
▼
┌─────────────────────────────────────────────────┐
│ API Gateway (auth, quota, model routing) │
│ - Resolves tenant → model + adapter + prompt │
│ - Starts trace_id (W3C) │
└──────┬─────────────────────┬────────────────────┘
│ │
(pull prompt) │ │ (pull config)
▼ ▼
┌────────────────────┐ ┌──────────────────────┐
│ Langfuse Prompt │ │ Model registry │
│ Registry (v12) │ │ (adapter v7) │
└─────────┬──────────┘ └──────────┬───────────┘
│ │
└──────────┬───────────────┘
│ payload ready
▼
┌──────────────────────────────────────────┐
│ vLLM engine (K8s Operator) │
│ ┌──────────────┐ ┌──────────────┐ │
│ │ Pool prefill │ → │ Pool decode │ │
│ │ (H100×N) │ │ (H100×M) │ │
│ └──────┬───────┘ └──────┬───────┘ │
│ │ KV cache fabric │ │
│ └──────────────────┘ │
│ - prefix caching of the system prompt │
│ - PagedAttention │
└──────┬───────────────────────────────────┘
│ tokens stream
▼
┌─────────────────────────────────────────┐
│ User sees the answer + thumbs/UX UI │
└─────────────────┬───────────────────────┘
│ feedback (15 s later)
▼
┌─────────────────────────────────────────┐
│ feedback_signals (Postgres) │
│ + Langfuse scores │
└─────────────────┬───────────────────────┘
│
┌────────────────────────┼────────────────────────┐
│ │ │
▼ ▼ ▼
triage quarterly Retrain cycle dataset_id
root cause or incident-driven enriched (DVC)
│
▼
Tune of v8
(next cycle)
In parallel throughout the request, OTel instrumentation:
spans → Tempo / Jaeger ; events → Langfuse ; metrics → Prometheus

The on-premise stack applied

Taking the above to a generic on-premise consultant-profile infrastructure (RTX 4090 + 4×H100 SXM cluster):

Layer	Typical resources
Network plane	Edge LB (HAProxy / nginx ingress) + Cilium CNI with BGP, covered in Cilium BGP
K8s compute plane	RKE2 with two manager nodes + a GPU node pool
Production GPU plane	4× H100 SXM (NVLink, 80 GB each), partitioned via MIG into prefill/decode pools
Development GPU plane	1× RTX 4090 (24 GB) for offline evals, drift-check embeddings, smoke tests
Storage plane	MinIO or Ceph object store; DVC remote + lakeFS backend
OLTP data plane	Postgres 18 with replication; pgvector 0.8 for small cases
Vector plane	Qdrant or Milvus for large RAG corpora
Stream plane	Kafka (Redpanda / pure Apache) + Schema Registry; CDC with Debezium or Flink CDC
Observability plane	OTel Collector + Tempo (traces) + Prometheus (metrics) + Loki (logs); Langfuse for the LLM-specific part
Runtime security plane	Tetragon, covered in the runtime security post

The real density is not the sum of the boxes: it is the operations that tie the boxes together. A cluster with every piece but no versioning discipline, no end-to-end trace_id propagation, no schema contracts and no cadenced retraining is a cluster that serves LLM once and then ages. The difference between a project and a platform is exactly that.

Ten bridges between stages where the system breaks

The route reveals something important: failures are rarely inside a stage; they are on the bridges between stages. Ten common bridges:

Data → Tune: the dataset does not propagate its (dataset_id, dataset_version) to the trainer. The same dataset trained twice produces two model_id values that cannot be told apart.
Tune → Eval: the trained model does not propagate its lineage to the eval run. The eval passes, but there is no record of which dataset it was trained on. Three months later, irreproducible.
Eval → Deploy: promotion happens without the serving system recording which version of the adapter it is serving at each moment. The day the model gives a dangerous answer, nobody knows which adapter answered.
Deploy → Observe: the engine does not emit gen_ai.request.adapter, gen_ai.prompt.version, gen_ai.dataset.version as span attributes. The traces exist but cannot be crossed with the lineage.
Observe → Retrain: the feedback is captured in a tool (Langfuse, Phoenix) but nobody reads it. The Retrain stage “exists”, but feedback piles up untriaged.
Retrain → Data: the enriched dataset goes into the next Tune without passing through versioning discipline, schema contract and holdout check. Silent contamination of the golden set.
Prompt versioning ↔ everything: the prompt_id, prompt_version does not propagate to the spans. The day the team discovers that a prompt change regressed the system, it cannot isolate which one or when.
Data versioning ↔ everything: the dataset_id, dataset_version does not show up in experiment tracking. “v8 is retrained” but nobody can prove it was over the enriched dataset and not the old one.
MCP ↔ tools: the system invokes tools (cancellation, policy changes) but does not record a gen_ai.tool.invocation_id linked to the trace. The actions end up dissociated from the answer that generated them.
Schema Registry ↔ data: datasets version content but not schema. A breaking change in expected_output breaks the eval silently; nobody notices anything until a human reviews the results.

The bridges are covered throughout the blog. Operations enforce them. The team’s culture keeps them.

How to walk through the blog

If you arrive at this post from outside and want a reading route:

The map: The six-stage LLMOps pipeline — the master map of everything else.
The context: MLOps specific to LLMs in 2026 — the landscape and why LLMOps is not classic MLOps.
Inference from the inside out: KV cache → PagedAttention deep dive → Disaggregated serving → Multi-tenant GPU cluster → vLLM on K8s → LLM operators on K8s.
Data: Data versioning with DVC and lakeFS → PostgreSQL + Qdrant ingestion → RAG over Kafka.
Tune: Continuous fine-tuning in production.
Eval: Evals: the layer after tracing → Guardrails and safety.
Observe: AgentSight LLM tracing → MCP observability with OTel → On-device eBPF + drift.
Retrain: Closing the feedback → dataset → adapter loop.
Cross-cutting: Prompt versioning with Langfuse and MLflow.
Supporting infrastructure (the base everything is built on): RKE2 with Cilium BGP, Hubble + eBPF observability, Tetragon runtime security.

What we have not covered (yet)

At the first level the main things are there. The following posts on the blog, when the topics justify it, could go deeper into:

Schema Registry for LLM data and prompts: the other half of the data contract.
A dedicated AI Gateway: LiteLLM, Portkey, Kong AI Gateway as a control plane.
OTel gen_ai semantic conventions: the emerging standard that ties the seven identifiers of the “identity” block into well-formed spans.
Federated learning over regulated customer data: how to train without centralising the corpus.
Capacity planning for shared multi-tenant clusters.
Disaster recovery of an LLM service: how to reproduce the state of the system 30 days back.
Cost accounting per tenant: tokens × weights × adapter × infrastructure → invoice.

References

W3C Trace Context — end-to-end propagation of traceparent and tracestate.
OpenTelemetry GenAI Semantic Conventions — gen_ai.* attributes for LLM spans.
Langfuse documentation — observability and prompt registry.
vLLM documentation — production inference engine with PagedAttention and LoRA hot-swap.
Kubernetes Operators — the declarative management pattern.
MLflow Tracking and Model Registry — lineage of runs and input artifacts.
DVC and lakeFS — dataset versioning, unified in Nov 2025.
OpenLineage — an open standard for lineage events between systems.
ENS / NIS2: compliance frameworks that apply to operators in the EU; recommended reading for the context in which the scenario operates.

Langfuse v4, day 2 (2 of 9): putting LangGraph in front, and what one agent turn costs in observations

Sun, 13 Sep 2026 07:00:00 +0200

Second article in a series about operating Langfuse v4 in production. The first one, what really goes into a trace, walked through the data model by reading the server code. This one measures the other side. Verified against Langfuse 4.15.2 (Python SDK), LangGraph 1.2.11 and langchain-core 1.6.3, with figures taken on 13 September 2026 on a bench with no network.

TL;DR

One agent turn is four observations plus five per tool call. Measured with an in-memory exporter over a minimal ReAct graph: zero tools, 4 observations; one, 9; five, 29; ten, 54; forty, 204. The formula holds exactly across the whole range.

The byte volume grows quadratically, not linearly. The same turn goes from 12.2 KB with one tool call to 1.87 MB with forty. 88 % of those bytes is the input field, because every observation re-serialises the full message history as it stands at that moment.

Tool definitions travel attached to every model call. The handler appends them to the input field as messages with role tool. Measured: 364 bytes per tool. With twenty tools that is 7.3 KB on every single step of the loop.

The AGENT observation type is assigned by string match on the name. A node called planificador comes out as CHAIN; the same node called agente_planificador comes out as AGENT. The logic is literally looking for agent in the class path or in the node name (langfuse/langchain/CallbackHandler.py:397).

An interrupt with a resume produces two separate traces. Verified with invoke, with stream and with subgraphs, and also reusing the same handler object. The SDK ships machinery to reattach the trace, but it only fires if the root run finishes with a control-flow exception, and in LangGraph 1.2.11 the interrupt comes back through the return value.

The fix fits in one line and is measured. Passing trace_context={"trace_id": Langfuse.create_trace_id(seed=thread_id)} when building the handler leaves the interrupt and the resume in the same trace, even with different handlers and therefore with different replicas.

The span processor drops by default whatever it does not recognise. It only exports spans from the SDK’s own tracer, spans carrying some gen_ai.* attribute, or spans from a closed list of thirty-five instrumentation prefixes (langfuse/_client/span_filter.py:11). Your own code’s spans do not arrive unless they come through one of those three doors.

You are here: OBSERVE, day 2

The series was planned with eight articles and grows to nine. The reason is ordering: the next two are about migrating and about sizing ClickHouse, and both need an input figure that nobody publishes, which is how many observations and how many bytes a real unit of work produces. A chat produces a trace with two observations. An agent produces two hundred. Sizing with the chat figure leads to a cluster that falls over the day someone connects the agent.

So before migrating, measure. And to measure there has to be in front what people actually deploy.

The analogy: the file that gets photocopied whole at every signature

A procurement file passes across six desks. At each desk somebody reads what is there, adds a sheet and passes it on.

There are two ways to leave a record of that route. The first is to write in a register which desk touched it, when, and which sheet it added. Six short entries. The second is to photocopy the whole file on the way into each desk and on the way out, and to archive the twelve copies. The register takes one sheet. The copies take, at the sixth desk, six times what they took at the first, and in total they take something on the order of the square of the number of desks.

LangChain’s callback instrumentation does the second thing. It does not record the delta of each step, it records the complete state on the way into and out of each step. With a chat that makes no difference, because the state is short and there is one step. With a forty-step agent, the archive weighs almost two megabytes per file processed.

This is not a defect that needs fixing. It is what lets you open a trace and see exactly what the model saw at step 17 without reconstructing anything. But it drives the sizing, and the number has to be known before signing the contract.

Which agentic platform is the most used

The question has two answers depending on the metric, and both are defensible.

By PyPI downloads over the last thirty days, checked on 13 September 2026:

Package	Downloads / 30 days
`langgraph`	52.3 M
`strands-agents`	35.2 M
`openai-agents`	22.9 M
`crewai`	19.1 M
`google-adk`	14.5 M
`pydantic-ai`	8.0 M
`llama-index`	5.0 M
`agno`	1.9 M
`autogen-agentchat`	0.6 M

By GitHub stars the order flips at the top: CrewAI is at 58.4 k, LangGraph at 38.1 k and the OpenAI Agents SDK at 28.6 k.

PyPI downloads are inflated by continuous integration and by images that reinstall on every build, so they indicate presence in pipelines rather than agents in production. Stars measure interest, not deployment. With both together, LangGraph is the one that shows up in more pipelines and the one with the most worked-through instrumentation path in Langfuse, so it is the one instrumented here. What follows about volume applies equally to any platform instrumented through LangChain callbacks, because the cost comes from the callback model, not from the graph.

How LangGraph gets into Langfuse

The route is worth stating, because the previous article left two open and only one is used here.

langchain-core 1.6.3 does not import OpenTelemetry anywhere. The check is direct against the installed wheel and returns not one file. LangGraph does not emit OTLP spans on its own either. What exists is LangChain’s callback system, and the Langfuse CallbackHandler hooks into it.

That handler does not talk to the classic ingestion API. It creates observations with the v4 SDK, which underneath is OpenTelemetry: LangfuseSpanProcessor sets up an OTLPSpanExporter against {base_url}/api/public/otel/v1/traces (langfuse/_client/span_processor.py:123). So it is OTLP, but generated by the SDK, not by third-party auto-instrumentation.

The difference matters for one concrete reason left hanging in the previous article. Because the data passes through the SDK calls, the classic masking function does apply: input, output and metadata all go through _process_media_and_apply_mask before becoming attributes (langfuse/_client/span.py:534). With auto-instrumentation of the OpenInference or OpenLLMetry kind, the messages travel in span events and masking does not touch them. For an agent handling customer data inside its tools, that difference decides the route.

The trade-off is that a callback handler only sees what goes through LangChain. HTTP calls a tool makes on its own, the database query, the subprocess, none of that shows up. And if you instrument your code with OpenTelemetry so that it does show up, you run into the filter described further down.

The observation type is decided by the node name

The first article insisted that v4 has ten observation types and that using the right one changes what the interface knows how to group. The LangChain handler emits six of those ten: tool, retriever, generation, agent, chain and span. It never emits event, embedding, evaluator or guardrail.

The assignment is direct for four of them: tool callback, type tool; retriever callback, type retriever; model callback, type generation. For chain callbacks, which is what every node of a LangGraph graph is, the criterion is this (langfuse/langchain/CallbackHandler.py:393):

elif callback_type == "chain":
 # Detect if it's an agent by examining class path or name
 if serialized and "id" in serialized:
 class_path = serialized["id"]
 if any("agent" in part.lower() for part in class_path):
 return "agent"

 name = self.get_langchain_run_name(serialized, **kwargs)
 if "agent" in name.lower():
 return "agent"

 return "chain"

Measured over the same graph with the node renamed and everything else identical:

Node name	Tool calls	Observations	Types
`planificador`	10	54	11 generation, 33 chain, 10 tool
`agente_planificador`	10	54	11 generation, 22 chain, 11 agent, 10 tool

Eleven observations change type because of eleven characters in a function name. This has two practical consequences. The first is that any dashboard counting AGENT observations is counting naming conventions. The second is that the interface draws the agent graph when the trace contains any observation of a type other than span, event or generation, and chain qualifies, so the drawing comes out either way. What changes is what you can filter and aggregate afterwards.

The operational recommendation is boring and it works: name the nodes that represent a model decision with a name containing agent, and leave the rest as chains. It is a naming convention promoted to data schema, which is not pretty, but it is what there is and it is stable.

What a turn costs, measured

The setup is a minimal ReAct graph: one node that calls the model, a ToolNode with one tool, a conditional edge with tools_condition and an in-memory checkpointer. The model is langchain-core’s GenericFakeChatModel, which returns a fixed list of messages, so neither network nor GPU is needed and the experiment reproduces on any laptop. The exporter is OpenTelemetry’s InMemorySpanExporter, passed to the client through the span_exporter parameter, so observations are counted with no Langfuse server in front.

Result, with the node named planificador:

Tool calls	Observations	generation	chain	tool	Attribute bytes
0	4	1	3	0	4.2 KB
1	9	2	6	1	12.2 KB
5	29	6	18	5	64.1 KB
10	54	11	33	10	173.5 KB
20	104	21	63	20	541.4 KB
30	154	31	93	30	1,108 KB
40	204	41	123	40	1,874 KB

The observation count is exactly 4 + 5n. The five per loop iteration are: the model node, the model call, the conditional edge, the tools node and the tool. The four fixed ones are the root graph, the first partial pass and the close.

The bytes do not follow that line. From 10 to 20 steps the volume multiplies by 3.1; from 20 to 40, by 3.5. That is the quadratic growth expected when each of the n steps drags along a history whose length is proportional to n.

The breakdown by field confirms it. With forty steps, out of 1,709 KB measured in that run, 1,502 KB sit in langfuse.observation.input and only 56 KB in the output. 88 % of what is ingested is the same history repeated at different stages of progress. The largest single observation in that turn is 22.2 KB, well below the 9.5 MB per OTLP span ceiling, so the problem is not the size limit: it is the aggregate.

And inside the input there is a second multiplier. The handler appends each tool’s definition to the generation’s input field, as messages with role tool (langfuse/langchain/CallbackHandler.py:1190). Measured with three-parameter tools and a one-line description:

Bound tools	Input field bytes of one generation
0	34
1	398
5	1,854
10	3,674
20	7,334

That is 364 bytes per tool, on every generation, always. An agent with twenty tools and forty steps ingests 7.3 KB × 41 generations = 300 KB in tool schemas alone, which never change over the course of the turn.

What this means for ClickHouse

A honest caveat belongs here. I compressed the full payload of those runs with ZSTD level 3, which is what events_full uses, and got ratios between 29× and 86×, improving the longer the turn. That number is inflated by how synthetic the bench is: my tool always returns the same string, so the redundancy is higher than in production.

What does hold is the shape. Repeating the history is exactly the pattern columnar compression absorbs well, so the ClickHouse disk is not where it hurts. Where it hurts is earlier: in the bandwidth towards the OTLP endpoint, in the worker CPU that decompresses and flattens, and in the ingestion queue, all of which see uncompressed bytes. The first article noted that the queue that saturates first is langfuse.queue.ingestion.depth, and this is why it saturates.

Quick sizing arithmetic, with the measured figures and an average ten-step turn:

1,000 turns a day is 54,000 observations and around 170 MB uncompressed a day.
10,000 turns a day is 540,000 observations and 1.7 GB uncompressed a day.
If the average turn goes from ten steps to twenty, observations double and bytes triple.

Sensitivity to the number of steps is the parameter to watch, more than the number of users. A prompt change that makes the agent take three more turns around the loop multiplies the observability bill with nobody touching the infrastructure. Article 5 of the series comes back to this with ClickHouse system tables to measure the real bytes per observation in your own installation, which is the only thing that replaces this estimate.

Interrupts: two traces where there should be one

A production agent with human approval uses interrupt(), hands control back, waits for a person and resumes with Command(resume=...). The operational question is whether the trace survives that pause.

The SDK ships explicit machinery for the case. There is a store of pending trace contexts keyed by thread_id, capped at 1,024 entries with eviction of the oldest (MAX_PENDING_RESUME_TRACE_CONTEXTS), and a function that recognises a Command carrying resume and recovers the stored context.

Measured, it does not fire in this combination of versions:

Case	Same trace?
Same handler object, `invoke`	No
Same handler object, `stream`	No
Interrupt inside a subgraph	No
Fresh handler for the resume	No

The reason is in the code and it is consistent. The pending context is only stored from on_chain_error when the failing run is the root one and the resulting level is DEFAULT, which is what happens with LangGraph’s control-flow exceptions (CallbackHandler.py:866). In LangGraph 1.2.11 the interrupt does not leave the root as an exception: the graph returns normally, with the interrupt in the return value. The root run ends through on_chain_end and the branch that stores the context never runs. The interrupted node’s observation does come out right: level DEFAULT and the Interrupt in the status message, not marked as an error, which is the correct behaviour.

The fix is verified and does not depend on the SDK changing:

from langfuse import Langfuse
from langfuse.langchain import CallbackHandler

trace_id = Langfuse.create_trace_id(seed=thread_id)
handler = CallbackHandler(trace_context={"trace_id": trace_id})

With that, the interrupt and the resume land in the same trace even if the handler is a different object and even if another replica serves the resume, because the identifier derives from the thread_id and not from in-memory state. It is the same technique the documentation recommends for joining several agents into one trace, applied to joining an agent with itself.

With one nuance to decide deliberately. Seeding with the bare thread_id puts every turn of that conversation into a single trace, which grows without bound while the conversation stays alive. If what you want is one trace per turn with the resume attached to the right turn, seed with thread_id plus the turn number, and leave conversation grouping to the session identifier, which is what it is for. Since v4 has no trace table, a huge trace is not a huge row: it is a filter returning many rows, and what suffers is the interface when opening it.

Thread, session, user, and what is only read at the root

The handler recognises five metadata keys: langfuse_session_id, langfuse_user_id, langfuse_trace_name, langfuse_tags and langfuse_prompt. The first four are read only when parent_run_id is None, that is, only on the root run (CallbackHandler.py:587). Putting them in the config used to invoke a subgraph does nothing.

The recommended route, and the one that works without surprises, is the context manager:

from langfuse import propagate_attributes

with propagate_attributes(
 trace_name="agent-turn",
 user_id=user,
 session_id=thread_id,
 tags=["production", "support-l1"],
):
 graph.invoke(payload, config=cfg)

What LangGraph contributes on its own arrives anyway. In the measured run these show up as observation metadata: langgraph_node, langgraph_step, langgraph_triggers, langgraph_path, langgraph_checkpoint_ns, checkpoint_ns and thread_id, plus ls_provider, ls_model_type and ls_integration. The thread_id is also promoted to trace metadata.

That is fine, but keeping in mind the first article: in events_core what is indexed is metadata_names, that is the key names, not the values. Filtering by langgraph_node = "planificador" is a scan. If you are going to segment by node routinely, the right place is the observation name, which is where the node already appears, or a tag, assuming that tags are not indexed either. The cheap field for slicing by node in v4 is the observation name itself.

The session identifier deserves an explicit decision. LangGraph’s thread_id and Langfuse’s session_id are the same concept and they do not connect on their own. Mapping them one to one is the reasonable thing, and it is also what makes the whole conversation navigable in the interface even when each turn is a separate trace.

The filter that drops your own spans

This is the finding that surprises most people while setting it up. The Langfuse span processor does not export everything that passes through the TracerProvider. It exports what satisfies one of these three conditions (langfuse/_client/span_filter.py:104):

It was created by the Langfuse SDK’s own tracer.
It carries at least one attribute starting with gen_ai.
Its instrumentation scope matches one of thirty-five prefixes in a closed list.

The list covers the expected and a bit more: openinference, litellm, haystack, langsmith, strands-agents, pydantic-ai, autogen-core, vllm and a couple of dozen opentelemetry.instrumentation.* entries.

The consequence for an agent is direct. If you instrument your tools with OpenTelemetry by hand so that the ERP call or the Postgres query shows up inside the agent’s trace, those spans are dropped silently, with no log and no error. There are three ways out, in order of cleanliness:

Create those observations with the Langfuse SDK, with start_as_current_observation(as_type="tool"), which is what the rest of this article assumes.
Put some gen_ai.* attribute on the span, which is abusing the convention to smuggle the span through, so only if the span does describe an interaction with a model.
Send those spans to your normal OpenTelemetry collector instead of Langfuse, and correlate by trace identifier. For a sovereign platform with its own collector, this is the sensible option: Langfuse keeps the model side, the collector keeps the system side, and the trace identifier joins them.

The behaviour being silent is what makes this take an afternoon to find. It belongs in the runbook.

The model and the cost with a gateway in the middle

One detail that affects anyone putting LiteLLM in front, which is the setup described in the operational pair and in GPU multi-tenancy.

The model name the handler puts on the generation comes first from the ls_model_name metadata and, failing that, from the component’s serialisation (CallbackHandler.py:1297). With a ChatOpenAI pointed at the gateway, that name is whatever alias you defined in LiteLLM, for example qwen-32b-internal. Langfuse computes cost from its own model table, which does not know that alias, so the generation arrives with tokens and without cost.

It is fixed by defining the model in Langfuse under that same name with its price per million tokens, which is also the only way for the cost to reflect what your GPU costs rather than what somebody else’s API costs. It is an afternoon of work and it stops the agentic platform’s cost panel from sitting at zero forever.

How this looks in code

The minimum to have in place for a LangGraph agent to produce useful traces in Langfuse v4:

import os
from typing import Annotated, TypedDict

from langfuse import Langfuse, get_client, propagate_attributes
from langfuse.langchain import CallbackHandler
from langchain_core.messages import HumanMessage
from langgraph.graph import StateGraph, START
from langgraph.graph.message import add_messages
from langgraph.prebuilt import ToolNode, tools_condition

# 1. Client. Masking applies because the data goes through the SDK.
def mask(*, data, **kwargs):
 return redact_pii(data)

Langfuse(
 public_key=os.environ["LANGFUSE_PUBLIC_KEY"],
 secret_key=os.environ["LANGFUSE_SECRET_KEY"],
 host=os.environ["LANGFUSE_HOST"],
 mask=mask,
 environment="prod", # lowercase, dashes, 40 chars, must not start with "langfuse"
 sample_rate=float(os.getenv("LANGFUSE_SAMPLE_RATE", "1.0")),
)

# 2. Graph. The decision node carries "agent" in its name on purpose.
class State(TypedDict):
 messages: Annotated[list, add_messages]

graph = StateGraph(State)
graph.add_node("agent_planner", model_node)
graph.add_node("tools", ToolNode(TOOLS))
graph.add_edge(START, "agent_planner")
graph.add_conditional_edges("agent_planner", tools_condition)
graph.add_edge("tools", "agent_planner")
app = graph.compile(checkpointer=checkpointer)

# 3. Per turn: deterministic trace seeded with thread and turn, session = thread.
def run_turn(thread_id: str, turn: int, text: str, user: str):
 trace_id = Langfuse.create_trace_id(seed=f"{thread_id}:{turn}")
 handler = CallbackHandler(trace_context={"trace_id": trace_id})
 cfg = {
 "callbacks": [handler],
 "configurable": {"thread_id": thread_id},
 }
 with propagate_attributes(
 trace_name="agent-turn",
 user_id=user,
 session_id=thread_id,
 tags=["agent", "support"],
 ):
 out = app.invoke({"messages": [HumanMessage(text)]}, config=cfg)
 return trace_id, out

# 4. Resume after human approval: same seed, same trace.
def resume_turn(thread_id: str, turn: int, decision):
 from langgraph.types import Command

 trace_id = Langfuse.create_trace_id(seed=f"{thread_id}:{turn}")
 handler = CallbackHandler(trace_context={"trace_id": trace_id})
 cfg = {"callbacks": [handler], "configurable": {"thread_id": thread_id}}
 return app.invoke(Command(resume=decision), config=cfg)

# 5. Score the turn. Scores do not travel over OTLP: they have their own route.
get_client().create_score(
 trace_id=trace_id,
 name="resolved",
 value=1,
 data_type="BOOLEAN",
)

Two notes on what is not in the example. sample_rate is head sampling and per trace, so it samples whole turns, which is what you want: half an agent trace is no use to anybody. And if the process is a web server, the client is built once at startup, not per request, whereas the handler can be per turn with no problem, because with the seeded identifier it no longer holds state worth preserving.

Agent instrumentation checklist

Nodes representing a model decision carry agent in the name. The rest do not.
Every turn has a deterministic trace identifier seeded with thread and turn number.
LangGraph’s thread_id goes as Langfuse’s session_id, always.
Trace attributes are set with propagate_attributes, not as metadata on a subgraph.
The gateway’s model alias is registered in the Langfuse model table, with a price.
Tools doing work outside LangChain are instrumented with the Langfuse SDK, not with bare OpenTelemetry.
There is an alert on ingestion queue depth, and the average number of steps per turn is known.
Someone has measured, on the actual installation, how many observations and how many bytes a real turn produces. The figures in this article come from a synthetic bench.
Sampling is set through an environment variable, so it can be lowered without deploying code.

Traps

Type AGENT does not mean there is an agent. It means somebody wrote agent in the name. And the other way around: a whole graph without that string produces not one agent-type observation.

Putting langfuse_session_id in a subgraph’s metadata does nothing. It is only read on the root run.

An interrupt splits the trace in two. Unless you seed the identifier. And with several replicas, the handler’s in-memory state is never going to save you, because the resume can land on another pod.

Your OpenTelemetry spans do not arrive. The default filter drops them without warning if the scope is not in the list and there are no gen_ai.* attributes.

The number to watch is not users, it is steps per turn. Volume grows with the square of the steps, and the steps are decided by the prompt, not by the infrastructure.

You store megabytes and the interface shows you a thousand characters. The materialised view’s truncation to two hundred characters and the thousand-character read ceiling are still there. You pay for the full ingestion and see a clipping, unless you open the specific observation.

Cost comes out at zero with the gateway in front. Model alias unknown to the price table.

flush_at above the queue size blows up when constructing the client. With an OpenTelemetry ValueError that does not mention Langfuse anywhere.

The series: the nine articles

What goes into a trace: version 4’s data model, limits, precedences, scores, masking and indexes.
Putting LangGraph in front (this article): instrumenting an agentic platform and the measured cost of one turn.
Migrating from version 3 to 4 with no window: the three write-mode steps, the resumable background migrations and where the rollback point of no return sits.
The worker queues: the map of all thirty-nine, which pool to dedicate to each group, the per-queue switches, sharding and concurrency.
Real ClickHouse capacity and cost: how to measure bytes per observation with the system tables, the difference between the full table and the listings table, and the merge cost of full-text indexes.
Retention, deletion and data protection: why a deletion does not free disk, the mask cleaner that ships disabled, the pending deletion queue and the S3 lifecycle that has to be implemented by hand.
Backup and cross recovery: restore ordering across Postgres, ClickHouse and object storage, what each mismatch breaks, and how far event replay goes.
Saturation runbook: what to alert on from the queue metrics, the stuck probes, draining through the readiness endpoint and the dead letter queue.
Getting the data out: the blob storage integration to Parquet, batch exports and the metrics API, to build the data lake.

Sources

Own measurement of 13 September 2026: LangGraph 1.2.11, langchain-core 1.6.3 and Langfuse Python SDK 4.15.2, with OpenTelemetry’s InMemorySpanExporter and GenericFakeChatModel, no network.
Langfuse SDK 4.15.2 source: langfuse/langchain/CallbackHandler.py, langfuse/_client/span_filter.py, langfuse/_client/span_processor.py, langfuse/_client/span.py.
PyPI downloads over the last thirty days via pypistats, checked on 13 September 2026.
GitHub stars for LangGraph, CrewAI and the OpenAI Agents SDK, checked on 13 September 2026.

Completing Keycloak for MCP: the protected resource no SDK hands you ready-made, and the enterprise extension that skips the whole flow

Sat, 12 Sep 2026 09:00:00 +0200

A follow-up to Keycloak in an AI platform, which flagged the gap: resource indicators are not supported and protected resource metadata belongs to the MCP server. This post tries to close it. Verified against revision 2026-07-28 of the specification, the official Python and TypeScript SDKs, Keycloak 26.7.3 and LiteLLM 1.102.0.

TL;DR

The specification splits three obligations and the authorisation server covers only one. The MCP server must publish its protected resource metadata. The client must send the resource indicator in both requests, and must do so even if the authorisation server does not support it. The MCP server must validate that the token was issued for it. Keycloak does not understand the indicator and considers the metadata someone else’s, so all three end up in the resource layer.

The Python SDK mounts the metadata on its own, but degraded. As soon as the resource server URL is configured, the route appears. What that automatic route publishes carries a single authorisation server, reuses as its scope catalogue the ones the middleware requires, and leaves the name and the documentation null. To publish a complete document you have to mount the route by hand.

Audience validation comes switched off in Python and does not exist in TypeScript. In Python, if the resource URL is configured and the corresponding flag is not enabled, the SDK emits a deprecation warning and behaves as if it were disabled; the docstring itself promises that version 3 will set it to true. In TypeScript the bearer verification does three things, and none of them is looking at the audience. Neither of the two ships a signature verifier with public keys.

Token passthrough is forbidden in writing, and the substitute has a name. The specification says the MCP server must not accept or relay tokens that were not issued for it, and that if it calls upstream APIs the token must be a different one. The mechanism is token exchange, which Keycloak supports in its standard version with a limitation you need to know: the audience is a client identifier, not a resource URL.

There is an official extension that changes the whole picture, and it is stable. Enterprise-managed authorisation replaces the redirect to the MCP authorisation server with an exchange at the corporate identity provider, which evaluates the policy and issues a grant based on an identity assertion. It has been stable since June 2026 and there are servers in production.

Keycloak implements it halfway and in experimental. It only knows how to act as receiver, not as issuer, behind a feature flag, and against draft 01 when the working group is on 04. The official documentation literally says not to use it in production.

LiteLLM already implements it on the client side and does not document it. In 1.102.0 there is an authentication mode towards MCP servers that runs the two stages of that grant. There is not a single documentation file that mentions it.

You are here: the resource side

The previous post walked through the identity provider. This one walks through what sits on the other side of the token, which is where the MCP specification puts almost all its normative obligations.

The natural reading order is When MCP grows up for the basic setup, Keycloak in an AI platform for the identity piece, and this one for the missing layer. The MCP gateway seen from the inside is in its own post.

The analogy: the visa and the border checkpoint

A consulate issues visas. A border post checks them. They are two different offices, and the classic mistake consists of assuming that because the visa is genuine it is good for entering through any door.

The visa has written on it which country it is good for. That is the token’s audience, and here the problem appears: the consulate we work with does not know how to write the destination the traveller asks for, because it does not understand that field on the form. It knows how to write a destination if asked for it under a different name, by way of a pre-agreed stamp, which is the workaround of scopes and the audience mapper.

And then there is the other half, the one almost nobody builds. The border post has to exist, it has to announce where it is and from which consulates it accepts visas, and it has to read the destination written on the visa before letting anyone through. If the border post limits itself to checking that the stamp is genuine and does not look at the destination, any valid visa from the same consulate is good for entering. That is exactly what the two official SDKs do by default.

Part 1. The split of obligations

From revision 2026-07-28, with the normative words as they are:

Obligation	Whose	Status with Keycloak
Publish protected resource metadata (RFC 9728)	MCP server, MUST	Out of its scope, by design
Send the resource indicator (RFC 8707) in authorisation and in token	Client, MUST, even if the authorisation server does not support it	It does not understand it
Validate that the token was issued for oneself	MCP server, MUST	The resource’s responsibility
Not accept or relay other tokens	MCP server, MUST NOT	The resource’s responsibility
Validate the issuer of the authorisation response (RFC 9207)	Client, MUST; authorisation server SHOULD emit it	Supported
Consent for each dynamically registered client	MCP proxy with a static identifier, MUST	Configurable

The most confusing part of that split is the second line. The specification obliges the client to send the parameter regardless of whether the authorisation server supports it. With Keycloak, that parameter is lost: the official documentation says it cannot recognise it, and the standard behaviour towards an unknown parameter is to ignore it. The client complies, the token comes out, and what does not come out is the correct audience. The failure appears later and somewhere else, which is the worst kind of failure.

It is worth knowing where that work stands. The matter is issue 14355 of the project, open, with a milestone in 26.8.0. There was a complete implementation in pull request 35711, with a mapper of its own and an extension point for resolving resources, which went back to draft in October 2025 because of a total lack of tests and because it was decided to redo it in phases, starting with a single resource per client. In March 2026 a new issue was opened for experimental support, also with a milestone in 26.8.0 and with no associated pull request. In 26.7.x there is no feature flag that enables it.

What is new in the revision and affects the setup

Four things changed with respect to the previous revision and are worth collecting:

RFC 9207. A new section on validating the authorisation response. The client must record the issuer from the validated metadata document, and the comparison is literal: you are not allowed to normalise case, elide the default port, add or remove the trailing slash or re-encode characters before comparing.
Offline tokens leave the catalogue. A new section on refresh tokens: MCP servers should not include the corresponding scope either in the challenge header or in the supported scopes of their metadata.
Dynamic registration becomes deprecated. It goes from optional to optional and deprecated, retained for compatibility with authorisation servers that do not support client identifier metadata documents. And a new requirement is added to registration: the application type is mandatory, and a client must not reuse credentials from another authorisation server, it has to register again.
Step-up elevation is rewritten. The menu of three server strategies disappears and is replaced by two rules: the challenge’s scope attribute describes what is needed for the requested resource, with no obligation to include what has already been granted, and scope accumulation becomes the client’s responsibility. With a new requirement for the server: it must take scope hierarchies into account, where a broad one implies the narrow ones.

Part 2. What the SDKs give, and what they do not

Here is the uncomfortable part, and it is the main reason for writing this post. Verified by reading the code of the Python SDK and of the TypeScript 2.0.0-alpha one.

Protected resource metadata

In Python the complete model exists and it mounts on its own. The metadata class carries the RFC fields: resource, authorisation servers with a minimum of one, key set URL, supported scopes, bearer methods with a default value of header, name, documentation, policy, terms, and the client certificate and DPoP fields. The handler serves the document with a one-hour cache directive, and the URL is built by inserting the well-known path in front of the resource path, as the RFC requires.

As soon as the resource server URL is configured, the route appears with no further work. But what that automatic route publishes is degraded in three points: it passes a single authorisation server, it takes as its scope catalogue the ones the middleware requires (which are not the same as the ones the resource supports), and it passes neither the name nor the documentation, which come out null. To publish a complete document you have to call the route creation function by hand.

In TypeScript it exists and does not mount on its own. The function that builds the document emits only five fields: resource, authorisation servers, supported scopes, name and documentation. It does not emit the supported bearer methods, nor the key set URL, nor anything about DPoP. And serving it is explicit: either you call the response function from your own handler, or you mount the metadata router. There is no point in the SDK that does it on its own.

Audience validation

This is the one to fix on day one.

In Python it comes disabled. The token verifier is a single-method protocol, that is to say a hole the implementer fills. The access token’s resource field is the indicator you set yourself. And the comparison only happens if you ask for it: there is a function that normalises as a URL and ignores the trailing slash, but the resource server URL reaches it empty unless the validation flag is enabled.

The detail to read twice is in the configuration: if the resource URL is set and the flag is not, the SDK raises a deprecation warning and behaves as if the flag were false. The docstring says version 3 will set the default to true. Until then, a Python server configured with authorisation accepts tokens issued for another resource, unless the implementer enables the flag or validates the audience inside their own verifier.

In TypeScript it is not validated at all. Bearer token verification does exactly three things: strip the prefix, check the required scopes, and require that the expiry is present and not past. A search for audience or key set across the server and middleware packages returns no validation. It is all left in the hands of whichever verifier you plug in.

Neither of the two ships a signature verifier with public keys. In Python the only example is introspection against the authorisation server, and its audience check sits behind a flag that also comes out false. In TypeScript the interface is empty. That is: the component that validates the token, which is the one holding up the most important normative requirement of the specification, is your own code in both cases.

The authentication challenge

Python builds the challenge with the error and its description, and adds the metadata URL only if it is configured. It returns 401 with invalid token and 403 with insufficient scope, both from the same function, so both carry the metadata URL. What it never emits is the scope parameter, which revision 2026-07-28 asks to be included, and which is exactly what the client needs in order to know what to ask for.

TypeScript does emit the scope when there are required scopes, in addition to the metadata URL, and it correctly maps invalid token to 401 and insufficient scope to 403.

That is, each SDK gets a different half right. The Python one validates better and warns worse; the TypeScript one warns better and does not validate.

Scopes, which are not per tool

In both SDKs scopes are a static list at the transport mounting level, not per tool. There is no per-tool scope declaration and no step-up elevation help on the server side. Emitting a 403 with the specific scope that call requires, which is what the specification describes, is your own code.

That is the origin of almost all the fine-grained authorisation you have to build, and it is why there is a Part 6.

Which revision each one announces

Python is already on 2026-07-28. The revision appears in the known versions and in the list of modern versions, described as the ones that use the stateless envelope per request. The server discovery method is registered with a default handler, and the method and name headers are validated against the body.

TypeScript has it, but in a separate list. The latest protocol version constant is still 2025-11-25, because that list is only the one for the initial handshake. The modern era lives in a separate module, with a comment explaining the reason: keeping them deliberately separate so that adding a revision there never leaks a modern version string into a 2025-era handshake.

And it is worth recalling the mismatch that already came up in the MCP gateway post: LiteLLM 1.102.0 still announces 2025-06-18.

Part 3. Building the protected resource

With the above, the list of what you have to write is short and concrete.

1. A token verifier that validates the signature and the audience. Against the realm’s key set, checking issuer, expiry and that the audience contains the MCP server’s canonical URL and only things that concern it. Here the side effect of the Keycloak workaround comes in: if a client asks for two scopes from two different resources, out comes a token with two audiences, because each mapper contributes its own to the array. There is no client policy condition or executor in 26.7 that limits the number of audiences per token. The practical defence is the resource’s: reject tokens whose audience includes other resources, instead of merely checking that its own is present.

2. The complete metadata document, with all the authorisation servers you accept, the real catalogue of scopes the resource understands, and the name and documentation filled in. In Python, mounting the route by hand instead of leaving the automatic one. In TypeScript, mounting it, full stop.

3. The authentication challenge with the scope. In Python you have to add it, because the SDK does not emit it. And you have to take scope hierarchies into account, which is a new requirement of the revision.

4. The decision between local validation and introspection. Validating the signature locally costs no network, but revocation does not take effect until the token expires: there are no revocation lists for signed tokens. Introspection costs a round trip per request and only confidential clients can invoke it. The reasonable combination is a short token life and introspection on the operations that change state.

5. Binding state to the user. The new revision is explicit about this, because with protocol sessions gone, state is carried in handlers that travel as an ordinary tool argument. Servers must verify every incoming request and must not treat possession of a handler as authentication; and they should bind the handler to the user on the server side, for example storing the state under a key that combines the user identifier derived from the verified token with the handler, and reject the handler if someone else presents it. It is a change in the way of working compared with the sessions of before.

Part 4. The hop from the gateway to the MCP server

The real shape in an inference platform is not client against MCP server: it is client against gateway, and gateway against MCP server. That second hop is where it is decided whether the architecture is correct.

The specification leaves no room. The MCP server must not accept or relay tokens that were not issued for it. And if it makes requests to upstream APIs, it may act as an OAuth client of theirs, but the token it uses there is a different token, issued by the upstream authorisation server, and it must not forward the one it received.

The standard mechanism for that is token exchange, and here you need to know three Keycloak details.

First, the permission model changed. The old version required fine-grained administration permissions and an explicit exchange authorisation on the target client. The standard version, supported since 26.2, does not require them: it is enough for the requesting client to be confidential and to have the corresponding switch enabled. In exchange there is a surprising condition: the subject token has to carry the requesting client in its audience, unless it exchanges its own token. That is, the gateway needs to appear in the audience of the user token, which is achieved with an audience mapper in a default scope of the gateway. Only then can it narrow the audience to that of the specific MCP server.

Second, the audience is a client identifier, not a resource URL. The parameter filters audiences, that is to say it narrows them, which is what you want. But it takes the identifier of a client registered in the realm. The practical consequence is that the MCP server has to be registered as a client, and its identifier should be its canonical URL, so that the resulting audience matches what the resource validates. It is a naming trick, and it has to be documented for whoever comes after. The documentation itself acknowledges it: token exchange does not yet support the resource parameter.

Third, the subject token type is limited. The standard version only accepts access tokens as subject.

There is also a new experimental feature in 26.7 that adds a parameterised scope type for validating whether the requesting user is authorised to act on behalf of another. Interesting for delegation, but experimental.

What the gateway does today

In LiteLLM 1.102.0 the authentication modes towards MCP servers number twelve, with a default value of none. The ones that matter for this discussion are four.

oauth2_token_exchange implements the standard exchange. It sends the correct grant type, the subject token and its type, and the audience, never the resource. The omission is deliberate according to the code itself: fabricating a target risks an invalid target error. It caches the resulting token under a key that combines the subject token with the whole configuration, and with a lifetime equal to the token’s minus one minute. And it blocks, never degrades: with no incoming token it returns 401, which the edge turns into a challenge with the resource metadata URL; a rejection from the identity provider returns 401; a gateway configuration error returns 500; a transport failure, 503.

true_passthrough and oauth_delegate forward the client’s authorisation header as is. That is token passthrough, by the name the specification gives it, and its legitimate use is narrow: when the token the client presents was already issued for the target MCP server and the gateway is a mere transport. Outside that case, it is non-compliant.

oauth2_id_jag is the surprise, and it deserves a part of its own.

Part 5. The enterprise extension, which changes the picture

In June 2026 the MCP project published an authorisation extension that solves a different problem from the one the standard flow solves, and which in an organisation with an identity provider of its own is the real problem.

The normal flow is per user: each employee authorises each client against each MCP server. It works for consumer applications and it does not work in a company, because onboarding a person requires authorising dozens of services one by one and offboarding requires revoking them one by one.

The io.modelcontextprotocol/enterprise-managed-authorization extension inverts that. It is in the stable specifications directory of the extensions repository, it comes from SEP-990, and the flow is this:

The MCP client authenticates the user against the corporate identity provider through the normal flow, and stores the identity assertion, which may be an OpenID identity token or a SAML assertion.
When the server indicates that enterprise-managed authorisation is needed, the client exchanges that assertion at the corporate provider for an authorisation grant based on an identity assertion. The provider evaluates the organisation’s policy there: group membership, roles, conditional access.
The client presents that grant to the MCP authorisation server and obtains an access token.
The sentence that defines the extension, verbatim: the user is not redirected to the authorisation endpoint of the MCP authorisation server.

The MCP authorisation server validates the signature against the corporate provider’s key set, plus audience, issuer and expiry, and uses the subject as the user’s stable identifier, with the email as a fallback for linking earlier accounts.

What that buys is exactly what a compliance file asks for: policy in a single place, an auditable decision at the identity provider, and centralised revocation that takes effect across all clients at once. The employee who loses access stops receiving grants, without touching any server.

The underlying standard is a draft from the IETF’s OAuth working group, under the name of JWT identity assertion authorisation grant, revision 04 of May 2026, signed by people from Okta, Ping Identity and an independent author. It profiles identity chaining across trust domains by combining token exchange with the JWT profile.

And here comes the problem

Keycloak implements it only halfway. It has its own documentation page, behind a feature flag. And three limitations you need in front of you before designing anything:

It only acts as receiver. It accepts assertions issued by an external provider and issues local tokens. Native support for acting as issuer, the documentation says, is not yet fully implemented. In the extension’s flow, the issuer is the corporate provider: if that provider is Keycloak, the missing piece is exactly the one you need.
It is based on draft 01 while the working group is on 04.
The official documentation says not to use it in production.

The asymmetry is striking and worth pointing out: Keycloak does not support a standard published eight years ago such as resource indicators, and does partially support a draft from a few months back.

What is already in the gateway, undocumented

In LiteLLM 1.102.0, the oauth2_id_jag mode runs the two stages of the flow. In the first it asks the corporate provider for a token exchange with the requested token type specific to this grant, sending the user’s identity token as subject and, optionally, audience, resource and scope. In the second it presents the resulting assertion to the resource’s authorisation server with the JWT bearer grant type. In both the gateway authenticates as a client with a JWT signed by private key.

Two operational warnings that come out of the code and are nowhere else:

The subject comes from the caller’s identity token, or from the one captured at login. And only the generic OIDC provider captures assertions: with Google, Microsoft or SAML there is none, and all users fail. The code raises a warning when loading the configuration.
A badly configured server refuses with a 500 instead of falling back to the static credential. It is the right decision, and it is worth knowing before it happens.

And the main warning: there is no documentation. Not a single file in the repository that mentions this mode. Anyone using it is reading code, like this post.

Part 6. Per-tool authorisation

None of the previous pieces answers the question the business ends up asking: whether this person may call this tool with these arguments. The identity provider gives roles and scopes, and with that you do not model a graph of relations, as was already argued in the previous post.

What is published today, with sources:

OpenFGA is the only one with a dedicated product guide, updated on 9 September 2026. The pattern is a tool type with an invocation relation, checked on each call, and a listing query so the client only sees the tools it may invoke. It is a pattern and a modelling language, not a reference implementation.
Pomerium ships a per-tool policy criterion, with exact, prefix, suffix and list matching.
Kong offers it as a product, with per-tool control lists.
Traefik Hub goes as far as parameter-level restrictions, which is the finest published grain.
Envoy has an MCP filter that extracts protocol attributes for fine-grained access control, declared to be in active development and with no OAuth of its own: the hook is external authorisation towards a policy engine.

The lowest-friction option in a platform that already has a service mesh is external authorisation against a policy engine. The most expressive one, the relations model.

Part 7. What still has no standard

Two gaps the specification does not cover and that have to be closed from outside.

Tool definition change. Neither revision 2026-07-28 nor the SDKs pin a hash of the description or of the input schema. The best practices page covers confused deputy, token passthrough, server-side request forgery in discovery, state handler hijacking, server confusion and validation of the authorisation URL scheme, but there is no integrity or pinning requirement. The tool that does it is the Invariant Labs scanner, under the Apache 2.0 licence, which pins hashes to detect definition change after approval and also ships a proxy mode with guardrails. It already came up in the MCP gateway post and it remains the only answer.

The public guidance that does get into this is the NSA’s on MCP security design considerations, from May 2026, done with Carnegie Mellon’s software engineering institute. Two of its observations are directly actionable: that authorisation in MCP is optional and that the specification imposes no requirement on token lifecycle management, so expiry and rotation are left in the organisation’s hands; and the warning against dynamic tool discovery with no origin verification and no authorisation checks, which is the most direct acknowledgement of the risk in a state guidance document.

Part 8. Who covers what

With all of the above, the decision table for the layer you put in front of the MCP server:

Piece	Publishes resource metadata	Validates audience	Token exchange	Per-tool authorisation
Python SDK, automatic route	Yes, degraded	Not by default	No	No
TypeScript SDK	Does not mount on its own	No	No	No
Traefik Hub (commercial)	Yes, automatic	Not documented	Not documented	Yes, down to parameter
Kong, OAuth plugin for MCP	Not documented	Yes	Not documented	Yes, per-tool lists
Pomerium	Partial	Not documented	Not documented	Yes, per-tool criterion
agentgateway (Solo.io)	Yes	Yes, by JWT policy	Yes, including the assertion grant	Yes, with expressions
mcp-context-forge (IBM)	Not in release notes	Not documented	Yes, since 1.0.6	Yes, per-role control
Envoy, MCP filter	No	No	No	Via external authorisation
LiteLLM 1.102.0	Yes, and authorisation server metadata too	Not on the incoming JWT	Yes, and the assertion grant	Yes, per-key and per-team permissions

Two warnings about that table. The Kong plugin requires a minimum version of 3.12, is in technical preview and needs a licence for its AI edition; it also introduces a breaking change in 3.13, which starts treating all traffic as MCP in order to close a possible authentication bypass. And on oauth2-proxy I found no primary documentation either for or against protected resource mode: I am not claiming it does not have it, I am claiming it is not documented where I could look.

Reference configuration

Keycloak: the per-resource scope, optional, with its audience. That it is optional and not default is the key piece, because it is what makes the client have to ask for it explicitly and makes it work as a substitute for the resource indicator:

kcadm.sh create client-scopes -r plataforma \
 -s name=mcp:inventario -s protocol=openid-connect \
 -s 'attributes."include.in.token.scope"=true'

kcadm.sh create "client-scopes/<id>/protocol-mappers/models" -r plataforma \
 -s name=aud-mcp-inventario \
 -s protocolMapper=oidc-audience-mapper \
 -s 'config."included.custom.audience"=https://mcp.ejemplo.es/inventario' \
 -s 'config."access.token.claim"=true'

And the MCP server registered as a client with its canonical URL as identifier, so that token exchange produces the right audience:

kcadm.sh create clients -r plataforma \
 -s clientId=https://mcp.ejemplo.es/inventario \
 -s enabled=true -s publicClient=false -s consentRequired=true

The gateway, with token exchange instead of token passthrough:

mcp_servers:
 inventario:
 url: "https://mcp.ejemplo.es/inventario"
 transport: "http"
 auth_type: "oauth2_token_exchange"
 client_id: "litellm-gateway"
 client_secret: os.environ/GW_SECRET
 token_exchange_endpoint: "https://sso.ejemplo.es/realms/plataforma/protocol/openid-connect/token"
 audience: "https://mcp.ejemplo.es/inventario"
 subject_token_type: "urn:ietf:params:oauth:token-type:access_token"
 token_exchange_profile: "rfc8693"

The resource verifier, which is what nobody hands you ready-made. The essential bit is the last check, the one that rejects other resources’ audiences instead of settling for finding its own:

RECURSO = "https://mcp.ejemplo.es/inventario"

async def verificar(token: str) -> AccessToken | None:
 claims = jwt.decode(
 token, await jwks(),
 algorithms=["RS256"],
 audience=RECURSO, # validates that I am there
 issuer=ISSUER,
 )
 aud = claims["aud"]
 aud = [aud] if isinstance(aud, str) else aud
 # and that nobody else is: the scopes workaround allows
 # tokens with two audiences, and that reopens the confused deputy
 if set(aud) - {RECURSO, CLIENTE_GATEWAY}:
 return None
 return AccessToken(
 token=token,
 client_id=claims["azp"],
 scopes=claims.get("scope", "").split(),
 expires_at=claims["exp"],
 resource=RECURSO,
 )

And when mounting the server, the flag that does not come set:

auth = AuthSettings(
 issuer_url=ISSUER,
 resource_server_url=RECURSO,
 validate_token_resource=True, # without this there is only a warning
 required_scopes=["mcp:inventario"],
)

Checklist

Enable audience validation in the Python SDK, or write it into the verifier if it is TypeScript. Without that, the central normative requirement is not met.
Reject tokens with other resources’ audiences, not just check that your own is present.
Mount the metadata document by hand, with all the authorisation servers and the real catalogue of scopes.
Add the scope parameter to the challenge, in Python, and account for scope hierarchies.
Register each MCP server as a Keycloak client with its canonical URL as identifier, so that token exchange produces the right audience.
Replace any token passthrough mode with token exchange, except the narrow case where the token was already issued for the target.
Check that the gateway appears in the audience of the user token, or the exchange will fail.
Bind state handlers to the user on the server side, and never treat them as authentication.
Decide between local validation and introspection knowing that the first delays revocation until expiry, and shorten the token life accordingly.
Pin a hash of the description and the input schema from outside, because the specification does not cover it.
Put a policy engine in for per-tool authorisation, and do not try to model it with roles.
If you are considering the enterprise extension, check first who issues the grant: Keycloak today only knows how to receive it, in experimental and against an earlier draft.

Traps and things that are not what they look like

The Python SDK mounts the metadata document on its own, and that is why it looks solved. What it publishes carries a single authorisation server and no name and no documentation.

The audience validation deprecation warning does not enable anything. It warns and carries on behaving as disabled.

The TypeScript SDK does not validate the audience at all. It is not configuration, it is that there is no code for it.

Python’s challenge does not carry the scope, so a client that receives a 403 does not know what to ask for.

The scopes workaround allows two audiences in the same token. Each mapper contributes its own, and there is no client policy that limits it. The defence is in the resource.

The token exchange audience is a client identifier, not a URL. If the MCP server is not registered with its URL as identifier, the resulting audience will not match what the resource validates.

Token exchange requires the gateway to be in the audience of the user token. It is the condition that wastes the most time the first time round.

Dynamic client registration is deprecated as of this revision, and the substitute, client identifier metadata documents, is experimental in Keycloak and has an open bug with documents carrying unknown fields, which blocks login with real clients.

LiteLLM’s identity assertion grant depends on the generic OIDC provider. With Google, Microsoft or SAML no assertion is captured and all users fail.

Step-up elevation is no longer the server’s responsibility. Scope accumulation moved to the client in this revision.

The authentication elevation signal goes in a 401, not a 403. Insufficient scope is 403; asking for a higher authentication level is a different thing and a different code.

Closing

The summary of this post fits in one sentence: the identity provider is the cheap half of the problem, and the expensive half is the protected resource, which no SDK hands you ready-made and which in the two official ones comes with audience validation disabled or absent.

From that comes an order of work. First the token verifier, with the audience validated in both directions, because without it everything else is decoration. Second, the complete metadata document, mounted by hand. Third, replacing any token passthrough with exchange, with the MCP server registered with its URL as identifier. Fourth, a policy engine for what roles do not model.

And a scheduling decision worth taking with the information in front of you. The enterprise-managed authorisation extension is the right route for an organisation with a provider of its own, it is stable and it has servers in production behind it. What is not ready is Keycloak as issuer. Anyone who wants that flow today has two honest options: delegate the mechanics to a gateway that implements it, or wait. Putting it into production with a feature the official documentation marks as experimental and expressly advises against is not one of them.

Sources

Especificación MCP, revisión 2026-07-28: autorización, registro de clientes y buenas prácticas de seguridad.
Extensión de autorización gestionada por la empresa y su especificación estable, del SEP-990. Anuncio de 18 de junio de 2026.
draft-ietf-oauth-identity-assertion-authz-grant, revisión 04 de 21 de mayo de 2026, grupo de trabajo OAuth del IETF.
Código de los SDK oficiales de MCP: python-sdk y typescript-sdk 2.0.0-alfa, clonados el 12 de septiembre de 2026.
Integración de Keycloak con Model Context Protocol, intercambio de tokens y concesión de autorización por aserción de identidad, consultados el 12 de septiembre de 2026.
Incidencias de Keycloak 14355, 47117 y 51039, y petición de cambios 35711.
Código de LiteLLM 1.102.0, commit 9071ca50 del 11 de septiembre de 2026.
NSA y SEI, Model Context Protocol: Security Design Considerations, mayo de 2026.
OpenFGA, autorización de servidores MCP, actualizado el 9 de septiembre de 2026.
mcp-scan de Invariant Labs, Apache 2.0.
Documentación de agentgateway, Traefik Hub, Kong y Pomerium, consultadas el 12 de septiembre de 2026.

Keycloak in an AI platform: where it fits, what it solves and the two standards MCP requires and it does not implement

Sat, 12 Sep 2026 07:30:00 +0200

This post opens the identity block of the operations track. The previous one covered the gateway side: the JWT that does not validate audience, the user who never reaches the trace. This one covers the piece on the other side. Verified against Keycloak 26.7.3, revision 2026-07-28 of the MCP specification and the text of Royal Decree 311/2022 in the BOE.

TL;DR

Keycloak 26.7.3 was released on 31 August 2026 and is still incubating at the CNCF, which it entered in April 2023. The cadence is four minor releases a year and support in practice covers a single branch: the twenty security bugs fixed in 26.7.3 were not backported to the 26.4, 26.5 or 26.6 branches at the time of release.

The Quarkus-based distribution separates build and runtime configuration. What is set in the build phase is frozen into the image, and changing it at startup forces a rebuild. In containers, the pattern is to build into the image and start with the optimised flag.

Storage is relational and only relational. The alternative engine that was in development was withdrawn in version 25.0. Since 26.0 sessions are persisted to the database by default, which changes the load profile: the dominant cost in the official benchmarks is database CPU.

Organisations solves multi-tenancy inside a realm, fully supported since 26.0. It groups members, associates email domains and links identity providers per organisation. It is the alternative to replicating realms per customer.

The measured bottleneck is password hashing. Since 25.0 the default algorithm outside FIPS environments is Argon2, with around 7 MB of memory per operation. The official sizing guide gives one vCPU for every 15 password logins per second, against 120 per second for client credentials and for refreshes.

Keycloak does not implement two standards the MCP specification marks as mandatory. Protected resource metadata belongs to the MCP server, not the authorisation server, so there is nothing to hold against it there. Resource indicators do belong to it and are not supported. Keycloak’s own documentation declares partial conformance with the last three MCP revisions for that reason. The practical substitute is scopes plus an audience mapper.

There is a recent critical bug worth looking at today. CVE-2026-18963, scored 9.1, allowed resetting any user’s password without authenticating, administrators included. Fixed in 26.7.2, released on 19 August 2026.

You are here: the cross-cutting layer

Identity is not a stage of the pipeline, it is a layer that crosses all of them. It shows up in stack hardening as one of the seven layers of defence, in the model’s chain of trust as workload identity, in the gateway as token validation, and in the post about authenticating MCP servers as authorisation server.

What was missing was the post about the piece itself, and above all the decision of where to put it and what to expect from it.

The analogy: the civil registry and building permits

A small town hall has a population register, which says who lives there, and it has permits, which say who can do what. They are two different things and it is worth not confusing them, because the register is good for the whole town while a permit is good for one specific job on one specific street.

Keycloak is the register. It knows who each person is, identifies them reliably, knows which groups they belong to and issues a credential document that expires. What it does not do is decide whether this person can modify this particular file: that is the permit, and it lives in each service.

Confusing the two produces the most expensive design mistake in platforms of this kind, which consists of putting into the token everything that will need to be decided later. The register document grows, gets handed out everywhere, and when a permission changes you have to wait for everyone’s document to expire.

There is a third element the analogy captures well. A credential document issued for presenting at the tax office should not be good for entering the municipal depot. That is the token’s audience, and it is the central subject of the second half of this post.

Part 1. What Keycloak is today

Project status

Item	Value
Latest stable version	26.7.3, released on 31 August 2026
Previous ones	26.7.2 (19 August), 26.7.0 (9 July), 26.6.0 (8 April)
Cadence	four minor releases a year
CNCF	incubating since 10 April 2023, not graduated
Support	the current branch; earlier ones do not receive fixes reliably

That last point deserves underlining because it conditions the upgrade policy. The analysis of 26.7.3 shows that its twenty security fixes were not backported to earlier branches at release time. Operating Keycloak means keeping up to date with the current branch, not staying on a “stable” version for a year.

Since 26.0 there is a backward compatibility guarantee across minor versions for features marked as fully supported, and breaking changes are opt-in.

The internal architecture in five pieces

Two startup modes. The development one enables HTTP, does not resolve the hostname strictly and disables the theme cache. The production one requires a hostname and TLS, or it will not start. Confusing them in production is the first antipattern on the final list.

The build phase. The build command materialises the optimisations into the image. Build options, such as the database engine, the enabled features or metrics, are frozen. Runtime ones, such as the database password or the hostname, are applied at startup. Changing a build option at startup forces a rebuild with the corresponding latency cost. The correct pattern in containers is to build into the image and start with the optimised flag.

Extensions via SPI. Authenticators, protocol mappers, user storage, event listeners, hashing algorithms and secret vaults. They are packaged as JARs and resolved in the build phase.

Infinispan cache. Embedded by default, with database-based discovery as the recommended mechanism. The discovery alternatives via Kubernetes DNS, TCP and UDP are deprecated. There are local caches, with default capacities of 10,000 entries in several of them, and replicated caches for sessions, client sessions, offline sessions, authentication sessions, login failures and action tokens. Remote mode, with an external Infinispan, is needed for first-generation multi-site deployment.

Relational storage. JPA over a database. The alternative engine was removed in version 25.0. The databases supported today include PostgreSQL from 14 to 18, MariaDB and MySQL in their extended support versions, SQL Server 2019 and 2022, Oracle 19c and 23.x, and the managed variants Aurora PostgreSQL and Azure SQL.

The minimum conceptual model

Before touching anything you need to be clear on eight concepts, and the order in which they relate:

Realm: the isolation boundary. Its own users, its own signing keys, its own policies. Two realms share nothing.
Client: an application that asks for tokens. Public if it cannot keep a secret, such as a browser application; confidential if it can, such as a service; and bearer-only if it merely validates tokens without requesting them.
Client scope: a reusable bundle of scopes and mappers, assigned to clients as default or as optional. It is the unit with which you bound what a token carries.
Protocol mapper: what projects an attribute, a role or an audience into the token. The audience mapper is the one that solves the central problem of this post.
Role, realm or client, and composite role, which aggregates others.
Group, hierarchical, with inherited roles and attributes.
Identity provider: federation towards another external OIDC or SAML. User federation: LDAP, Active Directory and Kerberos, with synchronisation and attribute mapping.
Authentication flow: a sequence of executions with required, alternative, conditional or disabled requirements, and subflows. It is where you build a second factor or a check of your own.

And one feature that changes architectures: organisations, fully supported since 26.0. It solves multi-tenancy inside a single realm, grouping members, associating email domains and linking identity providers per organisation, so that provider discovery is done by the user’s email domain. The classic alternative, one realm per customer, multiplies administration and shares no configuration.

Part 2. Who can authenticate against it

This is the table you need before drawing anything. The right-hand column is usually the one that decides.

Piece	Protocol	Cost
Kubernetes API	native OIDC, or the structured authentication configuration file	Free
LiteLLM	OIDC and JWT, plus UI SSO	JWT and SSO above five users: Enterprise
vLLM	static key only	No OIDC
Langfuse	generic OIDC and Keycloak provider	SSO free; per-project access control: Enterprise
Grafana	generic OAuth yes in the open edition	Team synchronisation: Enterprise
Backstage	generic OIDC and community-maintained Keycloak provider	Free
Open WebUI	OIDC, with role management by claim	Free
LibreChat	generic OIDC, with Keycloak documented	Free
Argo CD	native OIDC or Dex, with groups into its RBAC file	Free
MinIO	OIDC with role assumption via web identity	See the note below

Two clarifications that change decisions.

Kubernetes. The modern mechanism is structured authentication configuration, which allows several JWT issuers plus validations and mappings with CEL expressions. The corresponding proposal, KEP-3331, went alpha in 1.29, beta in 1.30 and is stable since 1.34. It is the route to use today, and it replaces the loose API server flags.

vLLM has no identity. All there is is a static key via environment variable. This is not a defect to fix, it is an architectural fact: the engine sits behind the gateway and its access control is network-based, not identity-based. Any design that exposes vLLM directly to users is giving up per-person traceability.

About MinIO a note of honesty is in order: the public documentation now redirects to the commercial edition, so the detail of how it maps policies from a claim is verified there and not in the open edition. Anyone depending on that mechanism should confirm it against their own version before designing on top of it.

Part 3. Humans and machines

They are two different problems and it is worth solving them separately.

People

Authorization code with PKCE, with the S256 method. The MCP specification requires it when the client is technically capable, and it goes further: it mandates rejecting the flow if the authorisation server metadata does not publish the supported challenge methods.

Workloads with no person behind them

Here the obvious option is client credentials with a Keycloak service account, authenticated with a secret or, better, with a JWT signed by private key.

But there is a less known and better option for workloads inside the cluster: federated client authentication, introduced in January 2026 under the banner of doing away with secrets. It lets a client authenticate with a token issued by an external provider, and one of those providers is Kubernetes. The announcement placed full support for the Kubernetes service account variant and for OIDC in version 26.6; the SPIFFE variant, by contrast, expressly says it will stay in preview until the SPIFFE client authentication standard is finalised at the IETF.

That means a pod can authenticate against Keycloak with the token the kubelet already projects into it, with no additional secret to rotate. It is worth confirming the exact status and the flag name against the release notes of whichever version you have deployed, because the feature is recent and has changed status between 26.5 and 26.6. It is preferable to token exchange, whose standard version is supported but is internal-to-internal only, while the old one, which covered external-to-internal and impersonation, is deprecated and in preview.

Keycloak against SPIFFE

The question always comes up and the answer is that they solve different things with a small overlap.

SPIRE attests the node and the process, and issues short-lived identity documents with no prior secret. Keycloak does not attest workloads: it trusts a credential someone had to place there, except in the federated authentication case just mentioned, where the one attesting is Kubernetes. In exchange, Keycloak federates people, issues tokens with business data and governs consent, things SPIFFE does not do.

The practical rule: SPIFFE for service-to-service identity inside the perimeter, Keycloak for anything that has a person behind it or crosses the perimeter. The SPIFFE detail is in the close of the chain of trust series.

Part 4. MCP, and the two missing standards

This part has the most consequences and is the worst documented outside the primary sources.

What the specification requires

From revision 2026-07-28, with the normative words as they appear:

The MCP server must implement RFC 9728 protected resource metadata, and the client must use it to discover the authorisation server.
The client must implement RFC 8707 resource indicators, including the resource parameter in both the authorisation request and the token request, and must send it regardless of whether the authorisation server supports it.
The MCP server must validate that the access token was issued specifically for it as the audience, and must not accept or relay any other token.
Authorisation servers and clients should support client identifier metadata documents. RFC 7591 dynamic registration becomes optional and is declared deprecated, retained for compatibility.

The revision also adds the issuer in the authorisation response, from RFC 9207, and a stateless model with state handlers where it is expressly stated that holding a state handler is not equivalent to being authenticated.

What Keycloak supports

From its own MCP integration guide:

Standard	Keycloak
OAuth 2.1 and authorisation server metadata	Supported
RFC 9207, issuer in the response	Supported
RFC 7591, dynamic registration	Supported
Client identifier metadata document	Supported, experimental, after enabling the feature
RFC 8707, resource indicators	Not supported
RFC 9728, protected resource metadata	Out of scope, it belongs to the MCP server

The conformance declared by the project itself is “partially supported, without resource indicators” for the last three MCP revisions.

Two practical conclusions:

The protected resource metadata document is published by the MCP server or a proxy in front of it, never by Keycloak. That is correct and there is nothing to fix, but you have to remember to implement it, because the client needs it to discover where to go.
Without resource indicators, the audience is bounded with scopes. The official route is to use the scope parameter instead of resource, and to hang from that client scope an audience mapper with the MCP server URL as the included custom audience.

It is a workaround, it works, and it has to be documented in the design because anyone coming after will look for the standard parameter and will not find it.

Part 5. The confused deputy

With the above on the table, the classic mistake becomes obvious on its own. A token with a generic audience turns any compromised service into a master key: whoever obtains the token a user presented to the dashboard can present it to the gateway, and from there to the MCP server.

The MCP specification is explicit in three sentences worth quoting verbatim in any architecture document:

MCP servers must not accept tokens that were not explicitly issued for them.
If the MCP server calls upstream APIs, it must not forward the token it received from the client.
MCP proxies that use static client identifiers must obtain user consent for each dynamically registered client before forwarding to third-party authorisation servers.

Translated into the specific figure this track has been dealing with: the gateway must not forward the user’s JWT to third-party MCP servers. It has to obtain a different token with the right audience, or act as an OAuth client in its own right. By default, as already documented in the MCP gateway post, the behaviour is to use the server’s own credential, which is the correct one; there are two modes that forward the caller’s token and both have a name in the specification.

Part 6. Two hours of task against five minutes of token

The specification pushes towards short-lived tokens and mandatory refresh token rotation in public clients. A two-hour agentic task comfortably exceeds the life of an access token and may also exceed session idle time if nobody refreshes.

Three real options, with their cost:

Offline tokens. They survive session expiry, and expire through their own idle timeout or through a maximum if that limit is enabled. The risk is obvious: a near-permanent credential, hard to inventory. And the MCP specification adds that MCP servers should not include that scope either in the authentication header or in the supported scopes of their metadata.

Automatic refresh in the client. It is the right answer when there is a client that can implement it, with refresh token rotation and secure storage. For an agent with no person behind it, it is better to renew via client credentials.

Long-lived virtual keys in the gateway. It works, and it is what most people end up doing. The price has to be written down in the risk analysis: identity ends up decoupled from the provider, so deactivating someone in Keycloak does not invalidate their key, and traceability back to the real person depends on the discipline with which the keys were created. The levers that compensate are expiry, automatic rotation and a mandatory alias, covered in the virtual keys post.

A note of honesty about the default values. The numbers going around (five minutes of access token, thirty of session idle time, ten hours of maximum) do not appear published in the administration guide and I did not locate them in the code. Treat them as a console observation and confirm them in your own realm before quoting them in a document.

Part 7. Sizing and operating

On Kubernetes

The operator exposes three custom resources: the server, the realm import and the OIDC client. Probes and metrics live on management port 9000, with the health and readiness paths, and metrics in Prometheus format provided the corresponding option was enabled in the build phase. That port is not exposed publicly.

For high availability there are three shapes. A single cluster spread across zones, with no additional infrastructure. First-generation multi-cluster, with an external load balancer and an external Infinispan at each site. And second-generation multi-cluster, introduced in 26.7.0 and still in preview, which connects clusters without external Infinispan, using the synchronously replicated database as the source of truth, with less cache and more database load.

The official sizing figures

From the CPU and memory sizing concepts guide, which is still published:

1 vCPU for every 15 password logins per second, tested up to 300 per second.
1 vCPU for every 120 client credential grants per second, tested up to 2,000.
1 vCPU for every 120 token refresh requests per second, tested up to 435.
Leave a 150 % CPU headroom for spikes, startup and failover.
1,250 MB of memory per pod with realm data and 10,000 cached sessions. Keycloak allocates 70 % of the limit to the heap and around 300 MB outside it, so the limit comes from subtracting those 300 MB from the expected usage and dividing by 0.7.

The stated assumptions behind those figures include Argon2 with five iterations, sessions in the database and the default cache of 10,000 entries.

The difference between 15 and 120 is the whole story of Keycloak’s performance: the cost is in password hashing. Since 25.0, the default algorithm outside FIPS environments is Argon2, with around 7 MB of memory per operation. Before it was PBKDF2, whose iterations were multiplied by ten in version 24. In FIPS mode PBKDF2 is still used.

From that comes an architectural consequence for an AI platform: machine workloads, which go through client credentials, cost eight times less than a human login. Sizing by number of services is cheap; sizing by morning login peaks is the expensive part.

The benchmark

From the 26.4 performance report, published on 1 October 2025: up to 2,000 logins per second and 10,000 refreshes per second with three pods, 74 vCPU in total and 8 GB per pod, over an Aurora PostgreSQL database, with almost linear vertical scaling.

Two useful findings from that report. With 20 ms of round-trip time between sites, the 99th percentile rises from 51 to 130 ms, and the project advises against deploying across different regions. And raising the cache from 10,000 to 200,000 entries brought the database CPU peak down from 77.77 % to 63.77 %, in exchange for somewhat longer garbage collection pauses.

Security

The urgent matter, if the deployment is below 26.7.2:

CVE-2026-18963, score 9.1, critical. A bypass of the email action token in the credential reset flow allowed resetting any user’s password with no prior authentication, administrators included. Fixed in 26.7.2, released on 19 August 2026. The temporary mitigation, if you cannot upgrade, is to disable password recovery in all realms. No known exploitation was recorded as of the end of August.

In 26.7.3 there are another nineteen, among them the missing hostname verification in the TLS certificate towards LDAP with a score of 8.8, two tenant restriction bypasses in token exchange with Microsoft and Google brokers, and two access control bypasses in organisations.

Official hardening comes down to four decisions:

Strict hostname, which is the default value since the second generation of that configuration. The server does not infer the URL from headers, which prevents poisoning email links and redirects.
Administration console on a different hostname, and network-restricted.
Proxy headers declared explicitly, trusting only a controlled reverse proxy.
Separate management port, not exposed.

Part 8. The mapping to the ENS

The codes and titles below are verified against the official text of Royal Decree 311/2022 published in the BOE, Annex II.

Code	Literal title	What a central IdP contributes
op.acc.1	Identificación	A unique identifier per entity, user or process that gains access
op.acc.2	Requisitos de acceso	Permissions are granted by role and group, not by loose credential
op.acc.3	Segregación de funciones y tareas	It supports it with roles, but the control is organisational
op.acc.4	Proceso de gestión de derechos de acceso	Least privilege and a specific remote access policy
op.acc.5	Mecanismo de autenticación (usuarios externos)	Flows and factors configurable per flow
op.acc.6	Mecanismo de autenticación (usuarios de la organización)	Reinforcements required already at the low level
op.exp.8	Registro de la actividad	It contributes part of the events, not all of them
op.exp.10	Protección de claves criptográficas	Management and rotation of the realm signing keys

Two nuances an auditor is going to look for.

The first, about op.acc.1: the text requires every entity that accesses the system, whether user or process, to have a unique identifier. That includes agents. A virtual key shared between several agents does not comply, and that is exactly what usually happens when JWT authentication falls outside the budget.

The second, about op.exp.8: the control requires recording the identifier of the user or entity associated with the event, the date and time, what information it is performed on, the type and the result. Keycloak contributes the authentication events. The usage events are contributed by the gateway and the MCP servers, and there the two already documented gaps reappear: adding and removing MCP servers generates no audit log and the trace user is set by the client.

The correspondence between control and product in this table is my own interpretation and must be validated by the certification body. The codes and titles are not: those are literal from the BOE.

Part 9. The configuration to leave written down

Three concrete pieces, so as not to stay at the level of concepts.

Hardened startup, as container arguments:

kc.sh start --optimized \
--hostname=https://sso.ejemplo.es \
--hostname-admin=https://sso-admin.interna.ejemplo.es \
--proxy-headers=xforwarded \
--health-enabled=true \
--metrics-enabled=true \
--http-management-port=9000 \
--cache=ispn --cache-stack=jdbc-ping \
--db=postgres

The metrics, health, cache and database options are build options: they go into the image, and passing them here forces a rebuild if they do not match.

The client scope with audience, which is the workaround forced by the lack of resource indicators. Created with the administration tool:

# one scope per protected resource
kcadm.sh create client-scopes -r plataforma \
 -s name=mcp-inventario -s protocol=openid-connect \
 -s 'attributes."include.in.token.scope"=true'

# the mapper that sets the token audience
kcadm.sh create "client-scopes/<id>/protocol-mappers/models" -r plataforma \
 -s name=aud-mcp-inventario \
 -s protocol=openid-connect \
 -s protocolMapper=oidc-audience-mapper \
 -s 'config."included.custom.audience"=https://mcp.ejemplo.es/inventario' \
 -s 'config."access.token.claim"=true'

The client asks for that scope and receives a token whose audience is only that MCP server. The MCP server validates the audience and rejects any other token, as the specification requires.

Realm import via GitOps, with the operator’s custom resource, so that the configuration does not live only in the console:

apiVersion: k8s.keycloak.org/v2beta1
kind: KeycloakRealmImport
metadata:
 name: plataforma
 namespace: identidad
spec:
 keycloakCRName: sso
 realm:
 realm: plataforma
 enabled: true
 # without this, the recovery email is attack surface
 resetPasswordAllowed: false
 bruteForceProtected: true
 sslRequired: all

Part 10. What Keycloak does not solve

Fine-grained authorisation over resources. The question “can this person see this particular trace” is a relation between objects, not a role. That problem is solved by Zanzibar-style engines, with OpenFGA and its object, relation and user tuples, or a policy language such as Cedar. Keycloak gives roles and scopes, and with those you do not model a graph of relations.

Machine secrets. Storage credentials, provider keys, database passwords. That is a secrets manager, covered in stack hardening.

Cryptographic workload identity with attestation. That is SPIFFE and SPIRE. In Keycloak it exists as an identity provider in preview, which is a different thing.

Authorisation inside the model. Which MCP tool an agent may invoke, and with what arguments. The specification offers scopes, the insufficient scope error and step-up elevation, but the per-tool and per-argument decision is taken by the MCP server or a policy engine, not by the identity provider.

Antipatterns

Forwarding the user’s token from the gateway to third-party MCP servers. Explicitly forbidden by the specification.
Static client identifier with dynamic registration and no per-client consent, which is the exact recipe for the confused deputy.
General scopes, of the full-access kind, and publishing the entire catalogue of supported scopes in the metadata.
Treating a state handler as authentication, which the new revision expressly forbids.
Following OAuth discovery towards any URL, which opens the door to requests towards internal addresses. It also applies to the authorisation server when it downloads a client metadata document.
Trust headers badly exposed in chat front ends. The Open WebUI documentation warns that an incorrect configuration allows authenticating as any user.
Starting in development mode in production, with the hostname not resolved strictly.
A single shared static key as the only control between the gateway and the engine.

Startup checklist

Check the version. Below 26.7.2 there is a critical password reset bug.
Build the image with the build phase done and start with the optimised flag.
Strict hostname, administration console on another hostname, separate management port and not exposed.
Decide between organisations and one realm per tenant before creating the second tenant.
One client scope per resource, with an audience mapper, because resource indicators are not supported.
In the gateway, define the two audience and issuer variables. Without them none of that is validated.
For workloads inside the cluster, look at the Kubernetes identity provider before handing out client secrets.
Size by login peaks, not by number of services: the difference is eightfold.
Export the Keycloak events to the same place the gateway’s go, because the activity logging control needs them together.
Write into the design what the system does with a two-hour task, before someone solves the problem with an offline token.

Closing

Keycloak is a mature piece that solves one bounded problem well: knowing who the caller is, and saying so verifiably in a document that expires. In a sovereign inference platform that is worth a lot, because the alternative is an inventory of static keys spread across ten services that no audit can walk through.

What is not advisable is asking more of it. Fine-grained authorisation lives in each service, workload identity with attestation lives in SPIFFE, and the decision of which tool an agent may call lives in the MCP server.

And there is one concrete gap worth noting down, because it does not close on its own: the MCP specification marks two standards as mandatory, and of the two that would fall to an authorisation server, resource indicators are not implemented. The workaround with scopes and an audience mapper works, but it is a workaround, and whoever comes after will look for the standard parameter. Leaving it written in the design saves an afternoon.

Sources

Keycloak 26.7.3, 31 de agosto de 2026, y descargas.
Configuración del servidor, caché, bases de datos y nombre de host.
Integración con Model Context Protocol, intercambio de tokens y autenticación federada de clientes, 27 de enero de 2026.
Dimensionado de CPU y memoria e introducción a alta disponibilidad.
Bancos de pruebas de rendimiento de Keycloak 26.4, 1 de octubre de 2025.
Especificación de MCP, revisión 2026-07-28, autorización y buenas prácticas de seguridad.
KEP-3331, configuración estructurada de autenticación.
Real Decreto 311/2022, Anexo II.
Registro de CVE-2026-18963 en la base de datos de Red Hat y notas de la versión 26.7.3 en GitHub.
CNCF, proyecto Keycloak.

Langfuse v4, day 2 (1 of 9): the data model changed underneath, and what really goes into a trace

Sat, 12 Sep 2026 07:00:00 +0200

First article in a series of nine about operating Langfuse v4 in production. The architecture of version 3 and its ten performance levers were already covered in Langfuse from the inside; the seam with the gateway, in the operational pair. This series is about day 2. Verified against Langfuse 4.35.0, commit of 11 September 2026.

TL;DR

Version 4 is observations first. There is no trace table. A trace is a set of observations that share an identifier, and the trace attributes live denormalised in every row: name, user, session, tags, version, release and environment (clickhouse/migrations/canonical/0039_create_events_full.up.sql). The whole intuition of “first I create the trace and then I hang things off it” stops describing the storage.

There are ten observation types, not three. Alongside the classic SPAN, EVENT and GENERATION come AGENT, TOOL, CHAIN, RETRIEVER, EVALUATOR, EMBEDDING and GUARDRAIL (packages/shared/src/domain/observations.ts:5). Using the right type changes what the interface knows how to group.

The real limits are in the code and they do not match what people assume. The body of a request to the classic ingestion route is capped at 4.5 MB (api/public/ingestion.ts:40). The OTLP body, at 512 MiB, compressed and decompressed, with a 413 when it is exceeded. An OTLP span above 9.5 MB is only written to the log, not rejected. Identifiers accept 800 characters. The trace name, 1,000.

The ingester looks for the user in five places and the session in six. With strict precedence order, starting from Langfuse’s own attributes and ending in the metadata of other frameworks. Among the session ones sits gen_ai.conversation.id, which is the OpenTelemetry standard, so a conformant instrumentation works without translation.

Scores do not travel over OTLP. Zero occurrences of the word across the 3,864 lines of the OTLP ingestion processor. They come in through the classic ingestion route, through the scores endpoint or through the interface. Any design that assumes an evaluator can write its score down the same channel as the trace is wrong.

The classic masking function does not cover what almost everyone thinks it covers. It only acts on data that passes through the SDK calls. The attributes produced by a third-party automatic instrumentation come out unmasked. There is a new function that does act on the batch attributes, but it leaves out span events, which is where the recent conventions put the messages.

Filtering by tags or by status message is a scan. In the listings table, the indexed fields are the span identifier, the trace identifier, the user identifier, the session identifier, the dates, the model and the metadata names. The name, the tags, the version and the status message have no index.

You are here: OBSERVE, day 2

The previous series about this pair covered selection and setup. The six-service architecture explained version 3; tracing with OpenTelemetry explained the pipeline; the pair with the gateway explained where the trace gets lost between the two pieces.

What is left uncovered is the work of the next three years: migrating, sizing ClickHouse, retaining and deleting, taking backups, alerting, getting the data out. This series is about that, and it starts with the question that conditions everything else, which is what gets stored.

The analogy: the archive that changed system

An archive with a folder per case file and documents inside each folder is easy to understand. You open the folder, you look at what is there. The problem shows up when you have to find every document of one kind over the last two years: you have to open every folder.

Large archives solve that the other way around. There is no folder: each document carries, copied into its header, the case reference, the client name and the date. Searching by client is immediate. Reconstructing a case file is grouping by reference. In exchange, each document weighs a little more, and what gets written into that header matters a great deal, because everything you can search for comes from there.

Langfuse v4 made exactly that change. What used to be a folder with attributes of its own now travels repeated in the header of every document. The two operational consequences of this article come from there: which fields are worth filling in and which fields are not worth filling with content.

The data model, as it is in the code

The tables

Three new pieces in ClickHouse:

events_full (migration 0039): the table of record, with the full input and output, compressed with ZSTD level 3 and indexed with full-text indexes over the lowercase versions. ReplacingMergeTree engine ordered by event timestamp and deletion flag, partitioned by month of the start time.
events_core (migration 0040): the listings table, lighter.
events_core_mv (migration 0041): the materialised view that fills the previous one, and which truncates to two hundred characters the input, the output and each metadata value.

That truncation to two hundred characters is the first thing to know. What you see in a listing is not what is stored; it is a clipping. And when reading a full trace a second cap applies on top, LANGFUSE_SERVER_SIDE_IO_CHAR_LIMIT, with a default value of a thousand (packages/shared/src/env.ts:513).

In the events_full table there is a configuration line that says a lot: index_granularity_bytes = '64Mi', with the comment that it avoids very small granules because of large rows. It is the schema itself acknowledging that the rows of this system are bulky.

The ten observation types

SPAN, EVENT, GENERATION, AGENT, TOOL, CHAIN, RETRIEVER, EVALUATOR, EMBEDDING and GUARDRAIL (domain/observations.ts:5), with their corresponding ingestion events (server/ingestion/types.ts:279).

For a platform with agents, the difference between marking a tool call as a generic SPAN or as a TOOL is the difference between being able to count tool calls per session and having to reconstruct it by hand. It is worth reviewing the instrumentation with this list in front of you.

The level is still DEBUG, DEFAULT, WARNING and ERROR (domain/observations.ts:31), with a free-form status message.

The limits, with their source

Field	Limit	File
Trace, span and observation identifiers	800 characters, no carriage return	`ingestion/types.ts:10`
Trace name	1,000 characters	`ingestion/types.ts:428`
Environment	40 characters, pattern of lowercase, digits, hyphen and underscore	`ingestion/types.ts:226`
Text-type score	500 characters	`domain/scores.ts:44`
Score config name	35 characters	`domain/score-configs.ts:11`
Classic ingestion body	4.5 MB per request	`api/public/ingestion.ts:40`
OTLP body	512 MiB, adjustable	`web/src/env.mjs:583`
OTLP span considered large	9.5 MB, only logged	`packages/shared/src/env.ts:193`
Name, status message, version, release, tags	no declared limit	`ingestion/types.ts:424`
Metadata	no declared limit	`ingestion/types.ts:425`

The last two are the dangerous ones, because they have no schema cap and the real ceiling ends up being the one on the request body. A three-megabyte metadata value passes validation and reaches ClickHouse.

A note on the langfuse prefix in the environment field: it is reserved for the product’s own internal traces and is stripped idempotently from public inputs (ingestion/types.ts:228), including the case of someone trying to sneak it in duplicated. The default value is default.

On rate limits: in the cloud there are per-plan quotas, but in self-hosted they are not applied. The service returns directly that there is no limiting (features/public-api/server/RateLimitService.ts). It is good news for bulk exports and bad news for anyone relying on that mechanism as protection.

The two instrumentation routes

What gets lost over OTLP, and it is one specific thing

The check is simple and the result is blunt: there is not a single occurrence of the word score in the OTLP ingestion processor, a file of 3,864 lines. Scores only come in through three places: the score-create event of the classic ingestion route, the POST /api/public/scores endpoint, or the interface.

This has a design consequence worth being clear about before building evaluators: the trace can travel through the OpenTelemetry collector with everything good that brings, on-disk queue, retries and sampling, but the score an evaluator puts on it has to go down a different channel and with the project keys.

Everything else does travel over OTLP. Whatever is not in the mapping lists is kept as metadata converted to a string, visible but outside the indexed fields.

The precedence chain, which is the map you need to have

Verified in packages/shared/src/server/otel/OtelIngestionProcessor.ts:

User identifier (line 2545), in this order: langfuse.user.id, user.id, the observation metadata, the trace metadata, and lastly the Vercel SDK metadata.

Session identifier (line 2565): langfuse.session.id, session.id, gen_ai.conversation.id, observation metadata, trace metadata, Vercel metadata.

Environment (line 2383): langfuse.environment, deployment.environment.name, deployment.environment; for each key the span attributes are checked first and the resource attributes after.

Model (line 2753): langfuse.observation.model.name, gen_ai.response.model, ai.model.id, gen_ai.request.model, llm.response.model, llm.model_name, model.

Level (line 1250): the native attribute, with accepted aliases where SUCCESS and OK become DEFAULT, WARN becomes WARNING and FATAL or CRITICAL become ERROR; if there is no attribute and the span status is an error, the result is ERROR.

Tags (line 3227): langfuse.trace.tags, langfuse.tags, observation and trace metadata, Vercel metadata, and tag.tags.

Input and output (line 1790): a chain of some twenty frameworks in strict order, starting from Langfuse’s own attributes and going through Genkit, the Vercel SDK, the standard inference operation details event of the conventions from 1.37 onwards, the per-role message events, the prompt and completion attributes, Logfire, MLflow, TraceLoop, OpenInference and the gen_ai.input.messages and gen_ai.output.messages attributes.

That this chain has twenty entries says something about the real state of the conventions: it was already noted that the generative AI conventions were extracted into their own repository without any published version. While that remains the case, the backend has to guess, and it guesses with a precedence list.

The practical part: if Langfuse’s own attributes are emitted, they always win. If you want backend-neutral instrumentation, gen_ai.conversation.id and deployment.environment.name are the two standard attributes that Langfuse does understand.

Metadata, and how it ends up in the table

A metadata value is flattened into two parallel arrays, one of names and one of values, with dot-separated paths and everything converted to a string (otel/utils.ts:239, applied in worker/src/services/IngestionService/index.ts:363). A nested object with twenty keys is twenty entries in each array.

There is a cap of 10,001 positions when reconstructing arrays from OTel attributes (OtelIngestionProcessor.ts:271), which is high but it exists.

Sessions, users and cardinality

A session is a text column with a Bloom filter index at 1 %. The same goes for the user. The Postgres table that accompanies sessions stores only whether it is bookmarked, whether it is public and its environment; there is no counter and no cap. There is no limit on traces per session in the code, and the documentation recommends session identifiers below 200 ASCII characters.

The cardinality difference matters and it is not intuitive: user and session are plain text, so high cardinality does not break the schema, it only inflates the Bloom filters. The environment, by contrast, is declared low cardinality. Generating a dynamic environment value per tenant degrades ClickHouse measurably.

Aggregated cost comes out of a cost details map with materialised input, output and total columns. The classic sessions endpoint is deprecated; the replacement is the version 2 observations one with a filter.

Scores: five types and three sources

The types are NUMERIC, CATEGORICAL, BOOLEAN, CORRECTION and TEXT (domain/scores.ts:45). The boolean requires a value of zero or one. The correction does not accept an associated config. The text one is capped at 500 characters.

The sources are API, EVAL and ANNOTATION. The EVAL source is reserved for the internal evaluators: the public API only accepts the other two. Annotation scores require an associated config except for the correction one.

On what costs money, a clarification that corrects a widespread belief: in 4.35.0 the entitlement for the number of model-based evaluators is unlimited on every plan, including the open edition (features/entitlements/constants/entitlements.ts:60). What is limited by plan is the number of annotation queues and the days of data access. And there is a version nuance: trace-level evaluators are discontinued in version 4.

Masking, and what it does not cover

This is the part to read twice if the deployment has to comply with anything.

The classic Python SDK function has the signature masking_function(*, data, **kwargs) and covers only the data that passes through the SDK calls: observation creation, update, and setting the trace input and output. It does not cover the raw OTel attributes produced by a third-party automatic instrumentation.

The replacement is a function that acts on the attributes and the resource attributes in the export batch. That one does reach automatic instrumentation, with a limitation declared in the documentation: it cannot change the span name, the identifiers, the parent relationship, the resource attributes, the events or the links.

And there is the gap that matters: the generative AI conventions from 1.37 onwards put messages and responses into span events. That is, the route by which prompts arrive in a modern instrumentation is precisely the one the new masking does not touch.

For a deployment with personal data, the conclusion is that client-side masking is not enough and a redaction processor has to go into the collector. It is the same reasoning that led in the previous article to recommending the collector in the middle, and it adds to the already documented problem of MCP tool arguments being written in clear, bypassing the message redaction switch.

On the server side there is no safety net: Langfuse has no server-side personal data masking. The encryption key that gets configured protects secrets, that is, model provider credentials, integration headers and SSO configurations. Trace inputs and outputs are not encrypted at application level.

Volume control

Three levers, in order of how high up they act:

In the SDK: sample_rate, or the LANGFUSE_SAMPLE_RATE variable, with head sampling at trace level. If the trace is not sampled, neither its observations nor its scores travel. LANGFUSE_TRACING_ENABLED set to false switches the whole thing off.

In the collector: probabilistic sampling and tail sampling. It is the only layer where you can decide based on the outcome, that is, keep every error and a fraction of the rest.

In the server: a deterministic per-project sampling, with a SHA-256 hash of the trace identifier (server/ingestion/sampling.ts). Deterministic per trace, so it does not cut traces in half.

What not to put into a trace

With the schema in front of you, the list is concrete.

Do not filter by tags or by status message. In the listings table the indexed fields are the span identifier, the trace identifier, the user identifier, the session identifier, the creation and update dates, the model name, the experiment identifier and the metadata names. The name, the tags, the version, the release and the status message are not. Filtering by them scans.

Do not use metadata as a store. It gets flattened into string arrays, truncated to two hundred characters in the listings table but stored whole in the full table, and on top of that it is indexed with a full-text index segmented on non-alphabetic characters. Bulky metadata multiplies the inverted index, not just the data.

Do not store full prompts without thinking about it. The input and the output carry ZSTD level 3 compression and two full-text indexes over their lowercase versions. Storing the entire context of a 142,000-token agentic call on every turn inflates the inverted index disproportionately. Client-side truncation is the cheap lever.

Do not use the environment to separate tenants. It is low cardinality, it has a 40-character cap and it isolates nothing: same project, same keys, same limits, same retention. It is there to segment dashboards. Real isolation means separate projects, as was already argued in the article on the seams.

Version 3 and the calendar date

The constant is in the code: V3_SUNSET_DATE = "2026-11-16" (features/public-api/server/deprecations.ts:11).

It is worth being precise about what it means, because the public documentation and the code do not say exactly the same thing. The code comment is literal: the classic ingestion route is never switched off and continues to accept score events; trace and observation events fail only in version 4’s exclusive write mode, not in the mixed or classic modes. And when they fail, they fail per event, not per batch.

The x-langfuse-ingestion-version: 4 header activates direct writing to the events table, which is the real-time route. Without it the deferred path is taken. Recent SDKs send it by themselves. Any value above four is rejected.

As for names, trace-level input and output attributes are deprecated in favour of the observation ones on the root span. The classic read endpoints carry a deprecation marker in the response body, and their replacements are version 2 of observations, version 3 of scores and version 2 of metrics. And dataset runs are renamed experiments.

There is a curious piece that deserves a mention because it shows up in the Redis dashboards and is misleading: a queue called version 4 classic API usage, which ingests nothing. It is a periodic task every fifteen minutes that scans the ClickHouse query log to detect which projects are still using the old API, and materialises the result into Redis for the interface’s migration notice.

How this looks in code

Two pieces, the client one and the collector one.

On the application side, what changes compared with the usual instrumentation is the observation type and where the identity comes from:

from langfuse import get_client, propagate_attributes

lf = get_client()

# identity propagates by context, not observation by observation
with propagate_attributes(
 session_id=id_conversacion, # the same across every turn
 user_id=sujeto_del_token, # from the claim, not from the body
 environment="produccion",
 tags=["soporte", "n2"],
):
 # the type matters: the interface groups by it
 with lf.start_as_current_observation(
 as_type="agent", name="resolver-incidencia"
 ) as raiz:

 with lf.start_as_current_observation(
 as_type="tool", name="buscar_en_inventario"
 ) as herramienta:
 resultado = mcp.call("buscar", args)
 # truncate here, not on the server
 herramienta.update(output=str(resultado)[:4000])

 # the score goes down its own channel, never over OTLP
 lf.score_current_trace(name="resuelta", value=1, data_type="BOOLEAN")

The as_type parameter accepts nine literals: span, generation, embedding, agent, tool, chain, retriever, evaluator and guardrail. The tenth type in the model, the event, has its own constructor because it does not open a scope.

On the collector side, the redaction processor that compensates for what the SDK masking does not reach:

processors:
 redaction:
 allow_all_keys: true
 blocked_values:
 - '\b\d{8}[A-HJ-NP-TV-Z]\b' # national ID number
 - '\b[\w.+-]+@[\w-]+\.[\w.]+\b' # email
 summary: silent
 tail_sampling:
 policies:
 - name: errores
 type: status_code
 status_code: {status_codes: [ERROR]}
 - name: resto
 type: probabilistic
 probabilistic: {sampling_percentage: 5}

exporters:
 otlphttp/langfuse:
 endpoint: http://langfuse-web.observabilidad.svc:3000/api/public/otel
 headers:
 Authorization: "Basic ${LANGFUSE_BASIC}"
 x-langfuse-ingestion-version: "4"
 sending_queue:
 enabled: true
 storage: file_storage
 retry_on_failure:
 enabled: true

The endpoint carries the base path without the signal suffix: the exporter adds its own. And the on-disk queue is the main reason for putting the collector in the middle, because the SDK’s batch processor drops when it fills up.

Instrumentation checklist

Review the type of every observation against the list of ten. Tool calls as TOOL, agents as AGENT, guardrails as GUARDRAIL.
Decide where the user identifier comes from and set it in a single place, high up the precedence chain.
Emit gen_ai.conversation.id if you want neutral instrumentation, or Langfuse’s own attribute if coupling does not matter.
Set the environment with deployment.environment.name, with few distinct values and never one per tenant.
Get scores out through their own channel, with the project keys, and do not expect them to travel with the trace.
Truncate input and output on the client before sending them, especially with long agentic contexts.
Put a redaction processor in the collector, because the SDK masking leaves out span events.
Do not filter by tags in dashboards that get queried often.
Send the ingestion version header, or use recent SDKs that send it by themselves.
Review the size of the metadata: there is no schema cap, the cap is the one on the request body.

Traps and things that are not what they look like

What you see in a listing is truncated to two hundred characters by the materialised view, and the full read has another cap of a thousand. The data is whole in the full events table.

Metadata has no size limit in the schema. A blob of megabytes passes validation.

An OTLP span over 9.5 MB is not rejected, it is only logged. There is no signal back to the client.

The environment is declared low cardinality. One value per tenant degrades ClickHouse.

Scores do not travel over OTLP. It is by absence in the code, not by configuration.

The classic masking does not cover automatic instrumentation, and the new one does not cover span events, which is where the recent conventions put the messages.

Langfuse does not encrypt trace inputs or outputs. The encryption key protects integration credentials.

In self-hosted there are no rate limits on the API. It is no use as protection.

The classic ingestion route is not switched off on 16 November, it changes behaviour depending on the write mode, and it fails per event instead of per batch.

Model-based evaluators are not limited by plan in this version, contrary to what is usually assumed. What is limited are the annotation queues and the days of data access.

The series: the nine articles

This is the first. The order is designed so that each one can be read on its own and so that together they cover the whole of day 2.

What goes into a trace (this article): version 4 data model, limits, precedences, scores, masking and indexes.
Putting LangGraph in front: instrumenting the most used open source agentic platform and measuring what one turn costs in observations and in bytes.
Migrating from version 3 to 4 without a window: the three steps of the write mode, the resumable background migrations and where the point of no return for the rollback sits.
The worker queues: the map of all thirty-nine, which pool to dedicate to each group, the per-queue switches, the partitioning and the concurrency.
ClickHouse capacity and real cost: how to measure the bytes per observation with the system tables, the difference between the full table and the listings table, and the merge cost of the full-text indexes.
Retention, deletion and data protection: why a deletion does not free disk, the mask cleaner that ships disabled, the pending deletions queue and the S3 lifecycle you have to implement by hand.
Backups and cross recovery: restore order between Postgres, ClickHouse and object storage, what each mismatch breaks, and how far event replay goes.
Saturation runbook: what to alert on from the queue metrics, the stall probes, draining through the readiness endpoint and the dead letter queue.
Getting the data out: the object storage to Parquet integration, batch exports and the metrics API, for building the data lake.

Sources

Código de Langfuse 4.35.0, commit 39e3cd7d del 11 de septiembre de 2026.
Anuncio de Langfuse v4, 17 de agosto de 2026.
Compatibilidad y migración, consultado el 12 de septiembre de 2026.
Sesiones, muestreo y enmascarado, consultados el 12 de septiembre de 2026.
Integración nativa con OpenTelemetry, consultado el 12 de septiembre de 2026.

The gateway does not live alone: the JWT that does not validate the audience, the user who never reaches the trace, and the seams you have to write by hand

Sat, 12 Sep 2026 06:30:00 +0200

Eighth article in the operational track of the control layer. The previous seven deal with the gateway from the inside: the pair with Langfuse, day 2, virtual keys, humans and agents, prefix routing, the MCP gateway and sizing for agents. This one deals with what surrounds it. Verified against LiteLLM 1.102.0 and Langfuse 4.35.0, both with a commit of 11 September 2026.

TL;DR

The gateway has five neighbours: a database, a cache, an identity provider, a trace backend and a collector. Each neighbourhood has its own physics, and these are the seven things it decides.

The proxy starts without Postgres, but then only the master key works. The DATABASE_URL variable is optional (proxy_server.py:1163); without it, any key other than the master one returns a 400 for a missing connection (user_api_key_auth.py:1886). With a database configured but down, the default behaviour is to return 503, unless general_settings.allow_requests_on_db_unavailable is switched on.

Redis is never mandatory, and that is precisely the trap. There is a circuit breaker with a threshold of five failures and a recovery of sixty seconds (constants.py:456). When it opens, the proxy fails no request: it degrades to local memory. What is lost silently is that rate limits stop being global and become per pod.

No observability backend can bring a request down. All callbacks are launched with asyncio.create_task and their exceptions are caught and logged (litellm_logging.py:3235). If Langfuse does not respond, events are discarded as the queue fills up. There is no added latency, and there is no reliable record either: the pair’s observability is a statistical figure, not an audit record.

JWT authentication validates neither the audience nor the issuer unless two variables are defined. In _build_decode_kwargs (handle_jwt.py:1003) the audience comes from JWT_AUDIENCE and the issuer from JWT_ISSUER; if they are empty, the code sets verify_aud=False and verify_iss=False and emits a warning once only. Any valid token signed by that provider, issued for another service, is accepted.

Role-based access control ships disabled. enforce_rbac has a default value of False (_types.py:4848 onwards). With that value, a token whose role resolves to nothing does not produce a 403: it carries on through the flow. Real authorisation, then, comes from team membership, if there is a team claim configured at all.

The user identifier that reaches Langfuse is not the one from Keycloak. The user_id field of the trace is filled in with user_api_key_end_user_id (integrations/langfuse/langfuse.py:604), that is, with the user field of the OpenAI body, which the client sets. The subject of the token and the owner of the key do not travel. Anyone who wants per-person traceability needs to configure end_user_id_jwt_field or make the client send that field.

Langfuse provisions nothing from a claim. There is no variable that maps an OIDC claim to an organisation or to a project. The only thing available is static assignment at sign-up with LANGFUSE_DEFAULT_ORG_ID and LANGFUSE_DEFAULT_PROJECT_ID, the same for everyone. On top of that, per-project access control sits behind the Enterprise licence in the self-hosted edition, as do audit logs and per-project retention.

You are here: between the four pieces

The LiteLLM and Langfuse pair walked through the seam between the gateway and the trace backend: integration routes, cost that arrives as zero, correlation and queues with discarding. This article widens the focus to the other three neighbourhoods, with identity at the centre, because that is the one that orders everything else and the one an auditor asks about.

The identity piece itself, that is, Keycloak, is dealt with in the next article. Here we deal with the gateway side.

The analogy: the building site with five trades

On a small building site, the problem is rarely inside each trade’s work, it is at the joins: where the plumber leaves the pipe sleeve and the bricklayer closes up the partition wall, where the electrician wants to run through the same gap. Each trade does its part well and the join ends up badly resolved because it belongs to nobody.

An inference platform has five trades. The gateway knows about tokens, the identity provider knows about people, the trace backend knows about events, the database knows about spend and the cache knows about counters. Each one has good documentation. The joins have nobody’s documentation, and that is where the user’s identity gets lost between the token and the trace.

There is one more detail of the analogy that is useful. On a building site, the part that causes the most problems is not the one that looks badly finished, it is the one that gets covered up. The same happens here: the five seams in this article fail silently.

Seam 1. What brings things down and what degrades them

It pays to have written down what happens when each neighbour falls over, because intuition fails in almost every case.

Neighbour	Does it start without it?	If it falls over
Postgres	Yes, master key only	503 by default; degraded with `allow_requests_on_db_unavailable`
Redis	Yes	Circuit breaker after 5 failures; limits become per pod
Langfuse	Yes	Events discarded as the queue fills; no impact on latency
OTel collector	Yes	Same as the previous one
Identity provider	Yes	Access to the interface and the JWT flow goes down; virtual keys carry on

Postgres

Without DATABASE_URL, _setup_prisma_client returns None without error (proxy_server.py:10375). The proxy serves /chat/completions with the master key. Any other key gets a 400. There is no global budget either, and start-up itself warns about it: Redis does not replace the database (proxy_server.py:9240).

With a database configured and down, the exception handler re-raises unless the degradation flag is active (db/exception_handler.py:56). There is also a DISABLE_PRISMA_HEALTH_CHECK_ON_STARTUP and a watcher that reconnects. The detail that matters for diagnosis: a transient database failure returns 503, not 401. If during an incident you see 401s, the problem is credentials; if you see 503s, it is the database or admission.

Redis

The circuit breaker is enabled by default, with a threshold of five failures, a recovery of sixty seconds and a minimum duration of five (constants.py:456, logic in caching/redis_cache.py:153). The socket has a timeout of 0.1 seconds (constants.py:453). With the circuit open, DualCache serves from memory without propagating an exception (dual_cache.py:215).

The consequence has to be measured, not assumed: with four workers and three replicas, a limit of one hundred requests per minute becomes one thousand two hundred. It is a failure mode that generates no errors and that can last for days without anyone noticing, until the bill arrives or the engine saturates.

The right alert is not about Redis, it is about litellm_service_latency labelled by service (integrations/prometheus_services.py:119), which exposes Redis and database latency separately.

The callbacks

Everything behind, that is, Langfuse and the collector, is launched outside the response path and its failures are caught. The classic integration uses the v2 SDK with its own background thread and a flush interval of one second, adjustable with LANGFUSE_FLUSH_INTERVAL (langfuse.py:198). There is a cap of fifty instantiated Langfuse clients (constants.py:554), because each client is a thread.

The OTLP integration uses the batch processor of the OpenTelemetry SDK (opentelemetry.py:3060) with its default values. In both cases, saturation means discarding, and discarding generates no error towards the client.

Seam 2. Identity: the three doors and what each one validates

Everything goes through a single authentication constructor (user_api_key_auth.py:1245), in this order:

Custom authentication from the enterprise edition.
Public routes.
Opaque OAuth2, if enabled, with a licence check.
JWT, if general_settings.enable_jwt_auth is active and the token has three parts.
Master key, with constant-time comparison.
Virtual key, with SHA-256 hash and lookup in the verification table.

A JWT works for /chat/completions, not just for administration: the routes allowed by default for a team include openai_routes (_types.py:4886), which contains the chat endpoint. Administration, on the other hand, is only reachable by a token with an administrator role.

The first thing to know: it is paid for

The JWT branch carries an explicit licence check, with the literal message that JWT authentication is an enterprise-only feature (user_api_key_auth.py:1416). SSO for the administration interface is free up to five billable users and requires a licence above that (ui_sso.py:976).

This conditions the architecture of any sovereign deployment that does not want to pay for a licence: the Keycloak integration is left for the administration interface and for provisioning, while inference traffic authenticates with virtual keys. Virtual keys are not worse, but they are a different thing, and it has to be said in the risk analysis: they are credentials decoupled from the identity provider, so removing a person from Keycloak does not invalidate their key.

The `litellm_jwtauth` fields that decide security

From the LiteLLM_JWTAuth class (_types.py:4848 onwards), these are the ones that matter, with their default values:

Field	Default	What it does
`enforce_rbac`	`False`	If false, an unresolved role does not produce a 403
`enforce_scope_based_access`	`False`	Without this, `scope_mappings` is not applied
`enforce_team_based_model_access`	`False`	Model access by team
`team_id_jwt_field`	`None`	Claim the team comes from
`team_id_upsert`	`False`	If false, a team that does not exist gives a 404
`user_id_upsert`	`False`	Same with the user
`public_key_ttl`	`600`	Cache of the public keys
`public_key_stale_ttl`	`3600`	Extra margin if the provider is down
`admin_jwt_scope`	`litellm_proxy_admin`	Scope that grants administration
`user_allowed_email_domain`	`None`	Restriction by email domain

The accepted algorithms are RS256/384/512, PS256/384/512, ES256/384/512 and EdDSA (handle_jwt.py:163). There is no symmetric one, which eliminates the algorithm confusion class of attack. It is a good design decision and it deserves recognition.

Key download supports several comma-separated URLs in JWT_PUBLIC_KEY_URL, resolves discovery if the URL points to the well-known OpenID configuration, retries three times with increasing waits and memoises the failure for thirty seconds (handle_jwt.py:680, :751, :88). If the provider is down, it serves the stale copy up to the sum of the two TTLs, that is, up to seventy minutes. For availability that is fine; for revocation it is not, and it has to be written down.

One detail of key rotation: with more than one key in the set, an exact match of the key identifier is required, but with a single key and no identifier it is accepted without checking (handle_jwt.py:913).

The validation that does not happen

This is the part to fix on day one. In _build_decode_kwargs (handle_jwt.py:1003), the audience is taken from the JWT_AUDIENCE environment variable and the issuer from JWT_ISSUER. If they are not defined, the code sets verify_aud=False and verify_iss=False, and emits a warning once only.

What that means in a real deployment with Keycloak: a token issued for the Grafana client, or for the Backstage one, signed by the same realm, is accepted by the gateway as a valid credential. Expiry is always validated, with zero margin. So is the signature. The audience is not.

There is a second route, issuers, with per-issuer configuration, which does validate issuer and audience unless explicitly disabled (handle_jwt.py:1186). That is the one to use when there is more than one provider.

The minimum fix is two lines of environment:

JWT_AUDIENCE=litellm-gateway
JWT_ISSUER=https://sso.ejemplo.es/realms/plataforma

And on the Keycloak side, an audience mapper in a dedicated client scope that injects that audience. The detail is in the Keycloak article, because the standard that would resolve this cleanly, resource indicators, is not supported.

From claim to permission

The administrator role comes from admin_jwt_scope appearing in the scope claim, and it returns a result without going through a team (handle_jwt.py:309, :1373). The rest is resolved by team.

If the team in the claim does not exist in the database, with team_id_upsert false a 404 is returned with the message that the team has to be created (auth_checks.py:3035). With the flag active, it is created by invoking team creation with a synthetic administrator identity (auth_checks.py:2874). That is the only automatic provisioning route that exists from Keycloak towards the gateway, and it works well. It pays to know that it creates teams with no budget and no limits, so a default value is needed in the configuration.

The vulnerability worth knowing about

CVE-2026-35030, scoring 9.1 on CVSS 3.1 and 9.4 on CVSS 4.0, advisory GHSA-jjhc-v7c2-5hh6. With JWT authentication active, the cache of the OIDC user info endpoint used the first twenty characters of the token as the key, so that two tokens with the same prefix were confused with each other. Fixed in 1.83.0.

In 1.102.0 the code uses the full SHA-256 hash (handle_jwt.py:962). A search for similar patterns across the tree finds no authentication cache indexed by token prefix: the truncations that remain are masked for logging. And it is worth recalling what has already been published about the MCP gateway: below 1.83.14 there is unauthenticated remote execution or SQL injection.

Seam 3. Identity does not reach the trace

Here lies the most expensive design flaw in the arrangement, because it is discovered at the first audit.

The user_id field of a Langfuse trace is filled in with user_api_key_end_user_id (integrations/langfuse/langfuse.py:604 and :724), which comes from the user field of the OpenAI request body or from end_user_id_jwt_field (litellm_pre_call_utils.py:1584). It is not the subject of the token nor the owner of the virtual key.

The knock-on consequences:

If the client does not send user, the trace has no user.
If the client does send user, the trace has whatever value the client cares to put there, unverified.
An agent calling with a service key leaves traces with no person behind them.

There are three ways to fix it, in order of robustness:

Configure end_user_id_jwt_field so that the end user comes from a token claim and not from the body. It is the only option in which the value is not controlled by the client, and it requires JWT authentication, that is, a licence.
Force the trace_user_id metadata field, which overwrites the field (langfuse.py:726; on the OTLP route, langfuse_otel.py:96). It can be set per team.
Contractually require the client to send user and validate it in a guardrail. It works, but it is a client declaration, not a proof.

Something similar happens towards the engine. The caller’s identity does not travel to vLLM unless litellm.add_user_information_to_llm_headers is switched on, whose default value is None (litellm/__init__.py:223). With it, x-litellm-* headers are injected with user identifier, team identifier and key hash (litellm_pre_call_utils.py:1356). Forwarding client headers requires general_settings.forward_client_headers_to_llm_api, and the authorization header is always filtered out, which is correct.

Towards MCP servers, by default the server’s own credential is used, not the user’s token. Only two modes forward the caller’s token, and doing so has a name in the MCP specification: token passthrough, which is explicitly forbidden. It comes up again in the Keycloak article.

Seam 4. Langfuse as a neighbour

What can be automated and what cannot

The exact list of variables for SSO in Langfuse 4.x, taken from web/src/env.mjs:

With the Keycloak provider: AUTH_KEYCLOAK_CLIENT_ID, AUTH_KEYCLOAK_CLIENT_SECRET, AUTH_KEYCLOAK_ISSUER, AUTH_KEYCLOAK_ALLOW_ACCOUNT_LINKING, AUTH_KEYCLOAK_CLIENT_AUTH_METHOD, AUTH_KEYCLOAK_CHECKS, AUTH_KEYCLOAK_ID_TOKEN_SIGNED_RESPONSE_ALG, AUTH_KEYCLOAK_SCOPE, AUTH_KEYCLOAK_ID_TOKEN, AUTH_KEYCLOAK_NAME.

With generic OIDC: the same set with the AUTH_CUSTOM_ prefix, plus AUTH_CUSTOM_FETCH_USERINFO and claim mapping with LANGFUSE_CUSTOM_SSO_SUB_CLAIM, LANGFUSE_CUSTOM_SSO_EMAIL_CLAIM, LANGFUSE_CUSTOM_SSO_NAME_CLAIM and LANGFUSE_CUSTOM_SSO_IMAGE_CLAIM.

Global ones: AUTH_DISABLE_USERNAME_PASSWORD, AUTH_DISABLE_SIGNUP, AUTH_DOMAINS_WITH_SSO_ENFORCEMENT, AUTH_SESSION_MAX_AGE.

And now what is missing. There is no variable that maps a claim to an organisation or a project. The only thing available is static assignment at sign-up with LANGFUSE_DEFAULT_ORG_ID, LANGFUSE_DEFAULT_ORG_ROLE with a default value of VIEWER, LANGFUSE_DEFAULT_PROJECT_ID and LANGFUSE_DEFAULT_PROJECT_ROLE (features/auth/lib/createProjectMembershipsOnSignup.ts:61). The documentation itself confirms it: enterprise SSO does not provision roles automatically at sign-up.

On top of that, in the per-edition permissions table (features/entitlements/constants/entitlements.ts:139), the open and self-hosted professional editions do not include per-project access control or the administration API. They belong to the enterprise edition, as do audit logs and per-project retention. In the open edition, every user inherits the role of their organisation.

The declarative bootstrap, which does exist

For on-premise without a licence, the workable route is bootstrapping through variables (web/src/initialize.ts), which is idempotent: LANGFUSE_INIT_ORG_ID (mandatory, if missing the rest is ignored with a warning), LANGFUSE_INIT_ORG_NAME, LANGFUSE_INIT_PROJECT_ID, LANGFUSE_INIT_PROJECT_NAME, LANGFUSE_INIT_PROJECT_RETENTION, LANGFUSE_INIT_PROJECT_PUBLIC_KEY, LANGFUSE_INIT_PROJECT_SECRET_KEY, LANGFUSE_INIT_USER_EMAIL, LANGFUSE_INIT_USER_NAME, LANGFUSE_INIT_USER_PASSWORD.

With that, one project per tenant is created from GitOps with one initialisation container per tenant and the keys in Kubernetes secrets.

Langfuse credentials per team

This part is well resolved on the gateway side and few people use it. TeamCallbackMetadata.callback_vars (_types.py:2152) accepts langfuse_public_key, langfuse_secret_key, langfuse_host and langfuse_environment, with an allow list in initialize_dynamic_callback_params.py:67. The gateway caches one logger per credential set (langfuse_handler.py:24) and, on the OTLP route, builds a full exporter per key (langfuse_otel.py:400).

That is to say: one Langfuse project per LiteLLM team is possible today, without a licence, and it is the correct way to isolate traces between tenants. What does not exist is automatic project creation, nor the correspondence between the team and the project. That is glue you have to write.

The ingestion route

Langfuse accepts only OTLP over HTTP, in http/protobuf or http/json. gRPC is not supported. The base route is /api/public/otel and the signal lives at /api/public/otel/v1/traces, verified in the repository. Authentication is Basic with the project’s key pair, plus the x-langfuse-ingestion-version: 4 header for the version 4 route.

The gateway builds it exactly like that (langfuse_otel.py:328, :347, :359) and then normalises the endpoint by adding the signal suffix (opentelemetry.py:3237). The result matches the real route, so the integration works without touching anything. The detail matters when a collector is put in the middle and somebody copies the endpoint by hand.

And the calendar warning that already appeared in the operational pair still stands: the classic callback is tied to the v2 SDK and the date marked is 16 November 2026. With a nuance worth reading before scheduling an emergency migration, which is detailed in the data model article: that date applies to Langfuse Cloud, and in the code of the self-hosted version the old route is not switched off, it changes behaviour according to the write mode.

When to put a collector in

Direct if Langfuse is the only trace destination. Via a collector if there is more than one consumer, or if any of these three things is needed that the proxy’s batch processor does not provide:

Retries with increasing waits and an on-disk queue, with the sending_queue and file_storage processors. With that, a Langfuse maintenance window stops losing traces.
Tail sampling, which lets you keep 100 % of errors and 1 % of the rest. The proxy cannot sample by outcome, because at the point it decides it does not yet know how things ended.
Attribute redaction before they leave the namespace. It is the only layer where you can delete the MCP tool arguments which, as has already been documented, are written in the clear, bypassing the message redaction switch.

Seam 5. Postgres, Redis and the memory policy

Separate Postgres, always. Each piece applies its own Prisma migrations over the public schema of the database it is given. Two Prisma clients migrating the same schema collide over table names and over migration history. On top of that, Langfuse 4.x requires Postgres 15 as a minimum, recommends 16 and wants UTC by default. The load profile is also the opposite: the gateway writes spend records at high frequency, while Langfuse does light transactional work because trace data goes to ClickHouse.

Separate Redis, and the argument is not the one it looks like. There is no key collision by design: the gateway uses hash tags of the form {api_key:...}:requests and an optional namespace, and Langfuse uses the BullMQ prefix plus its own prefix for its key cache. The decisive argument is another one: Langfuse requires maxmemory-policy=noeviction because its queues are data, not cache, while the gateway assumes a disposable cache. Sharing an instance means that a spike of rate limit keys can fill the memory and cause write errors in the ingestion queues instead of a harmless eviction.

If it is shared anyway, two minimum measures: REDIS_KEY_PREFIX in Langfuse, namespace in the gateway, and different logical databases.

And a reminder about surface area: Langfuse 4.x does not start without ClickHouse or without an S3-compatible bucket. CLICKHOUSE_URL and LANGFUSE_S3_EVENT_UPLOAD_BUCKET are mandatory and have no default value (packages/shared/src/env.ts:120 and :271). Its failure surface is far larger than the gateway’s, and that changes where to put the high availability effort.

Ports, for the NetworkPolicy

Piece	Port
LiteLLM (API, interface and metrics)	4000
Langfuse web	3000
Langfuse worker	3030
ClickHouse	8123 HTTP, 9000 native
S3-compatible storage	9000
Redis	6379
Postgres	5432
OTel collector	4317 gRPC, 4318 HTTP

The pairs to allow under a default-deny policy, following the method from hardening the stack:

litellm → postgres:5432, redis:6379
litellm → collector:4318 (or langfuse-web:3000 if going direct)
litellm → vllm:8000
litellm → keycloak:8443 (interface and JWT flow only)
collector → langfuse-web:3000
langfuse-web → postgres:5432, redis:6379, clickhouse:8123, s3:9000
langfuse-worker→ postgres:5432, redis:6379, clickhouse:8123, s3:9000
langfuse-web → keycloak:8443
ingress → litellm:4000, langfuse-web:3000
prometheus → litellm:4000

The Langfuse worker needs no inbound traffic beyond the health probe.

The glue you have to write

Put into a table, the real state of the tenant chain:

Segment	Automatic?	How
Keycloak → LiteLLM team	Yes	`team_ids_jwt_field` plus `team_id_upsert: true`
Keycloak → LiteLLM user	Yes	`user_id_jwt_field` plus `user_id_upsert: true`
Keycloak → Langfuse organisation	No	Static assignment at sign-up only
Keycloak → Langfuse project	No	Same
LiteLLM team → Langfuse project	No	`callback_vars` by hand, or an operator of your own
Keycloak user → trace `user_id`	No	`end_user_id_jwt_field`, or a contract with the client

The minimum viable architecture without a licence, then, is this:

One Langfuse project per tenant, created with the initialisation variables from GitOps.
That project’s keys in a Kubernetes secret.
A periodic job of your own that reads the teams through the gateway’s administration API and writes callback_vars with the corresponding keys.
People entering Langfuse through Keycloak, but assigned by hand to their organisation.
The end user identifier resolved by contract with the clients, and validated.

That is about one hundred and fifty lines of your own code. It pays to budget for them from the start, because the gap does not close by itself.

Reference configuration

general_settings:
 # only with a licence; without one, virtual keys
 enable_jwt_auth: true
 litellm_jwtauth:
 team_ids_jwt_field: "groups"
 team_id_upsert: true
 user_id_jwt_field: "sub"
 user_id_upsert: true
 end_user_id_jwt_field: "preferred_username"
 user_allowed_email_domain: "ejemplo.es"
 admin_jwt_scope: "litellm_proxy_admin"
 # the three that ship switched off
 enforce_rbac: true
 enforce_scope_based_access: true
 enforce_team_based_model_access: true
 allow_requests_on_db_unavailable: false
 proxy_batch_write_at: 60

litellm_settings:
 callbacks: ["langfuse_otel", "prometheus"]
 # without this, the engine does not know who is calling
 add_user_information_to_llm_headers: true
 turn_off_message_logging: true
 cache: true
 cache_params:
 type: redis
 host: os.environ/REDIS_HOST
 namespace: litellm

# the two that enable the validation that by default does not happen
JWT_AUDIENCE=litellm-gateway
JWT_ISSUER=https://sso.ejemplo.es/realms/plataforma
JWT_PUBLIC_KEY_URL=https://sso.ejemplo.es/realms/plataforma/.well-known/openid-configuration

LANGFUSE_HOST=http://langfuse-web.observabilidad.svc:3000
LANGFUSE_PUBLIC_KEY=...
LANGFUSE_SECRET_KEY=...

And on the Langfuse side:

AUTH_KEYCLOAK_CLIENT_ID=langfuse
AUTH_KEYCLOAK_ISSUER=https://sso.ejemplo.es/realms/plataforma
AUTH_KEYCLOAK_ALLOW_ACCOUNT_LINKING=true
AUTH_DISABLE_USERNAME_PASSWORD=true
AUTH_DISABLE_SIGNUP=true
LANGFUSE_DEFAULT_ORG_ID=plataforma
LANGFUSE_DEFAULT_ORG_ROLE=VIEWER
ENCRYPTION_KEY=... # 64 hexadecimal characters

Checklist

Define JWT_AUDIENCE and JWT_ISSUER on day one, or use the per-issuer configuration. Without that, neither audience nor issuer is validated.
Set enforce_rbac, enforce_scope_based_access and enforce_team_based_model_access to true, and test that a token from another client of the same realm is rejected.
Decide where the end user of the traces comes from and write it into the design. If it is not decided, traces come out with no person.
Separate Postgres. Separate Redis. If Redis is shared, different prefix and logical database, and review the memory policy.
Check that the proxy starts with Postgres down only if that is what you want, and that the on-call team can tell an admission 503 from a credential 401.
Alert on litellm_service_latency per service, not on Redis availability, because the circuit breaker hides the problem.
One Langfuse project per tenant with the initialisation variables, and the per-team keys in callback_vars.
Put a collector in the middle if you need an on-disk queue, tail sampling or attribute redaction.
Check that the gateway version is above 1.83.14 because of the earlier critical vulnerabilities.
Write into the risk analysis that virtual keys are credentials decoupled from the identity provider, and define the offboarding procedure.

Traps and things that are not what they seem

A properly signed JWT is not a JWT addressed to this service. Without the two environment variables, the audience is not checked.

enforce_rbac set to false does not mean there is no authorisation, it means that authorisation comes only from team membership, if there is a team claim. With no team claim and without the flag, a valid token reaches /chat/completions.

The stale copy of the public keys lasts up to seventy minutes, adding the two TTLs. Revoking a key at the provider does not take effect immediately.

With a single key in the JWKS and no key identifier, the identifier is not checked.

The trace user is set by the client. Any per-person spend report built on that field is a declaration, not a measurement.

Redis going down generates no errors, it generates per-pod limits. It is the most expensive of the failures that give no warning.

Langfuse does not start without ClickHouse or without an S3 bucket. Its failure surface is bigger than the gateway’s, and that changes how the high availability effort is shared out.

Langfuse’s per-project access control is paid for in self-hosted, as are audit logs. For an ENS file, that gets decided before building, not after.

Sharing Redis does not break through key collision, it breaks through memory policy. It is a failure that shows up under load and gets diagnosed badly.

Langfuse’s ingestion route is OTLP over HTTP only. A collector configured with the gRPC exporter points at somewhere that does not exist.

Closing

The five seams share a common pattern: they fail without warning. The token from another service gets in, the trace user comes from the client, the rate limit stops being global, the spend queue discards on reaching 64 MB and the Langfuse project does not create itself. None of those five things produces a visible error.

Day 2 work, then, consists above all of turning silences into signals. Two environment variables so that the audience is validated, three flags so that authorisation is real, an alert on service latency instead of on Redis availability, and an explicit contract on where the end user’s identity comes from.

And one decision worth taking early, before the budget is closed. The gateway’s JWT authentication and Langfuse’s per-project access control are both behind a licence. A sovereign platform can live perfectly well without either, with virtual keys and one project per tenant, but it is a different architecture and it has to be drawn as such from the start.

Sources

Código de LiteLLM 1.102.0, commit 9071ca50 del 11 de septiembre de 2026.
Código de Langfuse 4.35.0, commit 39e3cd7d del 11 de septiembre de 2026.
Autenticación y SSO en Langfuse, consultado el 12 de septiembre de 2026.
SCIM y API de organización, consultado el 12 de septiembre de 2026.
Integración OpenTelemetry de Langfuse, consultado el 12 de septiembre de 2026.
Postgres y caché en la documentación de self-hosting, consultados el 12 de septiembre de 2026.
Autenticación por token en LiteLLM y ajustes de configuración, consultados el 12 de septiembre de 2026.
GHSA-jjhc-v7c2-5hh6 y registro de CVE-2026-35030 en NVD.

Sizing for agents: 275 input tokens for every output token, and why the bottleneck is still decode

Sat, 12 Sep 2026 06:00:00 +0200

Seventh article in the operational track of the control layer. The fourth dealt with the coexistence of human traffic and agentic traffic on a fixed fleet. This one deals with the prior question: how big the fleet has to be, and how big the gateway in front of it. Verified against LiteLLM 1.102.0 (commit of 11 September 2026) and vLLM 0.29.0, with load data published between May and September 2026.

TL;DR

Until this year, sizing for agents meant extrapolating a chat profile and adding margin. In 2026 the first production-scale characterisations were published and the profile turned out to be something else. These are the seven conclusions.

The median prompt of an agentic call is 68,000 tokens, of which 63,000 arrive cached. The measurement covers 13.5 million sessions, 3.2 million users, 760.5 million model calls and 44.9 trillion prompt tokens in one week of June 2026. The median input-to-output ratio exceeds 275 to 1. Any calculation done on a profile of 800 prompt tokens and 250 output tokens describes a different system.

The load is prefill-dominant when counting tokens and decode-dominant when counting time. The genuinely new tokens that have to be prefilled on each turn are only between 1.5 and 7.3 times the output, not 275. With the prefix cache working, the split of execution time runs from 91 % to 98.6 % in decode. You size by memory bandwidth and by KV cache capacity, not by prefill FLOPs.

The sizing risk is not the average, it is the collapse of the prefix cache. The median hit rate within a turn is 98 %, it falls to 55 % at the boundary between turns and to 8 % when the client changes model. With dynamically retrieved context, prefix reuse drops to 1 %. One miss turns 142,000 prefix tokens into real prefill. Session affinity per replica stops being an optimisation and becomes a capacity requirement.

The intra-turn concurrency measured in production is 1.15 at the median and 1.4 at the 90th percentile. 93 % of tool batches contain a single invocation. The image of the agent opening twenty calls in parallel does not appear in the data. The real multiplier comes from the duty cycle and from the fact that 87 % of calls are started by the agent, not by the person.

The gateway is sized per worker, and each worker is a universe. One vCPU and 4 GiB per worker, with requests equal to limits. The limit counters, the caches and the cooldowns live in the worker’s memory unless there is Redis, so a limit of 100 requests per minute with four workers and three replicas allows 1,200. The Prisma pool is also per process: the number of connections open against Postgres is workers times connection_limit.

The CPU cost that grows with agentic load is the token counter. It serialises the entire tool catalogue on every request and tokenises it. The project itself measured 46, 53 and 100 ms for prompts of 50K, 75K and 100K tokens, and the operation passes through a concurrency limiter of four per worker. With agents sending long catalogues and six-figure contexts, that limit is the ceiling before the CPU is.

Autoscaling by memory does not work on this piece and the startup probe takes five minutes. The official chart deliberately leaves memory without a target because Prisma’s query engine never returns the maximum reached. That leaves CPU at 60 %, with a startupProbe of 300 seconds that makes any new replica arrive late. For agents there are two better metrics, and they are in /metrics.

You are here: the arithmetic that comes before buying

Classic capacity planning starts from a chat SLO and crosses two budgets, the VRAM one and the time one. That method is still correct. What changes with agents are the inputs: the prompt length, the number of calls per unit of human work, the split between prefill and decode, and the very definition of an SLO. This article replaces those inputs with measured figures and drags the result through to two numbers: gateway replicas and GPUs.

Out of scope is the splitting of traffic once it is built, which was already covered by the coexistence of humans and agents and prefix routing.

The analogy: the repair shop and the front desk

An appliance repair shop sizes its staff by counting faults, not phone calls. The difference matters because the person who brings in the washing machine occupies the front desk for three minutes, while the technician who takes it apart goes back to consult the machine’s data sheet fourteen times over the afternoon. If the shop manager sizes by visits to the front desk, they end up short of technicians and overstaffed at reception.

The inference platform has the same asymmetry. The person sends one turn. The agent sends fifteen calls for that turn, and each one drags along the complete file of everything done so far. The front desk is the gateway and the technicians are the GPUs, and the usual mistake is to size both with the same counter.

There is a second detail the analogy captures well. The technician does not read the whole data sheet each time: they have it open on the bench. That is the prefix cache. The day somebody moves them to a different bench between consultations, they have to fetch it from the archive again, and the work multiplies without the number of faults having increased.

Part 1. The real load profile

For two years, any capacity calculation for agents was done on assumptions. In 2026 three pieces of work appeared that measure the matter, and it is worth reading them together because they measure different scaffolds and their figures are not averageable.

The three sources

The characterisation at scale was published by a research team working on GitHub Copilot production telemetry (arXiv 2608.00101, 30 July 2026): 13.5 million sessions, 3.2 million users, 95.1 million user turns, 760.5 million model calls and 44.9 trillion prompt tokens, in one week of June 2026. It is the largest sample published to date.

The characterisation by benchmark suites comes from an architecture team (arXiv 2605.26297, 25 May 2026), which instruments five of them (ADE-Bench, DABStep, GAIA, SWE-bench Pro and Terminal-Bench 2.0) and measures the internal split of time.

The production measurement on a different scaffold was published by the vLLM project itself together with AgentX (vLLM blog, 8 September 2026).

The three disagree on turns per session: 15 calls at Copilot’s median, 43 turns in AgentX, 62.4 in SWE-bench Pro with Qwen. It is not a contradiction, they are scaffolds with different loops. The sizing lesson is that the number of calls per unit of human work depends on the client that connects, and that is why you have to measure it at home before fixing the size.

The figures you can actually use

Quantity	Measured value	Source
Median prompt per call	68K tokens, 63K of them cached	2608.00101
Input:output ratio (median)	above 275:1	2608.00101
Median session	3 user turns, 15 calls, 4.2 min	2608.00101
Mean session	6.1 turns, 40.6 calls, 62.6 min	2608.00101
Calls per turn (median)	4.5	2608.00101
Calls started by the agent	87 %	2608.00101
Range across user profiles	50× (23K to 1.1M tokens per turn)	2608.00101
Median input context	142K tokens; output 444 tokens per turn	AgentX
Accumulated context in SWE-bench Pro	mean 68.7 to 80.1K, maximum 146 to 166K	2605.26297
Reasoning tokens over the output	29.0 % to 67.6 %	2605.26297

Two operational observations come out of that and appear in no sizing guide.

The first: agents that fail accumulate up to 1.8 times more average context than those that finish well (2605.26297). The cost does not correlate with success, it correlates with difficulty and with getting stuck. A sizing done on the task that goes well falls short.

The second: the range across user profiles is 50 to 1. A user who only converses spends 23,000 tokens per turn; one who leaves the agent in a long loop spends 1.1 million. Segmenting by usage profile pays off more than any difficulty-based routing, because the variance is in the people, not in the requests.

What to measure before calculating

Three of your own figures, taken from a week of real traffic, are worth more than any table in this article:

Model calls per hour of human work, by user profile.
Prompt length distribution (median and 95th percentile), not the mean.
Aggregate prefix cache hit rate, with the time series, not the single number.

All three come from what the gateway already stores. In LiteLLM_SpendLogs you have the tokens per request and the key and team identifiers; in proxy_server_request you have the tools block of each call; and in vLLM’s /metrics you have vllm:prefix_cache_queries and vllm:prefix_cache_hits.

Part 2. Prefill, decode and the cache trap

Here is the finding that corrects almost everyone’s intuition, including mine when I started this article.

Counting tokens, agentic load is overwhelmingly input: 275 to 1. The apparently obvious conclusion would be to size by prefill capacity, that is by FLOPs. It is false.

What has to be prefilled on each turn is not the whole prompt, but what was added since the previous turn. That addition, measured, sits between 1.5 and 7.3 times the output length (2605.26297). The rest is served by the prefix cache. And the real split of the model’s execution time comes out like this: between 91.0 % and 98.6 % in decode, between 1.4 % and 9.0 % in prefill (2605.26297).

The hit rate that sustains that split is high and has been measured in three places: theoretical from 87.9 % to 99.3 % and empirical from 84.6 % to 99.5 % (2605.26297); above 96 % in production (AgentX); and a median of 98 % within a turn in Copilot (2608.00101).

Where it falls over

The same Copilot work measures the degradation, and it is the most actionable figure in the article:

98 % median hit rate within a turn. The detail matters: the first call of a turn hits around 45 %, the second rises to 86 % and from the third onwards it stabilises between 92 % and 94 %.
55 % at the boundary between turns, with idle gaps of 4.1 minutes at the median.
8 % when the client changes model.

And there is a fourth, worse regime: with dynamically retrieved context, that is when the scaffold injects search or RAG results that change on every iteration, prefix reuse falls to 1 % or less (arXiv 2608.15127, 15 August 2026).

A cache miss with a context of 142,000 tokens turns a prefill of a few thousand tokens into a six-figure one. The work multiplies by a factor of between 20 and 60 without the number of requests having changed. That is the difference between a fleet that meets its targets and one that seizes up mid-afternoon.

Two sizing rules come from that:

Without session affinity towards the same replica, the calculated sizing is worthless. LiteLLM’s concrete mechanism is session_affinity, which is not active by default and which was already covered in the prefix routing article. Without it, the calculation has to be redone assuming a prefill regime, which is between 20 and 60 times more expensive.
Changing model halfway through a session costs the entire cache. Any difficulty-based routing policy that moves a live conversation from one model group to another pays an 8 % hit rate on the next call. If you want to route by difficulty, do it at the start of the session.

Co-batching, which is the other effect nobody measures

Mixing a short request into the same batch as long-context requests degrades the short one’s inter-token latency. The measured figure: one long request in the batch raises time per token by 38.3 %; two raise it by 79.7 % (2608.15127). It is the technical reason why separating human and agent pools improves the human experience more than the capacity split suggests, and it is also the reason not to raise --max-num-seqs above what the KV budget allows.

Part 3. The KV budget, with the numbers of the agentic case

The formula does not change with respect to classic capacity planning, but the inputs do, and the result is surprising:

$$\text{KV}_{\text{bytes/token}} = 2 \times n_{\text{layers}} \times n_{\text{kv heads}} \times d_{\text{head}} \times \text{bytes}_{\text{elem}}$$

For a 70B-class geometry (80 layers, 8 KV heads with grouped attention, head dimension 128, 2-byte elements) that gives 0.3125 MiB per token. With the median agentic context of 142,000 tokens:

$$142{,}000 \times 0.3125\ \text{MiB} \approx 43.3\ \text{GiB per sequence}$$

For a 30B-class geometry (48 layers, everything else the same) that gives 0.1875 MiB per token, that is around 26.0 GiB per sequence.

It is worth pausing on what that means. An 80 GB H100 with the weights of a 70B model in FP8 already taking up space does not have 43 GiB free for a single sequence. The conclusion is harsh and has to be written down: serving real 70B-class agentic context demands tensor parallelism of at least 2 and a quantised KV cache. And the number of simultaneous sequences that fit, which is what sets --max-num-seqs, comes from dividing the available KV by that figure, not from a chat heuristic.

$$\text{sequences} \approx \frac{\text{VRAM}_{\text{KV available}}}{\text{KV}_{\text{bytes/token}} \times \text{max context}}$$

With FP8 in the KV, the divisor halves. It is the cheapest capacity lever there is in this load profile, and its cost in quality is validated with evals, not assumed.

Part 4. From people to concurrent requests

This is where most calculations overshoot, because it is assumed that an agent equals many simultaneous requests. The data says otherwise.

The concurrency of calls within a turn, measured in production, is 1.15 at the median and 1.4 at the 90th percentile, and 93 % of tool batches contain a single invocation (2608.00101). Intra-turn parallelism is small. Subagents appear in 44 % of AgentX sessions, with a median of 4 deployments among those that use them, so there is a long tail, but it is not the usual regime.

What does multiply is the duty cycle. The mean session lasts 62.6 minutes and contains 40.6 calls. At 444 output tokens and an interactivity of around 60 tokens per second, each call occupies around 7.4 seconds of decode, which gives between 5 and 7 active minutes out of those 62.6.

$$C = D \times \rho \times A$$

where $D$ is the people working, $\rho$ the duty cycle and $A$ the amplification. With $\rho \approx 0.10$ to $0.12$ derived from the previous figures, twenty developers give between 2 and 2.4 concurrent sequences in the steady state. It is the peak, not the average, that has to be sized for.

The amplification

Three factors published separately:

Tool failures: 9 % of turns trigger them, with a compute amplification of 4 times (2608.00101).
Subagents: 44 % of sessions, median of 4 deployments (AgentX).
Internal variance: the same task from the same agent varies by up to 30 times in cost within a single trace (2608.15127).

The product of the first two gives $(1 + 0.09 \times 3) \times (1 + 0.44 \times 3) \approx 2.9$. Nobody publishes that product as a single figure, so it is flagged as my own derivation, but an amplification factor of 3 is the reasonable design provision and it fits the measured variance.

For the twenty developers in the example: between 6 and 7 concurrent design sequences, with 142K contexts. At 43.3 GiB of KV per sequence in the 70B class, that is around 300 GiB of KV cache, that is four H100s dedicated to KV alone with the weights on top. With KV in FP8, half. With a 30B-class model, a third.

That calculation, done with the chat profile of 800 prompt tokens, would have given a single GPU. That is the distance between the two regimes.

Part 5. The gateway, which also has to be sized

The proxy consumes no GPU, but it has its own arithmetic and it is the part that produces the most surprises in production, because almost all of its state is per process.

A worker is a universe

The default value of --num_workers is 1 (litellm/constants.py:57). The usual startup is uvicorn directly with uvloop (proxy_cli.py:1477), and there are alternatives with gunicorn, hypercorn and granian.

What lives in each worker’s memory, and therefore multiplies:

The in-memory cache, limited to 200 elements with a TTL of 600 seconds (litellm/caching/in_memory_cache.py:27). With many deployments, the cooldowns and the counters compete for that same space and get evicted before they expire.
The key authentication cache, with a TTL of 60 seconds (proxy_server.py:1650), adjustable with general_settings.user_api_key_cache_ttl.
The limit and budget counters when there is no Redis (parallel_request_limiter_v3.py, local_only=True paths).
The router cooldowns, with DEFAULT_COOLDOWN_TIME_SECONDS = 5 (constants.py:75).
A periodic task scheduler of its own per worker (proxy_server.py:9674), coordinated between pods with a lock in Redis.

The practical consequence: a limit of 100 requests per minute with four workers and three replicas allows up to 1,200 without Redis. With Redis, the v3 limiter registers seven Lua scripts at startup (parallel_request_limiter_v3.py:608) and the limits become global.

One detail of the v3 limiter that matters with agentic load: LITELLM_TPM_TOKEN_RESERVATION_ENABLED comes enabled by default and reserves estimated tokens before the call, which adds a round trip to Redis per request. And there is an asyncio.Lock per instance that serialises the atomic batch checks (:665), documented by the code itself as a 99th-percentile risk under contention.

If Redis goes down, requests do not fail: there is a circuit breaker with a threshold of 5 failures and a recovery of 60 seconds (constants.py:456), and the proxy degrades to local memory. What is lost silently is the globality of the limits.

Postgres

The pool is also per process. The default values are database_connection_pool_limit = 10 and database_connection_pool_timeout = 60 (proxy_cli.py:62), injected as parameters of the Prisma URL. A pod with four workers opens forty connections. Three pods, one hundred and twenty. It is the fast route to exhausting Postgres’s max_connections without having served any traffic, and the reason the official chart offers PgBouncer inside the pod itself with maxDbConnections: 20 and maxClientConn: 1000.

On the request path there is only the authentication, which is cached for 60 seconds, and the rate limiting. Spend is deferred. The SpendLogs queue is capped at 64 MB and on overflow it discards the oldest elements with an error in the log (utils.py:6704). The write interval is proxy_batch_write_at, with a default value of 10 seconds plus a random spread of up to 5 (constants.py:1719, proxy_server.py:9704); the production guide recommends raising it to 60.

With a slow Postgres, the queue grows to those 64 MB and then loses billing data without returning any error to the client. It is the silent failure mode you have to alert on.

The token counter, which is the agentic trap

This is the piece that turns a correct sizing into an insufficient one, and it appears in no guide.

Before admitting a request, the proxy counts tokens. With chat load the cost is negligible. With agentic load it is not, for two reasons that add up.

The first is the prompt length. The project itself measured the admission counting cost at 46, 53 and 100 ms for prompts of 50K, 75K and 100K tokens (benchmarks, consulted on 12 September 2026), and published that with a counter written in Rust those figures fall to 4.9, 6.8 and 10.2 ms.

The second is the tool catalogue. The function that prepares the definitions serialises the entire catalogue to a pseudo-TypeScript and tokenises it (litellm_core_utils/token_counter.py:934), plus nine fixed tokens. The cost is linear in the size of the catalogue and it is paid on every request. That is exactly the pattern of an agent connected to an MCP gateway: a long catalogue, resent on every turn. The token bill of that catalogue was already covered in the MCP gateway article; here its other face appears, which is proxy CPU.

And on top of that there is an explicit ceiling. The counting runs in a thread with a capacity limiter whose default value is 4 per worker (token_counter.py:340, constants.py:407). With 100K prompts, four concurrent counts of 100 ms mean a theoretical ceiling of around 40 admissions per second per worker before the limiter starts queueing, and that is without having called the model yet.

What is cached is the tokeniser, with an LRU of 64 entries (utils.py:2206), not the result of the count.

Three levers, in order of return: raise TOKEN_COUNTER_MAX_CONCURRENT_COUNTS if CPU is spare, reduce the tool catalogue exposed per key, and watch the gateway overhead figure in /metrics to know whether this is happening.

Streaming and admission

There is no connection limit by default. --limit_concurrency exists but is optional, and it returns 503. The built-in admission control is only activated if general_settings.max_in_flight_requests_per_worker is defined (middleware/admission_control_middleware.py:285), with max_queued_requests_per_worker and admission_queue_timeout_seconds at 1.0 by default, rejecting with 503 and a retry-after: 1 header.

For agents, that header matters: an agentic client that retries without respecting it multiplies the load exactly when the gateway is saying it cannot take any more. The combination of gateway retries was already calculated by the fourth article and it reached 45 calls per turn in the worst case.

The default request timeout is 6,000 seconds (constants.py:512). The production guide recommends 600. With long agentic tasks, leaving it at 6,000 means a hung connection occupies a slot for an hour and a half.

Resources and autoscaling

The official chart recommends 1 vCPU and 4 GiB per worker, with requests equal to limits (helm/litellm-helm/values.yaml:235), and the production documentation suggests one worker per pod scaling horizontally. The project’s own high-performance benchmark, by contrast, uses four workers per pod with PgBouncer in front.

The HPA comes with targetCPUUtilizationPercentage: 60. And there is a comment in the chart worth reading twice: memory is deliberately left without a target because the resident consumption of Prisma’s query engine is a high-water mark that is never returned, so that an HPA on memory would add replicas and never remove them (values.yaml:260).

The other problem is startup. The startupProbe has failureThreshold: 30 with a period of 10 seconds, that is up to 300 seconds before the pod is considered ready (values.yaml:109). An HPA that reacts at 60 % CPU and takes five minutes to add capacity arrives late to any agentic burst.

The alternative, optional in the chart, is to scale by Prometheus metrics: targetRequestsPerSecond and targetTokensPerSecond over rate(litellm_proxy_total_requests_metric_total[1m]) and rate(litellm_total_tokens_metric_total[1m]). The chart itself warns that the token metric lags with long streaming responses, because tokens are accounted for on completion. For agents, that is almost always.

Part 6. What to watch: the metrics that work and the ones that broke

At the gateway

Metric	What for
`litellm_in_flight_requests`	Real saturation per worker. It is the scaling metric
`litellm_overhead_latency_metric`	Proxy cost per request; it rises with token counting
`litellm_overhead_with_guardrails_latency_metric`	The same including guardrails
`litellm_request_queue_time_seconds`	Arrival through to pre-call, includes authentication and ASGI queueing
`litellm_deployment_cooled_down`	Replicas out of the pool
`litellm_service_latency`	Redis and database latency, per service

The /health/backlog endpoint returns in_flight_requests, admitted_requests, queued_requests and rejected_requests (health_endpoints/_health_endpoints.py:1871).

What it does not expose: the depth of the SpendLogs queue or its bytes, the Prisma connections in use against the limit, nor the depth of the logging worker’s queue. All three are real failure modes and have to be instrumented from outside.

At the engine

Names verified in the vLLM 0.29.0 metrics documentation:

vllm:num_requests_running, vllm:num_requests_waiting
vllm:time_to_first_token_seconds, vllm:e2e_request_latency_seconds, vllm:request_queue_time_seconds, vllm:inter_token_latency_seconds
vllm:prefix_cache_queries, vllm:prefix_cache_hits, vllm:kv_cache_usage_perc

And the ones that break dashboards on upgrade:

vllm:time_in_queue_requests is duplicated by vllm:request_queue_time_seconds.
The direct prefix cache hit rate indicator disappeared. It is now computed as rate(vllm:prefix_cache_hits) / rate(vllm:prefix_cache_queries).
vllm:num_requests_swapped and vllm:cpu_cache_usage_perc are deprecated along with CPU swapping.
vllm:gpu_cache_usage_perc became vllm:kv_cache_usage_perc, which also breaks any HPA that uses it.

The SLO worth chasing

A 95th percentile of time to first token describes nobody’s experience when the unit of work is 43 turns. Coding tasks frequently exceed ten minutes and research tasks reach hours (2608.15127).

Two indicators serve better:

90th percentile of full-task latency, measured per session, not per request. It comes from grouping LiteLLM_SpendLogs by session identifier.
90th percentile of interactivity, in tokens per second per user. It is the one AgentX uses, with values between 58.3 and 74.2 in its runs.

To autoscale the engine, the official production-stack example uses vllm:num_requests_waiting with a threshold of 5, polling every 15 seconds and a cooldown of 360. With agentic load it is worth adding vllm:kv_cache_usage_perc as a second trigger, because with 26 to 43 GiB of KV per sequence the memory pressure arrives before the queue does.

Part 7. The engine levers, ordered by return

Verified against vLLM 0.29.0, released on 9 September 2026.

--long-prefill-token-threshold 512. The AgentX measurement attributes to it +93 % in tokens per GPU per second, with a 2.3-times improvement at the 90th percentile, by attacking the head-of-line blocking that a long prefill produces in front of short requests. It is the highest-return lever documented for this profile.

KV cache in FP8. It halves the memory budget per sequence, which is the limiting resource. Validate with evals.

--max-num-queued-reqs and --max-num-queued-tokens. New in 0.29. They are admission control in the engine, and they are the right mechanism to protect the SLO instead of letting the queue grow without limit.

max_num_batched_tokens downwards. High values improve time to first token; low values improve inter-token latency. In a regime where 91 % of the time is decode, inter-token latency is what is perceived. Chunked prefill is enabled by default whenever possible in the V1 engine.

Session affinity at the gateway. It is not an engine setting, but it is what holds up everything above. Without it, that 98 % hit rate turns into random distribution across replicas.

Two version warnings: Model Runner V2 becomes the default in 0.29 and V1 is deprecated with removal planned for 0.32; and starting with python -m vllm.entrypoints.openai.api_server is deprecated.

The full calculation, from start to finish

Case: twenty developers, a coding scaffold, a 30B-class model in the fleet, maximum context of 128K, KV in FP8.

1. Steady-state concurrency. $C = 20 \times 0.11 \times 3 \approx 6.6$ sequences.

2. KV budget per sequence. 0.1875 MiB per token in BF16, half in FP8, that is around 0.094 MiB. At 128K of context: around 12.0 GiB per sequence. For 6.6 sequences with margin, rounding to 8: 96 GiB of KV.

3. GPUs. The weights of a 30B in FP8 take up around 30 GB. On an 80 GB H100 with gpu_memory_utilization of 0.90 there are around 72 GB usable per card. With tensor parallelism of 2, the weights are split: around 15 GB per GPU, so around 57 GB of KV per GPU and around 114 GB in the replica. That covers the 96 GiB calculated, and TP=2 also adds memory bandwidth, which is the resource that dominates decode. Two H100s, one replica, TP=2. It is worth seeing the contrast: on a single H100 the full weights leave around 42 GB of KV, that is three and a half sequences, less than half of what is needed. With a second replica for availability and maintenance windows: four GPUs.

4. Requests per second at the gateway. 20 people, a mean session of 40.6 calls in 62.6 minutes, duty cycle included: of the order of 0.2 to 0.3 requests per second on average, with peaks of 2 to 3.

5. Gateway workers. The ceiling is not set by throughput but by token counting: four concurrent counts per worker, at 100 ms with long prompts. With peaks of 3 requests per second and six-figure prompts, two replicas of one worker each cover the case with margin for the rolling update, at 1 vCPU and 4 GiB each. Postgres will see 20 connections.

6. What needs switching on. Session affinity, admission control at the gateway and at the engine, --long-prefill-token-threshold, and Redis so that the limits are global.

The same calculation with the chat profile would have given one GPU and one worker. The distance is not in the safety margin, it is in the inputs.

A reference `config.yaml`

model_list:
 - model_name: agentes-30b
 litellm_params:
 model: hosted_vllm/qwen3-coder-30b
 api_base: http://vllm-agentes.inferencia.svc:8000/v1
 rpm: 600
 model_info:
 input_cost_per_token: 0.00000018
 output_cost_per_token: 0.00000072

router_settings:
 routing_strategy: simple-shuffle
 # without this, the capacity calculation in this article is worthless
 optional_pre_call_checks: ["session_affinity"]
 num_retries: 2
 allowed_fails: 3
 cooldown_time: 30

general_settings:
 # admission control: below this the queue grows without limit
 max_in_flight_requests_per_worker: 24
 max_queued_requests_per_worker: 48
 admission_queue_timeout_seconds: 1.0
 # the spend queue discards at 64 MB; shorter interval, less backlog
 proxy_batch_write_at: 60
 enable_drain_endpoint: true
 drain_endpoint_token: os.environ/DRAIN_TOKEN
 database_connection_pool_limit: 10

litellm_settings:
 # 6000 s by default: a hung connection holds a slot for an hour and a half
 request_timeout: 600
 callbacks: ["langfuse_otel", "prometheus"]
 cache: true
 cache_params:
 type: redis
 host: os.environ/REDIS_HOST
 namespace: litellm

Environment variables that go with it:

NUM_WORKERS=1
TOKEN_COUNTER_MAX_CONCURRENT_COUNTS=8 # default 4; raise only if CPU is spare
LITELLM_TPM_TOKEN_RESERVATION_ENABLED=true
PROXY_CONFIG_RELOAD_INTERVAL_SECONDS=30

And on the engine side:

vllm serve Qwen/Qwen3-Coder-30B \
 --tensor-parallel-size 2 \
 --max-model-len 131072 \
 --kv-cache-dtype fp8 \
 --long-prefill-token-threshold 512 \
 --max-num-queued-reqs 64 \
 --gpu-memory-utilization 0.90

Checklist

Measure three figures of your own over a week: calls per hour of human work and per profile, prompt length distribution, and the time series of prefix cache hit rate.
Calculate the KV per sequence with the model’s real geometry, taken from its config.json, not from a table.
Set --max-num-seqs from that budget, not from a chat heuristic.
Switch on session affinity before accepting the calculated number of GPUs.
Switch on admission control in both layers and check that the client respects retry-after.
Lower request_timeout from 6,000 to 600.
Raise proxy_batch_write_at to 60 and alert on the growth of the spend queue from outside, because there is no metric.
Leave the HPA on CPU at 60 %, never on memory, and assume the 300 seconds of the startupProbe when planning the margin.
Watch litellm_in_flight_requests and litellm_overhead_latency_metric; if the overhead rises with the size of the tool catalogue, the culprit is the token counter.
Go over the dashboards for the vLLM metric renames before upgrading.
Check that no routing policy moves a live session between model groups.
Redo the calculation when a new user profile arrives: the range across profiles is 50 to 1.

Traps and things that are not what they seem

The ratio of 275 to 1 does not mean the system is prefill-bound. The new tokens per turn are between 1.5 and 7.3 times the output. The time is between 91 % and 98.6 % in decode.

The aggregate prefix cache hit rate hides the problem. An aggregate 90 % may be 98 % intra-turn and 55 % at boundaries. You have to look at the series, not the number, and segment by request type.

vLLM’s hit rate counts the reprefill after an eviction as a hit. The counter exists internally and is not exposed, so the published number is optimistic under a memory-pressure regime.

Agents that fail cost more than those that succeed, up to 1.8 times more context. Sizing on the task that goes well leaves the calculation short.

The database pool is per process, not per pod. Four workers are forty connections.

The HPA on memory does not work with this piece, and not because of a configuration mistake: Prisma’s query engine does not return the memory.

The 300-second startupProbe turns any autoscaling into late reaction. For agentic peaks, it is better to over-provision than to trust the scaling.

Raising --max-num-seqs to “make use of the GPU” degrades perceived latency: two long requests in the batch raise time per token by 79.7 %.

The token counter has a limiter of four per worker and it appears in no sizing guide. With large tool catalogues and long contexts it is the first ceiling you hit.

The figures in this article come from three different scaffolds and are not averageable. Fifteen, forty-three and sixty-two turns per session are three systems, not three measurements of the same one.

Closing

The arithmetic of sizing has not changed. What has changed are the inputs, and now there are public measurements at scale instead of intuitions. Three figures sum up the change of regime: 68,000 tokens of median prompt, 87 % of calls started by the agent rather than by the person, and a time split that is decode for more than 90 % as long as the prefix cache hits.

From that comes an order of work that is not the usual one. First, session affinity, because without it the rest of the calculation describes a different system. Second, the KV budget with the model’s real geometry, which is what sets how many sequences fit. Third, the gateway, where the ceiling is not throughput but a token counter with four threads per worker.

And a warning about your own figures. The measured range across user profiles is 50 to 1. Any platform that does not segment its measurement by profile is averaging two different populations and producing a number that describes neither of them.

Sources

Agentic Coding in the Wild: Characterizing GitHub Copilot at Production Scale, arXiv 2608.00101, 30 de julio de 2026.
Agentic AI Workload Characteristics, arXiv 2605.26297, 25 de mayo de 2026.
From LLM Inference to Agentic Workloads, arXiv 2608.15127, 15 de agosto de 2026.
vLLM x AgentX: serving agentic workloads, 8 de septiembre de 2026.
vLLM metrics design y optimization guide, consultados el 12 de septiembre de 2026.
LiteLLM benchmarks y production setup, consultados el 12 de septiembre de 2026.
Código de LiteLLM 1.102.0, commit 9071ca50 del 11 de septiembre de 2026.
Autoescalado con KEDA en vLLM production-stack, consultado el 12 de septiembre de 2026.

LiteLLM's MCP gateway: the second front door, the tool catalogue nobody bills, and why filtering it can make selection worse

Fri, 11 Sep 2026 01:45:00 +0200

Sixth post in the operational track on the control layer. The previous five treat the gateway as the piece the tokens pass through: the pair with Langfuse, day 2, virtual keys, humans and agents and prefix routing. This one is about the other door. Verified against LiteLLM 1.102.0, commit of 10 September 2026.

TL;DR

The MCP gateway is not a module, it is a complete second surface. A native MCP endpoint with four route spellings, a proxy mode with three fixed tools, four REST routes, some thirty-five administration routes and a long set of OAuth discovery endpoints. It still lives under _experimental and it is nearly 20,000 lines.

I correct what I said in the previous post: access is NOT open by default. In 1.102.0, a key with no declared permissions and no team resolves zero servers. What does open things up is a per-server flag, allow_all_keys, and that flag overrides the key’s explicit scope unless you enable a setting that is off by default.

Permissions are enforced on execution, not just on the listing. Calling a tool that did not appear in tools/list returns 403. There are three chained checks before the call, including one on the allowed arguments. The same predicate governs listing and calling, by declared design.

The hierarchy is built out of intersections, with two escapes and a fail-open. The key’s access groups are additive on top of the key and team ceiling. The organisation ceiling replaces rather than intersects when there is nothing below it. And on an indeterminate error while resolving tool permissions, the code returns “unrestricted”. It is a deliberate decision and it has to be declared in the risk analysis.

Per-tool cost is worth zero and that is why its metric disappears. There are exactly two Prometheus metrics for MCP, and the spend one is only incremented if the cost is greater than zero. With the default configuration, that series never exists. There is no listing metric, no latency metric, no error metric, no server health metric.

The cost that does matter is not attributed to MCP anywhere. It is the tokens of the tool definitions, resent on every turn inside the prompt_tokens of the model call. Five typical MCP servers add up to 58 tools and some 55,000 tokens. The tool’s spend row, with its zero cost, has nothing to do with that.

And the gateway’s semantic filter does not act on the MCP door. It only applies when it is LiteLLM calling the model. A client connected to /mcp receives the whole catalogue, filter enabled or not. And even if it did act, filtering per request rewrites the cached prefix of the prompt, which is the opposite of what Anthropic and OpenAI do with their own tool searches.

The good observability ships switched off. Without LITELLM_OTEL_V2, a tool call comes out as a generic span with the model set to MCP: name. With it enabled there is a dedicated span and correct attributes, and the arguments and the result stay hidden unless you opt in. And in neither case is trace context propagated towards the upstream MCP server: the trace is cut at the gateway.

Tool arguments are written, and they are written in the clear. They go in full into the metadata column of the spend table, without passing through the switch that governs prompt storage nor through the redaction function, which only knows about messages and response. Whoever enabled redaction for GDPR probably believes they are covered.

The 1.83 branch concentrated four serious vulnerabilities, and one of them belongs to the MCP gateway itself. The endpoint for testing a connection before saving accepted the stdio transport’s command, args and environment in the request body. A low-privilege key got command execution on the proxy host.

And the specification moved underneath. The revision in force since July 2026 removed protocol sessions, the initialisation handshake and the session header. LiteLLM 1.102.0 advertises the June 2025 revision and its version enumeration does not even contemplate the two later ones.

You are here: the door nobody chose

The gateway was chosen for the reasons in the switchboard post: licence, fit inside the cluster, maturity. What was evaluated was the inference traffic.

Later, somebody connects an agent, the agent needs tools, and the same process that is already authenticated and deployed turns out to speak MCP too. The decision to turn it into the tool door is almost never taken: it is inherited. And that door has neither the same controls, nor the same observability, nor the same cost model.

When MCP grows covered the identity of your own MCP servers. MCP from the inside covered the protocol and its instrumentation. This post covers the piece in between.

The analogy: the switchboard that also hands out keys

An office building’s telephone switchboard. Its original job is to route calls: it decides who can call outside, how much each department spends, and it leaves a record of every conference call.

Over time, the switchboard operator is also given the key cabinet. They are already at reception, they already know who everybody is, it looks like the natural place. And from that moment there are two services at the same desk with very different rules.

Calls are priced by the minute and show up on the bill. Keys have no tariff, so on the department’s bill they come out at zero, and somebody might conclude that the key cabinet costs nothing. It does cost: it costs the operator’s time, it costs having to read the whole key catalogue every time somebody asks what is available, and it costs that nobody checks that the label on a key still says what it said the day its loan was authorised.

The analogy ends at the interesting spot. When somebody takes a key, the record says which key it was and what they said they wanted it for. That record is simultaneously the evidence of the action and a piece of data that perhaps should not be stored in the clear.

The real routes

The first thing that surprises you when you look at the code is how many spellings get you to the same place.

The native endpoint

The MCP sub-application is mounted at /mcp and inside it has four mounts: the root as a wildcard, /mcp, /{name}/mcp and /sse.

And there is also a hand-declared route for bare /mcp on the main application, with a comment that explains why: the mount cannot match its own bare prefix, and the resulting 307 redirect breaks MCP clients sitting behind a proxy that terminates TLS. It is the kind of detail you discover in production on a Friday.

/mcp/sse holds a surprise. It does not use the classic SSE transport: it uses the same HTTP session manager in stateless mode. The SseServerTransport object is constructed and is not referenced on any other line of the file. It is dead code, and the /mcp/sse/messages endpoint that this object advertises does not exist as a working route.

The /{name}/mcp route resolves in a documented order: server alias, comma-separated list with a maximum of 16 elements, tool set, access group. If nothing matches, 404.

And /mcp/proxy exposes a fixed surface of three tools. It is the progressive disclosure pattern, and it will come up again further down because it is the most important mitigation in this post.

The REST routes

Four, under the /mcp-rest prefix:

Method	Route	Authentication
GET	`/mcp-rest/tools/list`	Virtual key
POST	`/mcp-rest/tools/call`	Virtual key
POST	`/mcp-rest/test/connection`	Key plus administrator role
POST	`/mcp-rest/test/tools/list`	Key plus administrator role

The variants without -rest that appear in the internal types have no router registering them. The /mcp/* wildcard absorbs them as JSON-RPC.

The last two are the stars of the security section.

Administration and discovery

Some thirty-five routes under /v1/mcp, almost all of them with virtual key authentication. Three details worth knowing:

GET /v1/mcp/registry.json carries no authentication dependency. It returns 404 if the public registry is not enabled, which is the real protection.

POST /v1/mcp/server requires the proxy administrator role. POST /v1/mcp/server/register does not: it is the proposal path, which requires a key with a team, explicitly forbids the stdio transport and leaves the entry pending review.

And on top of that there is a long set of OAuth .well-known endpoints, with dynamic client registration, authorisation, token, revocation and introspection, plus their per-server variants.

Configuration and transport

The mcp_servers schema in config.yaml has fifty-odd fields, all of them optional. A warning for anyone coming from earlier versions: spec_version no longer exists in 1.102.0.

The default transport when omitted is http, that is, streamable HTTP. The other two values are sse and stdio. The string streamable_http is not valid in the configuration file, although it is accepted as a synonym when importing connector files.

Tool prefixing uses the format {server}{separator}{tool} with - as the default separator, configurable through an environment variable read at import time. The prefix chosen is the alias, or the name, or the identifier, in that order. There is an optional short mode that derives three characters from the hash of the server identifier and resolves collisions by rehashing.

Access control, and the correction to the previous post

In the humans and agents post I closed the list of traps with this one: “access to MCP servers is open by default if no level defines a list”. Verified in 1.102.0, that is false.

The function that resolves the allowed servers does the opposite. If the team declares nothing, it inherits from the key; if the key declares nothing either, the base set is empty and the scope is marked as unbounded with zero servers. There is no branch that expands to “the whole registry” other than the administration role one.

What does open things up, and this is the important part, is something else.

`allow_all_keys`

It is a per-server flag, in configuration, in the API and in the database, defaulting to false. When set to true, that server joins the set of open servers and any key can see it.

The detail to know: by default it overrides the key’s explicit scope. A key that declares access only to server A will also see server B if B has the flag. The key’s scope is only respected if you enable general_settings.mcp_allow_all_keys_respects_mcp_scope, which is off by default.

There is a second opening case: anonymous subjects, with no user identifier, role or key, additionally receive the servers configured with authentication delegation towards the upstream and the pass-through ones. It is consistent with the design of those two modes, and it is a surface you have to know about.

`require_key_mcp_access_defined`

It lives in general_settings, defaults to false, and needs no licence. What it inverts is only the empty-key inheritance towards the team: with the flag on, a key with no list of its own gets an empty set instead of inheriting the team’s.

It does not affect keys that do declare servers, which still intersect with the team. And it does not close off the access group grants, which are additive and are the escape route you have to audit separately.

There is a sibling, require_end_user_mcp_access_defined, with the same shape and applied to the end user.

The hierarchy

The function’s docstring is normative and says that all the rules are intersections. In detail:

Key and team: intersection when both declare. Inheritance in one direction when one is empty.
Key access groups: additive union on top of the previous result.
End user: intersection if it declares anything.
Agent: intersection.
Internal user: ceiling that only narrows.
Organisation: ceiling that intersects if there are restrictions below, and that replaces by becoming the ceiling when there are none.

For subjects with no key, admitted by SSO or by the gateway, the model changes to a per-source union instead of an intersection.

The fail-open

This is the point you have to take to the risk analysis and not leave in a technical note.

On an indeterminate failure while resolving the tool permissions of a key or a JWT, the code returns “unrestricted”, that is, all tools allowed. The organisation ceiling is skipped in the same situation. For subjects with no key the behaviour is the opposite, fail-closed.

There is a well-resolved intermediate case: when a permission is named but cannot be read, a specific error is raised and it is denied in both cases.

The doctrine is documented in the class docstring, which is appreciated. And it remains a behaviour that, in a system under the ENS, you have to declare in writing instead of discovering it during an audit.

What is done well

Permissions are enforced on the call, not just on the listing. The execution path chains three checks before contacting the server: allowed or forbidden tools, per key and team tool permission, and validation of the arguments against the allowed ones. All three return 403.

Listing and calling share the same predicate, and the docstring declares it as an explicit invariant. A client that calls a tool directly when it did not appear in its listing receives 403, it does not execute. The same over REST and over the responses API route.

The x-mcp-servers and x-mcp-access-groups headers only narrow, and they fail closed if they do not resolve.

Header forwarding towards the upstream is an explicit per-server allowlist, with a centralised decision about when to strip the caller’s Authorization, and a protection against cross-forwarding: in a multi-server listing, the global header is withheld if more than one server would consume it. That is RFC 9700 applied.

The bill that never shows up

Here is, in my opinion, the most important thing in this post, and it has nothing to do with security.

Every tool the gateway aggregates takes up space in the model’s context. Name, description and parameter schema travel in the request’s tools block, on every turn, and they are paid for as input tokens of the real model.

The figures published by Anthropic in November 2025 are the best public reference there is. Five connected MCP servers add up to 58 tools and some 55,000 tokens. The breakdown: GitHub, 35 tools and some 26,000 tokens. Slack, 11 tools and some 21,000. Sentry, five and some 3,000. Grafana, five and some 3,000. Splunk, two and some 2,000.

With tool search enabled, that set drops from some 72,000 to some 8,700 tokens, and the usable context goes from 122,800 to 191,300.

The second cost of a large catalogue is not economic and is not billed: it is that the model chooses worse. It has its own section further down, because of the two it is the one that decides whether the agent is any good.

How to measure it with what is already there

The gateway stores enough data, even if it does not cross-reference it for you.

In the listing’s spend row, with call type list_mcp_tools, it enriches the metadata with allowed_server_count, tool_count_total, per_server_tool_counts and per_server_list_outcomes.

In the model call rows, the proxy_server_request column keeps the request body with the tools block inside it.

The practical method: pair the listing rows with the LLM rows of the same session by session_id, read the total number of tools from the first, and pass the tools array of the second through litellm.token_counter(model=..., tools=...), which accepts that argument. Multiply by the number of turns, because the block is resent on all of them.

For the relative cost, a quicker comparison: mean prompt_tokens of keys with MCP enabled against those without, same model and same period.

Tool selection

The previous section treats the catalogue as a bill. This one treats it as what determines whether the agent gets it right. This is ground where the talk runs well ahead of the measurement, so it is worth separating what has been measured from what gets repeated.

How many tools a model can take

Anthropic’s official figure is that Claude’s ability to pick the right tool degrades when going from 30 to 50 available tools. The same documentation gives the inverse criterion, which is the more useful one: the full catalogue with nothing else is the right option with fewer than ten tools, when all of them are used on every request, or when the definitions add up to fewer than a hundred tokens.

There is also an administrative ceiling before the semantic one. VS Code with Copilot cuts off at 128 tools per request and returns an error when you exceed it.

The full curve gets cited far more than it has been measured. The soundest primary data I found is from May 2025: varying the number of candidates from 1 to 11,100 over a registry of more than 4,400 servers, below some 30 tools the success rate is above 90 %, between 31 and 70 intermittent failures appear, and past the hundred mark the degradation becomes dominated by retrieval. It is a qualitative heat map, not a numeric table.

The figure that circulates most (78 % with ten tools, 40 % with a hundred, 13.6 % with seven hundred) comes from a conference talk with no paper, no repository and no published methodology. Do not take it to a client presentation.

And there is a warning that affects everything else. An audit of 496 tasks from four tool calling benchmarks found 18.5 % misalignment between the label and reality. In the BFCL v4 subset, 80 % of the failures assigned to the agent came from fragile state comparisons. In one of the benchmarks with an automatic judge, 23 identical re-runs gave a range from 57.9 % to 76.8 %, almost 19 points of spread, enough to reorder the ranking. Any absolute figure in this section deserves a generous margin of error.

How it fails

The work with the cleanest taxonomy evaluates 36 servers, 220 tools and 1,000 tasks against twenty frontier models, with an average of 15.2 tools exposed per task of which 4.1 are relevant.

Failure mode	Share
Not using a tool when one was needed	10.5 %
Choosing the wrong tool	9.0 %
Malformed parameters	6.9 %
Not recovering from an error	10.3 %
Cognitive failures, not tool failures	63.3 %

The interesting part is in the per-model breakdown: the dominant failure mode changes with the model, not with the size of the catalogue. In one of them, 40.1 % of the failures are simply not calling any tool. In others the malformed parameter dominates. In another, the wrong tool. There is no canonical failure to attack.

On tools that do the same thing, which is the real case of a gateway with fifteen servers, there is a good measurement. A study with ten groups of five functionally equivalent tools and a thousand query pairs puts the selection bias between 0.3 and 0.4, that is, you would have to redistribute between 30 and 40 % of the probability mass for equivalent tools to be chosen equally. The same work shows that a biased continued training takes the selection of a specific endpoint from 0.6 % to 12.8 %.

That study contributes another figure that reorders priorities: shuffling a tool’s description moves the selection distribution substantially, whereas changing only the name has minimal and inconsistent effects. Applied to the gateway’s server-tool prefixing: it costs tokens and probably does not change which tool the model goes to. There is no study isolating the effect of name spacing, and it is a real gap in the literature.

If the description is what carries weight, the state of the estate is the problem. An audit of 856 tools from 103 MCP servers found undeclared limitations in 89.8 %, missing usage guidance in 89.3 % and opaque parameters in 84.3 %. Only 2.9 % were free of problems across all their components.

The optimum is not a number

The May 2026 work I cited above proposes a chance-corrected metric and uses it to size the list adaptively. Over 370 functions it reaches 90.3 % coverage while seeing an average of 7.4 tools, against 90.8 % seeing a fixed 50.

But the same paper publishes its own counterexample, and it is honest to cite it: over another benchmark of 3,251 tools, the fixed K of 5 wins in aggregate, 64.7 % against 61.9 %. The adaptive advantage concentrates in the hard queries, where the right tool sits between sixth and twentieth by similarity: there it finds 16.7 % of the cases where the fixed K finds 0 %.

The March 2026 work over 121 tools puts the plateau at K=3, with 97.1 % that does not improve going up to K=5. With the caveat that it measures retrieval against labels, not model accuracy.

And a July 2026 work gives the formulation that seems to me the right one: ranking by score is inconsistent with optimal acquisition when costs are heterogeneous, so the optimal K is a function of cost, not a constant. Trimming from 7 to 4.4 exposed tools while keeping success is possible; fixing a universal number is not.

Filtering can make the result worse

This is the part that appears in no product blog. A June 2026 benchmark with three registry sizes (25, 100 and 250 tools), six filtering methods, seven models and 26,460 runs:

Strategy	Success	Visible tools	Tokens
Expose the whole catalogue	32.1 %	125.00	56,062
Keyword, top-5	22.1 %	4.80	3,200
Keyword, top-10	22.4 %	9.54	5,356
State-aware	24.0 %	25.95	13,522
Full causal path	24.0 %	26.24	13,697

Four of the five filtering strategies perform worse than not filtering. The fifth, which does win comfortably, exposes 0.99 tools per step and requires hand-written precondition and effect contracts for every tool, plus a search of the causal path to the goal state. It is a planner with the goal given, not a retrieval filter, and it is not comparable with anything a gateway offers.

There is more evidence in the same direction. A work over a corpus of 43,000 tools measured that the agent’s completion rate drops when using retrieved sets against hand-annotated sets: the filter introduces its own recall ceiling. And top-K breaks the tasks that need several tools: on a benchmark of 16,464 endpoints, a fine-tuned retriever reaches a Recall@3 of 68.6 % but only 39.7 % of the queries have their complete tool set within the first three. Almost 29 points of gap between “the one I needed is there” and “all the ones I needed are there”.

Filtering is not free on security either. A March 2026 work attacks the retrieval layer by injecting adversarial tools that saturate the top-k: with injection rates of 1.2 % it manages to dominate the top-k between 91 % and 97 % of the time. It is an attack surface that exposing the whole catalogue does not have.

The most elegant counterpoint comes from Anthropic’s own numbers. With the same catalogue of more than fifty tools, the gain from tool search drops from 25 points on one model to 8.6 on the next, because the unfiltered baseline rises from 49 % to 79.5 %. The margin left by filtering shrank to a third in one generation of models. Any architecture decision taken here has a short shelf life.

The prompt cache trap

Here is what makes a homemade top-K filter a bad idea even if it worked.

Tool definitions do not go just anywhere in the prompt: they go at the beginning. Anthropic documents the order of the cacheable prefix as tools, then system, then messages, with a hierarchy where each level rests on the previous one. And the invalidation rule is explicit: modifying tool definitions, be it names, descriptions or parameters, invalidates the entire cache, and a change at one level invalidates that level and all subsequent ones.

OpenAI says the same and adds a detail: the prefix must remain unchanged, and order counts. The same set of tools served in a different order is a cache miss.

With vLLM serving an open model there is no tool block of its own. The tools parameter is serialised by the chat template inside the first system message, behind the system prompt text. For automatic prefix caching the effect is identical, because it is still near the absolute start of the prompt.

The consequence is the one that matters. A filter that returns a different set of tools on the next turn does not just invalidate the tool block: it invalidates everything behind it, which is the system prompt and the whole conversation, which is exactly the part that grows with the turns. At list prices, a prefix token goes from costing 0.1 times when read from cache to 1.25 times when written. An agent with sixty thousand tokens of stable prefix goes from some six thousand equivalent tokens per turn to some seventy-five thousand, plus the prefill that gets paid again in latency.

The two big providers have separately arrived at the same solution, and it is the opposite of filtering in the prefix. Anthropic excludes deferred tools from the prefix and, when the model discovers one through search, injects it inline inside the conversation: “the prefix stays intact, so prompt caching is preserved”. OpenAI: “all tools are loaded at the end of the model’s context window (…) this allows the cache to be preserved from one request to the next”. Anthropic’s operational recommendation is the stable core pattern: leave the three to five most used tools undeferred.

The closest academic work, from January 2026, states the problem without measuring it and recommends exactly that, keeping a fixed set of reusable general-purpose functions and implementing dynamic capability through code generation.

The July 2026 MCP specification picks up both halves of the matter. Servers “should return the tools from tools/list in a deterministic order to allow caching in the client and improve the hit rate of the model’s prompt cache”. And it adds mandatory ttlMs and cacheScope in listing results, so that the client can freeze the catalogue for a known window instead of re-querying it and propagating any variation to the prompt.

Nobody has publicly measured the cost in euros of this effect. The only attempt I found is a self-published audit of ten thousand turns that puts it at 2.4 % to 3.5 % of cache misses, with non-reproducible methodology. It is a gap in the literature, and the order of magnitude in the previous paragraph comes from applying list prices, not from a measurement.

And now, LiteLLM

With all of the above in mind, the gateway’s levers read differently.

The semantic filter does not act on the MCP endpoint. It is registered as a generic LiteLLM callback, and its only dispatcher is the proxy’s precall hook, which additionally discards anything that is not completion, acompletion or aresponses. The tools/list path does not invoke that hook at any point; there is a comment in the code that admits it in passing, when explaining why the listing’s JWT signature is done separately. An external MCP client connected to /mcp receives the whole catalogue, filter enabled or not.

Lever	Serves external MCP clients	Breaks the prompt cache
Per-server `allowed_tools`	Yes	No, it is static
Name and description overrides	Yes	No, it is static
Selection by route or header	Yes	No, if it is stable per agent
`/mcp/proxy` and virtual tools	Yes	No, the surface is fixed
Semantic filter	No	Yes, it rewrites the block per request

When it does act, the filter has details you need to know before enabling it. It embeds only each tool’s description, with the name as a fallback if it is missing; the parameter schema does not go in. Of the query it embeds only the last user message: on turn twelve of an agent conversation, that might be “yes, go ahead”. The tool embeddings are cached and indexed at start-up; the query’s is computed on every request, with a synchronous call inside a coroutine.

Its failure modes are almost all open, which is the reasonable thing: if the embedding model does not respond, if nothing passes the threshold, if the names do not match the catalogue, it returns all the tools. The exception is context overflow, which fails closed with a 400, and which if it happens while building the index during start-up is memoised and makes all subsequent requests fail until you restart. The defaults are text-embedding-3-small, top_k of 10 and a threshold of 0.3, under litellm_settings.mcp_semantic_tool_filter.

/mcp/proxy does serve external clients, and that is where its real value lies. It replaces the catalogue with three tools with opaque identifiers. How search_tools searches deserves a line, because it is not what you would assume: by default it is neither a regular expression nor a semantic match, it is a keyword counter, how many tokens of the query appear as a substring in the concatenated name and description. It only moves to embeddings if you configure a model in mcp_tool_search. The result cap is hard-coded at five in this mode, and the client cannot ask for more. In exchange, call_tool goes through the same permission checks as a normal call plus a validation of the argument schema that the normal route does not do at that point.

There is a related variant: with mcp_tool_search_enabled in the key’s permissions, the catalogue is replaced by four virtual tools, and there the search one does expose top_k to the client.

The catalogue order is not deterministic, and that breaks the client’s cache without anybody filtering anything. The list of allowed servers comes from iterating a Python set of strings, so with the random hash seed the order differs between processes, between uvicorn workers and between restarts. There is no sorting anywhere in the listing path, and there is no tool cache: every tools/list queries the upstreams live, so the order within each server also depends on what the upstream returns that time. The gateway does not offer today the guarantee that the July specification recommends. The fix, sorting the catalogue before serving it, fits on one line.

Nor is there any pagination towards the client. The catalogue is returned whole, always.

And the lever almost nobody uses is the static one. Every MCP server accepts tool_name_to_display_name and tool_name_to_description, and those overrides really do replace what is sent to the client in tools/list, not just what the administrator sees in the interface. The tools/call routing translates from the new name to the real one before any permission check. With that you can trim a four-hundred-token description written by a third party, rename an ambiguous tool so it does not compete with a similar one, and do it without touching the prompt cache, because it is static and holds for all turns.

It is also the rug pull mitigation left pending from the security section: if the description the model sees is the one the operator wrote, whatever the upstream server changes stops reaching the context. Two limits. They are configured through the database or the management API, not through config.yaml. And the reverse name translation is by exact equality and is not disambiguated between servers, so two servers with the same display name route unstably, made worse by the non-deterministic order of the previous paragraph.

Alongside that, per-server allowed_tools remains the blunt and effective instrument: it filters the listing the client sees, reduces tokens, and is static.

The ladder, in order

Trim. Per-server allowed_tools, plus a per-key allowlist. Cheap, static, does not break the cache and reduces the permission surface at the same time.
Rewrite the descriptions with the overrides. That is where the evidence is, because the description carries weight and the name almost none, and along the way it pins the contract against the rug pull.
Split by route. Each agent to /{server}/mcp or to a tool set, never to the aggregated endpoint. Stable per agent, so the cache survives.
Sort the catalogue before serving it if you control the deployment, while the gateway does not do it.
/mcp/proxy when the catalogue goes past thirty or fifty tools. It is progressive disclosure and it works with external clients, which is more than you can say for the semantic filter.
The semantic filter, only when the one calling the model is LiteLLM. With a generous top_k, knowing that it embeds only the last user message and that context overflow is a closed and sticky failure.
What not to do: a home-grown top-K filter that rewrites the tools array on every turn. It costs the whole prefix and, according to four of the five measured strategies, probably makes selection worse.

The declared cost, which is worth zero

The calculation has four precedence steps: cost set by a post-call hook, per-tool cost in tool_name_to_cost_per_query, server default cost, and 0.0.

Without configuring mcp_server_cost_info, all MCP traffic is accounted at zero and consumes neither key nor team budget.

There are exactly two Prometheus metrics for MCP:

litellm_mcp_tool_calls_total
litellm_mcp_tool_call_spend_metric

Both with the same eight labels: tool name, server name, key hash, key alias, team, team alias, user and end user.

And the second is only incremented if the cost is greater than zero. With the default configuration, that time series never appears in Prometheus. It is not that it is worth zero: it is that it does not exist.

Nor is there a tools/list metric, or MCP latency, or MCP errors, or upstream server health. For that you are left with the generic request metrics.

In the reports, /spend/calculate has no MCP branch: it accepts a model and messages, or a completion response, and it always calls the LLM cost calculation. The per-team report does include MCP calls, grouped by the model column, which for these rows is worth MCP: tool_name. That is: every tool appears as if it were a model, with zero input and output tokens. Only the sessions endpoint separates the two call types explicitly.

Observability, and the trace that gets cut

There are two OpenTelemetry stacks in LiteLLM and which one runs depends on an environment variable.

Without LITELLM_OTEL_V2=true, the old logger runs, and it has no MCP handling: the only appearance of MCP in the whole file is the metadata key in a list. The tool call comes out as a generic litellm_request span with the model set to MCP: name.

With v2 enabled there is a dedicated span, with a dedicated role, client kind and a parent in the proxy’s request span. The span name is tools/call tool-name. The attributes are correct and follow the conventions:

Attribute	Value
`gen_ai.operation.name`	`execute_tool`
`mcp.method.name`	`tools/call`
`mcp.session.id`	MCP session identifier
`gen_ai.tool.name`	Tool name
`gen_ai.tool.call.arguments`	Only with content capture enabled
`gen_ai.tool.call.result`	Only with content capture enabled
`server.address`, `server.port`	From the upstream URL, redacted
`litellm.cost.total`	The cost that is probably worth zero

Content capture is off by default. Without it you see neither the arguments nor the result in the trace.

A note for anyone working with semantic conventions: mcp.tool.name does not exist. The MCP registry has exactly four attributes (mcp.method.name, mcp.session.id, mcp.protocol.version and mcp.resource.uri), all four in development, and the tool name goes in gen_ai.tool.name. LiteLLM gets it right. And the GenAI conventions were extracted into their own repository which as of today has no release at all, so there is no version number to cite.

In Langfuse, the vendor mapper returns an empty dictionary for any span that is not an LLM call. An MCP call appears as a raw span with its attributes, with no native input, output or model fields.

The trace is cut at the gateway

Incoming context is read. The traceparent the client puts in params._meta, following the specification’s propagation proposal, is extracted and turned into a link on the span, never into its parent. The reason is documented in the code: parenting to the remote trace would leave the span hanging off a trace whose root never reaches the backend. The client’s baggage is discarded on purpose to avoid forging identity attributes.

Outgoing context is not propagated. There is no propagator injection call anywhere in the tree. The headers that go out towards the upstream MCP server are built with authentication and the authorised extra headers, and nothing else.

The operational consequence: there is no end-to-end trace. If the upstream MCP server is instrumented, its spans live in another trace and there is no way to correlate them automatically.

Auditing and the conflict with privacy

A tool call leaves a normal spend row, with call type call_mcp_tool, spend equal to the cost per query, model MCP: name, zero tokens in the three token columns, a dedicated mcp_namespaced_tool_name column, and the usual key, team, user and session fields.

And in the metadata column, in JSON, the complete structure of the call: name, arguments, result, server, namespaced name, MCP session identifier, authentication mode and server resource with the URL redacted.

The problem

That structure is written into the metadata column without passing through the privacy switch. The assignment is direct and is not conditioned by the function that governs whether prompts and responses are stored, which does control the messages and response columns.

And the redaction function does not cover it either: it only touches messages and response. A search for MCP in the redaction file returns nothing.

The practical result: an operator who enables turn_off_message_logging for GDPR keeps sending the full arguments of every tool to every logging destination they have configured. To the SIEM, to Langfuse, to S3.

If an argument carries a national ID number, an address, a case number or a medical record, it has been exported.

There is an aggravating factor that does not depend on LiteLLM. The revision of the specification in force introduces an extension that allows mirroring tool arguments into HTTP headers visible to load balancers, proxies and web application firewalls. The specification recommends not marking sensitive parameters that way, and it is a recommendation, and the upstream server decides.

The fit with the ENS

The conflict is real and has a familiar shape: activity logging (op.exp.8) requires the call to be traced and non-repudiable; protection of personal data (mp.info.1) requires that log not to become a secondary repository of personal data with a different purpose. The tool argument is simultaneously the evidence of the action and the data. Redacting it destroys non-repudiation; storing it in the clear creates the repository.

The defensible way out you can build today: a hash of the argument in the audit log, the argument in the clear with encryption (mp.info.3), separate retention and its own access control. The hash sustains non-repudiation and the clear text is only opened under procedure.

There are two other measures this material touches directly. Change management (op.exp.5) because registering an MCP server and, above all, changing the description of an already approved tool are changes to the security configuration. And maintenance and updates (op.exp.4) because, as you can see in the next section, an old version of the gateway is a non-compliance with a name and a number.

Two honesty warnings. The codes follow the numbering of Annex II of RD 311/2022 and you have to check them against the text in force before citing them in a compliance document. And on NIS2: as of the date of this post the directive is still not transposed in Spain, with the draft bill on cybersecurity coordination and governance going through the process and a reasoned opinion from the Commission over the delay. What is enforceable today in the public sector and its suppliers is the ENS.

What is not logged

An administrative gap: registering, modifying and deleting MCP servers generates no audit log. There are three TODO comments in the management code that say so, one of them with an if is_audit_logging_enabled(): pass. In the same deletion block there are two more TODOs about not cleaning up orphaned permissions on keys and teams.

Security

The vulnerability you need to know about

CVE-2026-42271, published on 8 May 2026. Affects LiteLLM from 1.74.2 up to versions prior to 1.83.7. CVSS 3.1 of 8.8.

The POST /mcp-rest/test/connection and POST /mcp-rest/test/tools/list endpoints, which exist to preview an MCP server before saving it, accepted the stdio transport’s command, args and env in the request body. A low-privilege key executed arbitrary commands on the proxy host. It also affects Red Hat OpenShift AI on several branches.

It is the story of this post condensed into one line: the gateway you put in to contain MCP was the hole.

It does not come alone. The 1.83 branch concentrates three more serious ones:

CVE	Component	CVSS	Fixed in
`CVE-2026-42208`	SQL injection in key validation, with no prior authentication	9.8	1.83.7
`CVE-2026-35030`	OIDC userinfo cache keyed on `token[:20]`, collision and impersonation	9.1	1.83.0
`CVE-2026-40217`	Remote execution through bytecode rewriting in the guardrails test endpoint	8.8	no version recorded

Plus two privilege escalations fixed in 1.83.10 and 1.83.14.

The operational conclusion is short: a deployment below 1.83.14 is exposed to remote execution or to an unauthenticated SQL injection. The current branch is 1.102.0.

And a note for context: the whole MCP ecosystem has the same pattern. The unauthenticated remote execution in MCP Inspector, the command injection in mcp-remote when connecting to an untrusted server, DNS rebinding disabled by default in the Python and TypeScript SDKs until late 2025, the directory escapes in the reference servers. Connecting to an untrusted MCP server compromises the client, not just the other way round.

The rug pull nobody covers

Documented since April 2025: a server can change a tool’s description after the client has approved it. The description is what the model reads to decide when and how to use it, so changing it is changing the behaviour without asking for permission again.

The specification in force does not address it. It defines list change notification and cache freshness fields, which are recency mechanisms, not integrity ones. The closest normative warning is that clients must treat tool annotations as untrusted unless they come from trusted servers.

LiteLLM does not cover it either. The only “pinning” in the code is of the configuration’s server identifier, so that it does not change when you edit the YAML. There is no hash or comparison of the description or the input schema between successive listings. A search for related terms in the MCP module: nothing functional.

The control exists outside the gateway. mcp-scan, from the same researchers who documented the attack, implements description hash pinning precisely to detect this.

In ENS terms it is an unauthorised change to the security configuration that today goes undetected. And the home-made fix fits in a cron job: store the hash of the description and schema of every approved tool, compare against tools/list periodically, alert on the difference.

Three more holes in the code

No URL validation towards the upstream. A new server’s url field has no validator. The SSRF protection utilities exist in the tree and are used in OAuth discovery and in downloading OpenAPI specifications, but not in the egress of tools/call. A server pointing at a private range or at the cloud metadata IP is accepted as is.

The stdio transport is local execution on the proxy host. There is a command allowlist: npx, uvx, python, python3, node, docker, deno, extendable through an environment variable. The code’s own comment admits the residual risk: the permitted runtimes can execute code through arguments. With docker and npx in there, it is de facto arbitrary execution for any proxy administrator.

Two server modes skip the gateway’s authentication. Pass-through and authentication delegation to the upstream completely omit virtual key validation. They are well hardened (fail-closed if the target is mixed or unresolvable, strict comparison against truthy values, explicit blocking of the client credentials flow so as not to lend out the proxy’s service account), and the result is still that those servers have no gateway access control.

The limit that truncates in silence

The tools/list pagination stops and returns what it has accumulated in three cases: repeated cursor, one thousand pages, or expiry of a global deadline. The docstring declares it: a slow or faulty upstream produces a partial catalogue instead of an error.

The first two log a warning. The third, the deadline, exits without logging anything at all. A slow server makes part of its tools disappear from the catalogue with no trace left behind.

Correction on the slot leak

In the previous post I took issue 34534 at face value, according to which every MCP tool call acquires a max_parallel_requests slot and does not release it. Tracing the 1.102.0 code, that diagnosis no longer holds: the release path exists, on both success and failure, and the limiter is registered as a litellm callback and not just as a proxy hook.

With one honest caveat: I found no regression test covering the full cycle for MCP, and the definitive confirmation is a real proxy with max_parallel_requests: 2 and five MCP calls in a row. If anybody reproduces it today, I would like to know.

The specification moved underneath

The MCP revision in force is 2026-07-28, published on 28 July 2026, and in between there was another one, 2025-11-25. The changes in the latest one are not cosmetic:

Protocol sessions and the session header are removed. Cross-request state becomes an explicit identifier in the tool arguments.
The initialisation handshake is removed. Every request carries version and capabilities in _meta. MCP becomes stateless.
server/discover appears as a mandatory call.
Elicitation changes shape: out go server-initiated requests, in comes a multi-round pattern where the server returns an “input required” result and the client retries with the answers.
ping, logging/setLevel and the roots change notification are removed. SSE stream resumption is removed.
Standard HTTP headers are mandatory on POST, which lets a gateway or a web application firewall decide without parsing the JSON.
On authorisation: mandatory issuer validation in the client, and dynamic client registration is deprecated in favour of client identifier metadata documents.
Roots, sampling and logging are deprecated, with a minimum window of twelve months.

LiteLLM 1.102.0 advertises 2025-06-18. Its version enumeration has three values and none of them is later than that date. The literal is sent in the initialize against the upstream.

One nuance that is not entirely bad: the version the gateway advertises to its clients is not hard-coded in LiteLLM, the SDK negotiates it. And there is partial implementation of the intermediate revision, because the elicitation and sampling handlers reference it in their docstrings. That is: partial implementation of one revision while advertising an earlier one.

In practical interoperability it does not hurt yet, because the client ecosystem is still mostly on the 2025 revisions. It hurts in planning: July’s stateless redesign invalidates the gateway’s session model, and that is work ahead.

Two ecosystem notes for anyone choosing. There are clients that speak remote streamable HTTP natively (Claude and Claude Code, ChatGPT with OAuth and dynamic registration, VS Code with Copilot, Cursor, Zed, Kiro) and others that still demand stdio or a local bridge with mcp-remote, among them Windsurf, Cline and Continue.

Tool execution from the proxy itself

Besides serving MCP outwards, the proxy can consume it itself. It is enabled with an mcp block in the body, with server_url equal to the litellm_proxy sentinel, and it works both on /v1/responses and on /chat/completions.

The flow: it separates the gateway’s tools, lists them, filters by allowed, deduplicates, translates them into OpenAI function format and concatenates them. It calls the model. If there are tool calls and auto-execution is permitted, it runs them against the MCP servers. It builds the follow-up and makes one second call.

Two limits you need to know before designing on top of this.

The loop is a single hop. There is no multi-step iteration. The calls from the first response are executed and the follow-up is returned. A plan that chains tools requires the client to call again.

Auto-execution fails closed and as a block. It requires all MCP references to have approval set to “never”; a single one requiring approval disables auto-execution for the whole request.

And a third one, which is the one that bites: the multi-turn state of /v1/responses is rehydrated from the spend table. The query is literal, it looks up the session of the previous request_id and retrieves all the rows of that session ordered by time.

With disable_spend_logs: true nothing is written, the query comes out empty and the history is lost in silence, with no 4xx error. The code itself acknowledges it: it does not expect retries because deployments that do not write spend logs have nothing to wait for. It is a coupling between a performance option and a conversation feature that nobody expects to find.

Health and day-to-day operation

Health checks exist, they are on demand and they are not periodic. GET /v1/mcp/server/health accepts repeated identifiers and returns per-server status. The implementation opens a client and runs an empty operation inside the session, with a ten-second timeout.

It skips the check entirely if the server requires per-user authentication, and those always return unknown status.

The surprising part: the status, last check and error columns have existed in the database since a May 2025 migration, and the check function returns an object without writing to the database. Status is recalculated on every request. There is no periodic MCP health job.

When an upstream server is down, the behaviour is to degrade and not block. The listing fans out in parallel and each failing server contributes an empty list plus a result classified into six categories. Those results reach the client in the response’s _meta under a vendor key, without filtering the upstream’s text or URL. The reason for the design is in the code: an empty contribution with no signal makes a broken upstream indistinguishable from a healthy server with no tools.

There is no cache of the previous list. A server that is down disappears from that turn’s tools.

On registering and deregistering without restarting, there are two worlds. Database servers can be created, edited and deleted through the API, and every operation reloads the registry with an atomic swap. Propagation between replicas goes through Redis publication on the configuration change channel, if there is Redis, and through polling every 30 seconds if there is not.

Servers declared in config.yaml are loaded once, at start-up. There is no re-read. Changing them requires a restart.

And there is a precedence trap between the two worlds, with two different warnings in the code. A database row with the same identifier as a configuration entry hides the configuration one entirely. And the other way round, a configuration identifier that matches the name or alias of a database server captures its permissions, because identifiers are resolved before names. Both cases emit a log warning, once.

A hardened `config.yaml`

general_settings:
 # Access is not open by default, but this closes the empty-key
 # inheritance towards the team, which is the one that surprises people.
 require_key_mcp_access_defined: true
 require_end_user_mcp_access_defined: true
 # Without this, allow_all_keys overrides the key's explicit scope.
 mcp_allow_all_keys_respects_mcp_scope: true

litellm_settings:
 # Does not cover MCP arguments. See the auditing section.
 turn_off_message_logging: true
 callbacks: ["langfuse_otel"]

mcp_servers:
 inventario:
 url: https://mcp-inventario.interna.svc/mcp
 transport: http
 auth_type: bearer_token
 auth_value: os.environ/MCP_INVENTARIO_TOKEN
 # Explicit allowlist. Also enforced on tools/call.
 allowed_tools:
 - buscar_articulo
 - consultar_stock
 # Never true on a server with data.
 allow_all_keys: false
 access_groups: ["operaciones"]
 timeout: 20
 max_concurrent_requests: 8
 mcp_info:
 description: "Inventario de almacen, solo lectura"
 # Without this, MCP spend is zero and its metric does not exist.
 mcp_server_cost_info:
 default_cost_per_query: 0.0005
 tool_name_to_cost_per_query:
 consultar_stock: 0.0002

And in the process environment:

# Without this there is no MCP span, only a generic litellm_request.
LITELLM_OTEL_V2=true
# Full block hashes, not truncated, if correlating
# with engine events.
LITELLM_MCP_CLIENT_TIMEOUT=45
LITELLM_MCP_TOOL_LISTING_TIMEOUT=20

For clients, the rule is not to use the aggregated endpoint. Each agent points at /{server}/mcp or at a tool set, or at the three-tool proxy mode if the catalogue is large.

Checklist

Check the version. Below 1.83.14 there is remote execution and an unauthenticated SQL injection with an assigned CVE.
Inventory which servers have allow_all_keys and enable the setting that makes it respect the key’s scope.
Audit the access groups separately, because they are additive on top of the key and team ceiling.
Declare in writing the fail-open behaviour on tool permission resolution errors.
Measure the real cost: number of tools per server from the listing row, and tools block tokens from the stored request body.
Choose a catalogue strategy before connecting the third server: three-tool proxy mode, semantic filter, or selection by route. Not the aggregated endpoint.
Declare mcp_server_cost_info even with a token value, or the spend metric will not exist.
Enable the OTel v2 logger, and decide deliberately whether to enable content capture.
Assume the trace is cut at the gateway and plan correlation with the upstream by other means.
Treat the spend table’s metadata column as a personal data repository: encryption, its own retention and separate access control.
Set up the rug pull check, because nobody does it: hash of the description and schema of every approved tool, periodic comparison against the listing.
Restrict egress towards MCP servers at the network level, because the gateway does not validate the destination URL.
Do not enable the stdio transport unless you need it, and if you do, review what is on the command allowlist.
If you use /v1/responses with multi-turn state, do not enable disable_spend_logs.

Traps and things that are not what they seem

MCP access is not open by default in 1.102.0. I got that wrong in the previous post. What opens things up is per-server allow_all_keys.
allow_all_keys overrides the key’s explicit scope unless you enable a setting that ships disabled.
require_key_mcp_access_defined only closes key-to-team inheritance. Access group grants remain additive.
On an indeterminate failure resolving tool permissions, the code fails open for keys and JWTs.
The default cost of an MCP call is 0.0, and the spend metric is only emitted if the cost is greater than zero, so the series does not exist.
MCP calls appear in the per-team report as if they were models, named MCP: tool and with zero tokens.
The OTel logger with MCP support is off by default. Without the environment variable there is no MCP span.
Trace context is not propagated towards the upstream MCP server. There is no end-to-end trace.
mcp.tool.name does not exist in the semantic conventions. The name goes in gen_ai.tool.name.
Message redaction does not cover tool arguments. They are written in the clear into the spend table.
Registering, modifying and deleting MCP servers generates no audit log. There are three TODOs in the code.
The semantic tool filter is not applied on tools/list. Only on /chat/completions and /v1/responses.
The catalogue order is not deterministic, because it comes from iterating a set of strings. It breaks the client’s prompt cache even though nobody filters anything.
search_tools in proxy mode counts keywords by default, it uses neither regular expressions nor embeddings, and its cap of five results is hard-coded.
Description overrides really do replace what the client sees, and they are the only catalogue lever that does not break the cache. They are only configurable through the database or the API.
spec_version no longer exists in the configuration schema.
/mcp/sse does not use the SSE transport, and the messages endpoint its object advertises is dead code.
Listing pagination truncates in silence when the deadline expires, without leaving a single log line.
A database row hides a configuration entry with the same identifier, and the other way round a configuration identifier can capture the permissions of a database server.
Servers declared in config.yaml require a restart. Only the database ones reload hot.
The health check does not write its result to the database even though the columns exist, and there is no periodic job.
The gateway does not validate the upstream MCP server’s URL. The anti-SSRF utilities exist and are not used on that path.
docker and npx are on the stdio command allowlist.
The /v1/responses tool loop is a single hop, and disable_spend_logs: true breaks its multi-turn behaviour with no error.
LiteLLM advertises the June 2025 revision of the protocol, two behind the one in force.

Closing

There is a thread running through the six posts in this track that is more visible here than anywhere else. The gateway is chosen for one job and ends up doing five, and each new job arrives with its own defaults, which are almost never the ones you would have chosen.

In the case of MCP the asymmetry is twofold. On one side, the part that looks dangerous is better built than you would assume: access control is enforced on execution and not just on the listing, the hierarchy is built out of intersections, header forwarding is an allowlist, and default access is closed, contrary to what I myself wrote two days ago. On the other, the part nobody looks at is worse: tool arguments come out in the clear bypassing the privacy switch, spend is worth zero and its metric never gets to exist, the trace is cut, and a change to the description of an approved tool goes undetected by anybody in the whole chain, neither the specification nor the gateway.

The architecture decision that comes out of this is that the MCP door has to be treated as what it is, an interconnection of systems with third parties, and not as one more function of the LLM proxy. That means its own inventory, formal registration, change control over every tool’s description, network isolation towards the upstream and a log designed knowing that it is going to contain personal data.

And it means accepting that the cost you really pay for connecting fifteen MCP servers is in no column of the spend table. It is spread across the prompt_tokens of every model call, on every turn, and across the accuracy you lose when the catalogue is too large for the model to choose well. That number can be computed with what the gateway already stores. The first step is wanting to compute it.

Sources

LiteLLM, MCP Gateway: https://docs.litellm.ai/docs/mcp.
LiteLLM, código: litellm/proxy/_experimental/mcp_server/server.py, mcp_server_manager.py, rest_endpoints.py, auth/user_api_key_auth_mcp.py, cost_calculator.py, faults/list_outcomes.py, litellm/proxy/management_endpoints/mcp_management_endpoints.py, litellm/proxy/spend_tracking/spend_tracking_utils.py, litellm/litellm_core_utils/redact_messages.py, litellm/integrations/otel/, litellm/responses/litellm_completion_transformation/session_handler.py: https://github.com/BerriAI/litellm.
LiteLLM, incidencia 34534, fuga de slot de concurrencia en llamadas MCP: https://github.com/BerriAI/litellm/issues/34534.
NVD, CVE-2026-42271, ejecución de comandos por los endpoints de prueba de MCP: https://nvd.nist.gov/vuln/detail/CVE-2026-42271.
NVD, CVE-2026-42208, inyección SQL sin autenticación en la validación de clave: https://nvd.nist.gov/vuln/detail/CVE-2026-42208.
NVD, CVE-2026-35030, colisión de clave de caché de userinfo OIDC: https://nvd.nist.gov/vuln/detail/CVE-2026-35030.
NVD, CVE-2025-49596, ejecución remota en MCP Inspector: https://nvd.nist.gov/vuln/detail/CVE-2025-49596.
NVD, CVE-2025-6514, inyección de comandos en mcp-remote: https://nvd.nist.gov/vuln/detail/CVE-2025-6514.
NVD, CVE-2025-66416, protección de DNS rebinding desactivada por defecto en el SDK de Python: https://nvd.nist.gov/vuln/detail/CVE-2025-66416.
Invariant Labs, MCP Security Notification: Tool Poisoning Attacks (tool poisoning, rug pull, shadowing): https://invariantlabs.ai/blog/mcp-security-notification-tool-poisoning-attacks.
Invariant Labs, Introducing mcp-scan (fijado de herramientas por hash): https://invariantlabs.ai/blog/introducing-mcp-scan.
Model Context Protocol, changelog de la revisión 2026-07-28: https://modelcontextprotocol.io/specification/2026-07-28/changelog.
Model Context Protocol, changelog de la revisión 2025-11-25: https://modelcontextprotocol.io/specification/2025-11-25/changelog.
Model Context Protocol, Security Best Practices (confused deputy, indicadores de recurso, validación de audiencia): https://modelcontextprotocol.io/docs/2026-07-28/tutorials/security/security_best_practices.
NSA, Model Context Protocol: Security Design Considerations for AI-Driven Automation, mayo de 2026: https://media.defense.gov/2026/Jun/02/2003943289/-1/-1/0/CSI_MCP_SECURITY.PDF.
CISA y agencias aliadas, Careful Adoption of Agentic AI Services, mayo de 2026: https://www.cisa.gov/resources-tools/resources/careful-adoption-agentic-ai-services.
Anthropic, Advanced tool use (coste en tokens de las definiciones, búsqueda de herramientas, precisión de selección): https://www.anthropic.com/engineering/advanced-tool-use.
Anthropic, Code execution with MCP: https://www.anthropic.com/engineering/code-execution-with-mcp.
Anthropic, Prompt caching (jerarquía tools → system → messages e invalidación): https://platform.claude.com/docs/en/build-with-claude/prompt-caching.
Anthropic, Tool search tool (herramientas diferidas fuera del prefijo, umbral de 30 a 50 herramientas): https://platform.claude.com/docs/en/agents-and-tools/tool-use/tool-search-tool.
OpenAI, Prompt caching (el orden de las herramientas cuenta): https://developers.openai.com/api/docs/guides/prompt-caching.
OpenAI, Tool search (las herramientas descubiertas se cargan al final del contexto): https://developers.openai.com/api/docs/guides/tools-tool-search.
vLLM, Automatic Prefix Caching: https://docs.vllm.ai/en/latest/features/automatic_prefix_caching.html.
Microsoft, Use tools in chat (límite de 128 herramientas por petición en VS Code): https://code.visualstudio.com/docs/copilot/agents/agent-tools.
RAG-MCP, prueba de estrés de 1 a 11.100 servidores candidatos, mayo de 2025: https://arxiv.org/abs/2505.03275.
Benchmarking the Benchmarks, desalineación del 18,5 % entre etiqueta y realidad, junio de 2026: https://arxiv.org/html/2607.02577v1.
MCP-Atlas, taxonomía de fallos sobre 220 herramientas y veinte modelos, febrero de 2026: https://arxiv.org/html/2602.00933v3.
BiasBusters, sesgo de selección entre herramientas equivalentes y peso de la descripción frente al nombre: https://arxiv.org/html/2510.00307.
MCP Tool Descriptions Are Smelly, auditoría de 856 herramientas de 103 servidores, febrero de 2026: https://arxiv.org/html/2602.14878v1.
ToolMenuBench, seis estrategias de filtrado y 26.460 ejecuciones, junio de 2026: https://arxiv.org/html/2606.15508.
Retrieval Models Aren’t Tool-Savvy (ToolRet), corpus de 43.000 herramientas, marzo de 2025: https://arxiv.org/abs/2503.01763.
Tools Are Not Islands, brecha entre Recall@3 y conjunto completo, julio de 2026: https://arxiv.org/html/2607.25718.
ToolFlood, saturación adversaria de la capa de recuperación, marzo de 2026: https://arxiv.org/html/2603.13950.
Scores Are Not Decisions, el K óptimo como función del coste, julio de 2026: https://arxiv.org/html/2607.27083v1.
Don’t Break the Cache, caché de prompts en tareas agénticas de horizonte largo, enero de 2026: https://arxiv.org/html/2601.06007v2.
How Many Tools Should an LLM Agent See? A Chance-Corrected Answer, mayo de 2026: https://arxiv.org/html/2605.24660v1.
Semantic Tool Discovery for LLMs, marzo de 2026: https://arxiv.org/abs/2603.20313.
OpenTelemetry, convenciones semánticas de GenAI y MCP: https://github.com/open-telemetry/semantic-conventions-genai.
Boletín Oficial del Estado, Real Decreto 311/2022, Esquema Nacional de Seguridad: https://www.boe.es/buscar/act.php?id=BOE-A-2022-7191.
IBM ContextForge, gateway y registro MCP: https://github.com/IBM/mcp-context-forge.
Docker MCP Gateway: https://github.com/docker/mcp-gateway.
agentgateway: https://github.com/agentgateway/agentgateway.

Prefix routing: what LiteLLM does not do, who does, and how much it really improves

Fri, 11 Sep 2026 00:30:00 +0200

Fifth article in the operational track of the control layer. The pair with Langfuse covered observability, the proxy’s day 2 covered availability, virtual keys covered governance and humans and agents covered the coexistence of two classes of traffic. Here a claim of mine from June gets taken apart. Everything that follows is verified against LiteLLM 1.102.0 (commit of 10 September 2026), vLLM 0.29.0 and the main branches of production-stack, llm-d and Dynamo from the second week of September 2026.

TL;DR

LiteLLM does not do prefix-aware routing, neither in the open version nor in the paid one. It is not a licensing question. A search across the whole tree for kv_aware, cache_aware, radix, prefix_cache and block_hash returns zero matches in the router. Nor is there any call from the router to /metrics or to any vLLM endpoint. In June I wrote the opposite in the inference router article and I was wrong.

What LiteLLM has is affinity, which is a different thing. Four pre-call checks pin a request to a replica by session identifier, by key hash, by previous response identifier or by exact hash of a marked prefix. None is active by default. Affinity distributes conversations; prefix routing distributes prefixes shared between different conversations, which is the case that multiplies throughput.

The only one that looks at the prompt is fragile in three ways. If the client does not send cache_control, the key comes out null and the check does nothing. If the cut point goes in the last turn, which is what the automatic injection does by default, the key changes on every turn and it never hits. And the pin’s TTL is 300 seconds hardcoded in two lines, while the provider offers one-hour caches.

The engine does not answer questions, it publishes events. There is no vLLM endpoint you can ask whether it has a prefix. What there is, is a ZMQ socket that emits stored and removed blocks in msgpack, switched off by default, with silent discarding when the consumer does not keep up. A router that wants to be exact has to replicate the engine’s hash in full: chained SHA256, deterministic seed, block granularity and the extra keys in their order.

And by default it does not publish the hits, only the additions. The index of a router that only listens drifts away from the engine’s real LRU without noticing. To see the hits you have to ask for it request by request with kv_cache_report_mode: full.

The three routers that do do it are very different from each other. The vLLM production-stack one indexes chunks of 128 raw characters in a trie that is never purged, models no eviction and dies on every restart. The llm-d one offers an approximate mode and an exact event-driven one, all its plugins are in alpha or beta, and its default tokeniser does not produce real tokens. NVIDIA Dynamo’s keeps an event-driven radix tree with a weighted cost function, and ships with the parameter that avoids starving a freshly scaled replica switched off.

The big improvements only appear in the best possible case. The two- and three-digit numbers are published by the projects themselves, with synthetic shared prefixes and a round-robin balancer as the reference. The most complete independent measurement, from Meta, obtains 2.3 times more throughput and also identifies load regimes where affinity leaves capacity at between half and two thirds of the reference. Another, on a multi-region deployment with 89 % reuse, measures single-digit improvements.

The correct architecture has two layers, not one. LiteLLM on top for identity, budget and audit, pointing at a single destination per model group. The KV-aware router underneath, inside the engine’s domain. Mixing it into a single piece is not an option that exists today.

You are here: between the gateway and the engine

This article occupies the gap between two layers that already have an article of their own. Above, the L7 switchboard and the gateway that has been operated since the first article in this track. Below, the fundamentals of the KV cache and hit rate engineering.

The gap is the piece that decides which replica each request goes to, knowing what each replica has in memory. It is the only function in the stack that cannot be resolved in the gateway or in the engine separately.

The analogy: the library with four counters

A library with four counters. Each librarian has in front of them a pile of the books they have consulted lately. When a reader comes in asking for the third volume of an encyclopaedia, if that volume is in the pile at counter two, serving them there takes seconds. At any other counter someone has to go down to the store room.

A normal balancer sends the reader to the counter with the shortest queue. It is the right decision if all the counters are interchangeable, and it is the wrong decision here, because they are not: what distinguishes them is not the queue, it is the pile.

Session affinity resolves one specific case of this. The returning reader is always sent to the same counter, because they will probably ask for the fourth volume of the same encyclopaedia. It works, and it does not resolve the general case: when a hundred different readers ask for the same third volume, session affinity spreads them across the four counters and four librarians go down to the store room for the same book.

Prefix routing is looking at what the reader asks for before assigning them a counter. It has two costs that the analogy makes visible straight away. The first is that somebody has to keep track of what is in each pile, and that bookkeeping takes work. The second is that if everybody asks for the third volume, counter two becomes the library’s only desk while the other three look on.

The correction: what I wrote in June

In the inference router article, on 2 June 2026, I closed the prefix routing section with this sentence: “vLLM Production Stack router implements it natively. NVIDIA Dynamo too. LiteLLM in its enterprise version has a beta.”

The first two hold up. The third is false and was no less false in June. While preparing the humans and agents article the first hint appeared, and full verification against 1.102.0 confirms it.

Case-insensitive search over litellm/, enterprise/ and gateway/, for the terms kv_aware, kv-aware, cache_aware, cache-aware, radix, prefix_cache, prefix-cache, kv_cache, block_hash, prefix_hash, prefix_score and longest_prefix: zero matches. Search for vllm inside litellm/router.py, litellm/router_strategy/ and litellm/router_utils/: zero. Search for num_requests_waiting, gpu_cache_usage or any scraping of the backends’ /metrics from the router: zero. LiteLLM’s router never asks the engine anything.

Search for premium_user in the files where those checks would live: a single result, and it is about something else. There is no paid version with this feature. The only routing piece that is behind a licence is per-tag budgets, in litellm/router_strategy/budget_limiter.py.

Where the error came from. LiteLLM’s documentation has an entry on dynamic TPM and RPM allocation marked as beta and enterprise, and a pre-call check called prompt_caching. Skim-read and added together, they give the impression of prefix routing in beta. Read in the code, prompt_caching is something else, and it deserves a section of its own because it is still recommended for this problem.

What LiteLLM does have: four affinities

The optional_pre_call_checks field accepts eight values. None is enabled by default: apply_default_settings passes an empty list. One of the eight, forward_client_headers_by_model_group, is declared in the type and in the interface schema and is consumed by no branch of the code. It is a dead value.

The four that pin a request to a replica are these.

Check	What it pins	Where it stores it	TTL
`session_affinity`	Session identifier	Redis with a Lua script, mirrored in memory	`deployment_affinity_ttl_seconds`, 3600 by default, refreshed
`deployment_affinity`	API key hash	Same	Same
`responses_api_deployment_check`	`previous_response_id`	Same	Same
`prompt_caching`	Exact hash of a marked prefix	`DualCache`	300 s, hardcoded

The other four are router_budget_limiting, encrypted_content_affinity, enforce_model_rate_limits and the dead value.

`prompt_caching`: why it is no use for this

The key is a SHA256 over the serialisation of the cacheable prefix, and “cacheable prefix” means everything up to and including the last block marked with cache_control: {"type": "ephemeral"}. Three consequences, checked by running the functions in isolation over the repository’s code.

If the request carries no cache_control, the extraction returns an empty list, the key comes out null and the check pins nothing. Automatic injection of cut points is only enabled with litellm.enable_anthropic_prompt_caching, which defaults to false, and only for Anthropic and Bedrock providers.

If the cut point is in the system prompt, the key does not change when turns are added and the pin works. This is the good case and it requires the client to mark it deliberately.

If the cut point is in the last message, the key changes on every turn and the pin never hits. And this is exactly the default case of the automatic injection, which places one point on the system and another at index -1. Since the second is the last, the cacheable prefix becomes the whole conversation.

On top of that there is the TTL. Two lines with ttl=300, one of them with the comment # store for 5 minutes. There is no parameter to change it. Issue 28427 has had it open for months with no response from the maintainers, and it points to the mismatch with the one-hour ephemeral caches that LiteLLM itself knows how to request via cache_control.ttl: "1h".

There is a nuance the issue does not capture and that makes diagnosis worse: in the default case the pin does not even last five minutes, because the key changes first.

`session_affinity`: the one to use

For conversations, this is the right piece and it is reasonably built. The session identifier is looked for in this exact order of headers:

x-litellm-trace-id
x-litellm-session-id
Any x-<vendor>-session-id, with the regular expression ^x-.+-session-id$ and a value of at least eight alphanumeric characters
Only if the User-Agent is Codex’s: session-id, session_id, thread-id, conversation_id
Plain x-session-id, for opencode

And if nothing matches, two fallbacks: a metadata.user_id in Anthropic format, and the W3C baggage with session.id=. The third point is the one that captures Claude Code with nothing configured.

Storage is a Lua script in Redis that does GET, SET NX EX and EXPIRE, so the first writer wins and the TTL is refreshed on every request that confirms the pin. It is an inactivity TTL, not a total session duration. Without Redis it degrades to a pod-local reservation, which is atomic but not shared.

A detail of the pipeline order that already came up in the previous article and still holds: the affinity checks run inside async_get_healthy_deployments before tag filtering. If the pin returns a single deployment and that deployment does not satisfy the request’s tag, the later filter removes it and the request fails. Affinity gets no vote on the tags.

The routers that do read the prompt, and why they are no good

LiteLLM has four pieces that read message content: AutoRouter with semantic matching, RequestComplexityRouter, AdaptiveRouter and QualityRouter. They all live in async_pre_routing_hook, which runs before looking for healthy deployments.

The reason they do not solve this problem is in what they choose. All four choose a model group, not a replica. They decide whether an easy question goes to the thirty-billion-parameter model or to the small one. None has access to the replica list or knows what is in each one’s memory.

What did land: admission control

A minor correction to the previous article, which placed this in the 1.101 branch as an upcoming feature. In 1.102.0 it is already there, in litellm/proxy/middleware/admission_control_middleware.py, with the three settings in general_settings: max_in_flight_requests_per_worker, max_queued_requests_per_worker and admission_queue_timeout_seconds, the last with a default value of 1.0 seconds.

It stays inactive as long as the first one is not declared. It exposes an authenticated /health/backlog and the three metrics litellm_admission_admitted_requests, litellm_admission_queued_requests and litellm_admission_rejected_requests_total, the last with a reason label. It is a per-process semaphore, not coordinated across pods.

What the engine exposes upwards

Before looking at the routers that do do this, you have to understand what raw material they work with. The natural question would be whether vLLM has an endpoint you can ask about a prefix. It does not.

The only cache router in the entry points is vllm/entrypoints/serve/dev/cache/api_router.py, and it only exposes destructive writes: POST /reset_prefix_cache, /reset_mm_cache and /reset_encoder_cache. All three are behind VLLM_SERVER_DEV_MODE=1 and do not exist in a normal deployment. There is no read.

What there is, is an event stream.

The ZMQ stream

It is enabled with --kv-events-config, and it is off by default. The configuration accepts a publisher (null or zmq), an endpoint (by default tcp://*:5557, which binds), an optional replay endpoint, the number of batches retained for that replay (10,000), and a high water mark limit of 100,000.

That limit is the one that matters operationally: above it, ZMQ discards events if the consumer does not keep up. Silently.

The transport is a PUB socket with msgpack serialisation and three-part frames: topic, eight-byte sequence number and payload. The sequence number lets a subscriber ask for replay from where it left off through the optional ROUTER socket. With data parallelism the port shifts by rank, so you have to subscribe to 5557+rank.

Three event types. BlockStored carries the hashes, the parent hash, the token identifiers, the block size, the LoRA adapter name, the extra keys, the medium (GPU, CPU or storage) and a session identifier. BlockRemoved carries only hashes. AllBlocksCleared carries no payload and is emitted when the prefix cache is reset.

They are published once per scheduler step, only if there are events. With the feature switched off the cost is zero. With it switched on, the full token identifiers of each block travel over the wire, so bandwidth scales with tokens per second and not with requests per second.

An example in vLLM’s own repository declares itself experimental on its eighth line.

The hash that has to be replicated

An exact router does not receive the engine’s state, it reconstructs it. For that it has to compute the same hashes, and the engine computes them like this:

# vllm/v1/core/kv_cache_utils.py
def hash_block_tokens(hash_function, parent_block_hash, curr_block_token_ids, extra_keys):
 if not parent_block_hash:
 parent_block_hash = NONE_HASH
 return hash_function((parent_block_hash, tuple(curr_block_token_ids), extra_keys))

It is chained: each hash identifies the complete prefix up to that boundary, not the isolated block. The default algorithm is SHA256, configurable with --prefix-caching-hash-algo among four values.

Five things a router has to get right for its hashes to match the engine’s:

The seed. Since 0.29, NONE_HASH is deterministic from a fixed seed, so two different processes produce the same hashes with nothing configured. It is a silent change of semantics from earlier versions, which required setting PYTHONHASHSEED. With the xxhash algorithms it is still needed.

The block size. By default 16 tokens, but the attention backend can override it if the user did not pass --block-size: 256 with sliding-window sparse attention, 64 with ROCm’s AITER, 64 or more with FlashAttention on XPU. A router must not assume 16, it has to read it from the event.

The hashing granularity, which is not necessarily the physical block size. There is --prefix-match-unit, which with several cache groups resolves to the greatest common divisor of the cacheable groups’ sizes.

The extra keys, in their order: LoRA adapter name (by name, not by identifier), multimodal identifiers with their relative offset inside the block, cache_salt only in block zero, and the hash of the prompt embeddings. Sampling parameters do not go in, which is correct: the prefix KV does not depend on the temperature.

And truncation. By default the hashes travel over the wire as 64-bit integers truncated from a SHA256, not as the full bytes. It can be disabled with VLLM_KV_EVENTS_USE_INT_BLOCK_HASHES=0. With the default value, a router has a theoretical collision risk that the engine does not have.

The hits are invisible by default

This is the detail that takes the most effort to discover and has the most consequences. The events are incremental: only what gets newly cached is announced. A reused block generates no event.

A router that only listens sees the additions and the removals, but does not see the hits. Its notion of what is recent and what is cold drifts away from the engine’s real LRU with nothing to warn it.

A full mode exists, and it is per request. sampling_params.extra_args["kv_cache_report_mode"] = "full" makes BlockStored events be emitted for blocks that were hits too. The function’s docstring says so plainly: it generates events so that external consumers, “for example a gateway”, know which blocks were reused. That is to say: for this to work, the router has to inject a parameter into every request it forwards.

The metrics, and what broke

For aggregate measurement two counters remain: vllm:prefix_cache_queries_total and vllm:prefix_cache_hits_total, both in tokens and not in requests, with model and engine labels. There are two equivalents for the external connector, which allow a hit in HBM to be distinguished from one brought in from CPU or NVMe.

And there are two removals that break existing configurations. vllm:gpu_prefix_cache_hit_rate no longer exists: zero appearances in the repository. It was deprecated in 0.8, hidden in 0.9 and removed in 0.10. And vllm:gpu_cache_usage_perc was renamed to vllm:kv_cache_usage_perc, which breaks Grafana panels and autoscalers that depended on the old name.

The hit rate has to be derived in PromQL with rate() over the two counters. The one that appears in the text log is something else: it uses a window of the last thousand requests.

And it has a bias you need to know about before taking decisions with it. When a request is evicted, its blocks return to the free queue keeping the hash, so the later reprefill finds them again and counts as a hit. A high hit rate can mean useful reuse between users or it can mean the engine is evicting and recomputing in a loop. The internal statistics separate the two things with a preempted field, which is not exposed as a Prometheus metric.

The last token

A small detail with an effect on long conversations: the maximum hit length is the request’s token count minus one. There is always something to recompute in order to get logits. Since the reservation requires alignment to the block size, in practice that can force recomputing the whole final block. With blocks of 16 it is noise. With a backend that prefers 256, it is up to 256 tokens recomputed per turn.

The ones that do do it

vLLM production-stack

Eight routing logics in a single --routing-logic selector: roundrobin, session, kvaware, loadaware, prefixaware, disaggregated_prefill, disaggregated_prefill_orchestrated and priority. Two of them, loadaware and priority, are not in the Helm chart’s enum and are only enabled through extraArgs.

The naming trap is that prefixaware and kvaware are radically different things.

prefixaware is a hash trie in the router’s process. It chunks the prompt string every 128 characters, not tokens, and hashes each chunk with 64-bit xxhash. Its own docstring says so: the chunk size is “in number of characters”. That 128 is hardcoded, there is no flag or Helm value to change it.

Four consequences come out of that. --prefix-min-match-length is measured in characters and is quantised to multiples of 128. A shared prefix that differs by one character inside the first chunk breaks the whole match, because there is no partial match inside a chunk. For chat, it concatenates the content of all the messages with newlines and without applying the chat template, so the indexed string is not what the engine tokenises. And the tie-break between tied replicas is random.choice, without looking at load.

The trie has no notion of saturation. A popular prefix concentrates all the traffic on the same replica indefinitely. The docstring assumes the simplified model and declares it: “we assume there is no prefix cache eviction”.

It has no pruning either. HashTrie only has insertion and longest prefix lookup; there is no deletion method. When a replica disappears, its URLs stay forever in every node’s sets, and filtering happens at query time. It works, and the trie grows monotonically: each distinct prompt adds its length divided by 128 permanent nodes, with no bound, no TTL and no LRU. The router’s memory limit in the chart is 1,000 MiB. By contrast, the round-robin router in the same file does bound its caches to 1,024 entries.

And it dies on every restart: it is in-process memory state, with no snapshot and no initial load.

kvaware and loadaware are better and have a different price. They do not index themselves: they ask the LMCache controller, which knows about real tokens. That requires lmcache==0.3.11 and vllm==0.13.0 pinned exactly, and the official router image drags in the whole of vLLM. The limitation to read before planning anything: they only look at request_json.get("prompt", ""), with an open TODO for chat completions. With /v1/chat/completions, both tokenise an empty string.

loadaware is the only one of the three that breaks affinity on saturation, with the formula score = relative_match - beta * relative_load and beta = 1.0 by default. The authors’ own reading: a replica at twice the average load loses the equivalent of a full cache hit. The guide suggests 0.25 to favour cache and 2.0 to favour balancing.

On published results, the documentation’s benchmarking file literally says that the measurement platform is coming soon. The project blog’s two posts claim between three and ten times lower response latency, and only one declares a partial configuration: Llama 3.1 70B on four nodes with tensor parallelism 2 over 80 GB A100s, inputs of 9K tokens and outputs of 10, against AIBrix 0.2.0 and a flat deployment. Charts with no labelled points, no absolute figures and no declaration of which routing logic was used in each.

llm-d

There has been a reorganisation here that invalidates a good part of the third-party documentation. The Endpoint Picker code left the Gateway API Inference Extension repository and was merged into llm-d’s, which has been renamed to llm-d Router. The Kubernetes project keeps the InferencePool API and a lightweight reference EPP that does nothing about prefixes. If you look for KV-aware routing in the extension’s repository, today it is not there.

The InferencePool is at v1 and is GA. The other two APIs moved to the llm-d.ai group at v1alpha2, and when they coexist with the old ones the EPP prefers the new ones and ignores the old.

Scoring is a weighted sum. The default weights, when no configuration is passed: queue 2, KV utilisation 2, prefix 3. Any scorer referenced without a weight counts as 1.0. There is no configuration through environment variables: the old ENABLE_*_SCORER no longer exist, everything goes in an EndpointPickerConfig in YAML.

The prefix scorer consumes an attribute published by a producer, and there are two producers.

The approximate one computes the hashes in the EPP itself without talking to the engine, and keeps a local LRU per pod and block pair that is populated after the decision. It is an approximation by construction. It has a hard floor: a configured block size below 64 tokens is raised to 64 at request time, because the index stores one entry per pod and block and going down to 16 would multiply memory by four.

The exact one builds the real index from the engine’s ZMQ events, with one subscriber per pod that is installed and torn down with the endpoint’s registration and deregistration. It has adapters for vLLM and SGLang, deduplicates evictions by reference counting, and the replay buffer that allows the index to be rebuilt after an EPP restart requires vLLM 0.26.0 or above.

The trap is in the tokeniser. The EPP tokenises through a plugin, and the default backend is estimate: byte packing with no tokeniser, roughly 4 bytes per token, with identifiers that do not correspond to real engine tokens. The exact scorer needs real tokens, and for that you have to explicitly configure the vllm backend, which does HTTP against the engine’s /v1/completions/render endpoints. If it is omitted, the auto-created estimate satisfies the dependency and the system degrades silently.

On maturity, the file itself declares it: all in-tree plugins are in alpha or beta, and they will be promoted to stable when the project approaches 1.0. An alpha plugin configured without the corresponding flag makes the EPP fail to start.

And on replicas, the warning is explicit: active-active mode should be avoided with approximate prefix routing, because the EPP replicas do not share state and each one only sees the traffic it has served itself, which significantly degrades the hit rate. The exact index is safe in high availability, and even so the reference guides fix a single replica because the accounting of in-flight requests is local to the process.

The cost is the best documented in the whole field, and it deserves a table. Simulator with Qwen3 8B, 100 pods, 100K input tokens:

Requests/s	Cap on tokens to match	Peak CPU	Peak memory
5.0	4,096	1.19 cores	0.26 GiB
5.0	100,000	3.82 cores	0.65 GiB
98.7	4,096	35.17 cores	2.46 GiB
98.8	100,000	46.50 cores	3.41 GiB

With outputs of 10K tokens at 50 requests per second, 32.53 cores and 12.54 GiB. The maintainers’ sizing rule is half a core to one core per request per second on large agentic loads. And idle consumption scales with the number of pods because of continuous scraping: with 100 pods, about 7.5 cores with no traffic.

NVIDIA Dynamo

A precise event-driven index, with a global radix tree in each frontend’s process. Since all the replicas consume the same event plane, there is no need to synchronise routers with each other for the prefix state.

The cost function is explicit and weighted. It discounts from the prefill the blocks that overlap in GPU, host and disk with different weights (1.0 for the overlap credit, 0.75 for host, 0.25 for disk), and adds the potential decode blocks and the active requests.

There is a parameter to look at before deploying: --router-kv-overlap-score-credit-decay is at 0 by default, that is, disabled. Its description says what it is for: preventing loaded, cache-rich replicas from winning again and again while the freshly autoscaled ones receive too little traffic. With the default value, a new pod has no cache, therefore never wins the scoring, therefore never generates cache.

The approximate mode exists (--no-router-kv-events, with a TTL of 120 seconds) and the documentation says it is not the recommended route in production.

Dynamo is the most honest project in its measurement methodology: its comparison guide recommends contrasting --router-mode random against kv over the public Mooncake trace from FAST'25, with a declared cache ratio of 59 %, and it generates the table locally instead of publishing a number.

The rest of the field

SGLang has renamed its router to sgl-model-gateway, version 0.3.2 within SGLang 0.5.19. Its cache_aware policy is an approximate radix tree over raw characters, with switching to shortest queue when the system is unbalanced according to two combined thresholds. One detail that can cost an afternoon: the Rust defaults and the Python CLI defaults do not match, and on the maximum tree size they differ by four orders of magnitude (10,000 nodes against 2^26).

AIBrix 0.7.0 offers a standard mode with a local hash table and a KV event synchronisation mode behind a flag and a build tag, with vLLM 0.7.0 or above.

Ray Serve LLM has PrefixCacheAffinityRouter with an explicit alpha API warning, a character-based prefix tree in a decoupled actor, and its imbalance threshold at infinity by default, that is, out of the box it never breaks affinity on load.

Envoy AI Gateway 1.1.0 implements none of this and delegates to InferencePool and the EPP. KServe the same, with its LLMInferenceService at v1alpha1. Mooncake Conductor is still an unimplemented proposal.

Comparison table

Project	Version and date	Index	Where it lives	Kubernetes	Declared maturity
production-stack `prefixaware`	main branch, 09-Sep-2026	Approximate, character trie, no eviction	Router process	Optional	WIP in the README
production-stack `kvaware`/`loadaware`	same	Exact via LMCache controller	LMCache controller	Optional	No chat completions
llm-d Router approximate	v0.10.0, 17-Aug-2026	Approximate, LRU per pod	EPP process	Yes	Beta, active-active discouraged
llm-d Router exact	same	Exact through ZMQ events	EPP process	Yes	Beta
NVIDIA Dynamo	v1.4.2, 27-Aug-2026	Exact, event-driven radix tree	Frontend process	No	GA with experimental subsystems
SGLang `sgl-model-gateway`	0.3.2, 03-Sep-2026	Approximate, character radix	Gateway process	No	Default policy
AIBrix	v0.7.0, 16-Jun-2026	Both	Gateway plugin	Yes	Synchronisation behind a flag
Ray Serve LLM	Ray 2.58	Approximate, characters	Ray actor	No	Declared alpha
Envoy AI Gateway	v1.1.0, 21-Aug-2026	None of its own	Delegates to EPP	Yes	Stable 1.x API

How much it really improves

Here two types of source have to be separated.

The projects’ own numbers. Red Hat published the llm-d case in May: Qwen3 32B, eight vLLM pods over 16 H100s with tensor parallelism 2, synthetic shared-prefix load with 150 groups of five prompts, 6,000 tokens of system prompt, 1,200 of question and 1,000 of output. Result: up to 109 % more throughput and up to 99 % less TTFT, around 200 concurrent users within SLO against around 20. The repository’s result files give more detail: peak throughput from 6,986 to 14,892 output tokens per second, p90 TTFT from 135.5 seconds to 0.26.

That same file publishes the regression, and that is why it is worth reading: p50 inter-token latency rises by 22.4 % with vLLM and by 45.4 % with SGLang. The trade-off is declared: routing by affinity concentrates more concurrent work on the pods with a warm cache.

It is the best possible case. A synthetic shared prefix and a reference that is a flat Kubernetes Service.

The independent measurements. There are three and they say different things.

CacheRoute, from Meta, August 2026, is the most complete. Llama 3.3 70B in fp8, 30 targets with tensor parallelism 2 over 60 H100s, a semi-synthetic trace from production telemetry with 128,824 business keys, and all the alternatives reimplemented in the same harness instead of comparing against a dumb balancer. It obtains 176 requests per second with a p99 SLO of 3.5 seconds against 76 for the best competitor, and a hit rate of 93.2 % against 72.0 %.

And it publishes two negative findings that nobody else publishes. There are load regimes where affinity reduces capacity to between half and two thirds of the reference, to the point that the authors make a mirrored-traffic trial mandatory before deploying. And growing the fleet can cool a prefix even though total cache capacity grows, because the time between revisits stretches beyond the eviction window. Scaling horizontally can make the hit rate worse.

GORGO, June 2026, measures on a multi-region deployment with a dataset where intra-user prefix reuse is 89.4 % and the average prompt is almost 18,000 tokens. An improvement of 6.9 to 15.5 % in p95 TTFT. With extremely high reuse, the improvement is single-digit, because network latency and queuing dominate.

LAAR, from IBM Research Tokyo, April 2026, is the uncomfortable counterpoint: in its comparison, session affinity routing was the worst of the baselines, and at 64K contexts a load-based routing gives lower absolute latency at the cost of fewer hits.

The joint reading fits in three conditions. The big improvements require high, concentrated prefix reuse, long prompts, and a weak reference. Take away any of the three and the numbers deflate.

The five failure modes

Hot spot from affinity. The busiest bucket inherits the load’s skew, and tail latency starts following the hottest target instead of the fleet average. Mitigations exist in the four main products and in three of the four they ship switched off: Dynamo’s credit decay at 0, Ray’s imbalance threshold at infinity, and production-stack’s prefixaware with none at all.

Starvation of the freshly scaled pod. A new pod has no cache, so it never wins the scoring, so it never warms up. Dynamo documents it as the reason for the decay parameter. I found no primary documentation from any of the projects about the specific interaction with KEDA.

Cost of the index. The llm-d numbers in the previous table are the best public reference that exists. Raising the cap on tokens to match from 16,384 to 400,000 can more than double the EPP’s CPU with little traffic.

Traffic with unique prefixes. Each project degrades to something different. SGLang routes to the smallest tree, which is not the same as the least loaded. Ray falls back to power of two choices. Dynamo has an explicit preset, --load-aware, to say “I want the load model without reuse”. And production-stack’s prefixaware keeps inserting into the trie, that is, keeps paying the cost while receiving nothing.

Sibling bursts. Parallel sampling, best of N, or an agent that opens five branches at once. With event-driven routing, no engine has yet emitted the block stored event, all the siblings score zero overlap, and the prefix is precomputed on every replica. Dynamo has a prediction TTL for this; the others do not.

The architecture that comes out of all of the above

Two layers, with the responsibilities separated.

On top, LiteLLM, with what it does do well: non-forgeable identity through the virtual key, budgets, spend logging, concurrency limits, fallback between providers and API unification. For the gateway, each model group has a single destination: the URL of the router below.

model_list:
 - model_name: qwen-30b
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://kv-router.inferencia.svc:8000/v1
 api_key: os.environ/VLLM_KEY

router_settings:
 routing_strategy: simple-shuffle
 num_retries: 1
 optional_pre_call_checks: ["session_affinity"]
 deployment_affinity_ttl_seconds: 3600

general_settings:
 max_in_flight_requests_per_worker: 64
 max_queued_requests_per_worker: 64
 admission_queue_timeout_seconds: 1.0

Below, the KV-aware router, inside the engine’s domain, talking to the vLLM replicas and consuming their events.

Four consequences of this separation that have to be accepted before building it.

Per-replica visibility in LiteLLM is lost. For the gateway there is a single deployment, so its cooldowns, its per-deployment counters and its strategies stop meaning anything. Observability of which replica served what goes downstairs.

Retries stack up. The client SDK retries, LiteLLM retries, and the router below may have its own failover. In the previous article 45 calls per turn came out with three layers; with four it is worse. Lowering num_retries in the gateway to 1, or to 0, is the sensible decision when there is a router below that already retries.

LiteLLM’s session affinity stops making sense for the KV, because it no longer chooses a replica. It is kept if it is used for something else, such as pinning a conversation to a specific provider.

And the choice of the router below conditions the rest of the cluster far more than the gateway’s does. llm-d requires Kubernetes 1.32 or above, Gateway API, the inference extension and a conformant Gateway. Dynamo does not require Kubernetes. production-stack has a Helm chart and works outside it. That decision is the expensive one, not the gateway’s.

When not to build this

Three cases where the correct answer is not to do it, at least not yet.

A single replica per model. With one replica, the engine’s prefix cache already does all the work and there is nothing to route. That is the case for a good part of the four H100 deployments with one large model and tensor parallelism.

Low or scattered prefix reuse. Before building anything, the measurement is rate(vllm:prefix_cache_hits_total[5m]) / rate(vllm:prefix_cache_queries_total[5m]) per replica, with a mental correction for the bias from reprefills after eviction. If it is already high with blind distribution, reuse is concentrated inside each conversation, and there LiteLLM’s session affinity costs one line of configuration and resolves the case.

Traffic where the network dominates. The GORGO result is the warning: with enormous prompts and a distributed deployment, the improvement is eaten up by queuing and network latency.

The case where it does pay off is specific: several replicas of the same model, a large system prompt shared between different users, or a RAG with a corpus of documents that repeat across requests. There the prefill saved is real and it is most of the work.

Checklist

Measure the current hit rate per replica with the two vLLM counters before touching anything, and check whether the engine is evicting with vllm:num_preemptions_total.
Check that prefix caching is switched on. It is no longer --enable-prefix-caching that switches it on: the default value is derived from the model and the log of the decision is at DEBUG level. What is actionable today is --no-enable-prefix-caching to switch it off.
Audit the prompts before routing: a timestamp or a session identifier at the start of the system prompt breaks the match from block zero and no routing layer fixes it.
If the KV event stream is enabled, size the bandwidth by tokens per second and watch discarding at the high water mark. With data parallelism, subscribe to one port per rank.
If an exact router is chosen, verify the vLLM version against the one the replay buffer requires.
If llm-d is chosen, explicitly configure the vllm tokenisation backend. The default estimate does not fail, it degrades.
If production-stack is chosen with kvaware or loadaware, check first whether the traffic goes through /v1/chat/completions, because today it does not work there.
Review the load-based affinity-breaking parameter, because in three of the four main projects it ships disabled.
Lower num_retries in LiteLLM when there is a router below that already does failover.
Trial with mirrored traffic before changing production. It is the explicit recommendation of the only independent measurement that went looking for bad regimes, and found them.

Traps and things that are not what they seem

LiteLLM has no prefix routing in any version. Neither the open one nor the paid one. I got it wrong in June.
prompt_caching does nothing if the client does not send cache_control, and with the default automatic injection it fails on every turn because it marks the last message.
That pin’s TTL is 300 hardcoded seconds, incompatible with the one-hour caches that LiteLLM itself knows how to request.
production-stack’s prefixaware indexes characters, not tokens, in non-configurable chunks of 128, and its trie is never purged and models no eviction.
production-stack’s kvaware and loadaware do not support chat completions. They tokenise an empty string.
The default tokeniser in llm-d’s EPP is a byte estimate, and with it the exact scorer degrades silently.
llm-d’s active-active mode is discouraged with approximate prefixes, and the reference guides fix one replica even with the exact one.
Dynamo’s overlap credit decay ships at zero, which can leave a freshly autoscaled replica with no traffic.
The SGLang router’s default values differ between the Rust and the Python CLI, by as much as four orders of magnitude.
vllm:gpu_prefix_cache_hit_rate no longer exists and vllm:gpu_cache_usage_perc is now called vllm:kv_cache_usage_perc. Old panels and autoscalers are measuring nothing.
The hit rate counts the reprefill after eviction as a hit. A high number can be reuse or it can be thrashing.
KV events do not publish the hits by default, only the additions, unless full mode is requested request by request.
The hashes travel truncated to 64 bits unless it is disabled with an environment variable.
The default block size is 16, and the attention backend can change it without anyone asking.
The Endpoint Picker left the Gateway API Inference Extension repository. Looking for it there today leads nowhere.

Closing

June’s error has a reading that goes beyond the erratum. It came from reading a product’s documentation and not its code, and from the fact that LiteLLM’s documentation uses the word “caching” in a pre-call check that does something else. The distance between what a name suggests and what a function does is, at this point in the track, the most repeated pattern of the whole series.

The architectural conclusion is that KV-aware routing is not a gateway function. It is not one today in LiteLLM and probably should not be: it requires keeping an index synchronised with each engine’s memory, replicating its hash function, consuming an event stream and sizing CPU per request per second. That belongs to the engine’s domain, not to that of identity and budget.

What remains is a two-part decision. The first is whether the traffic deserves it, and that is answered with two Prometheus counters and without installing anything. The second is which of the three routers fits the cluster you already have, and there the deciding variable is not the hit rate but whether you are willing to bring in Gateway API and the inference extension, or to pin vLLM 0.13.0, or to operate a Dynamo frontend.

And the honesty the material demands: the only independent measurement that specifically went looking for regimes where this makes things worse found them, and they are not marginal. Half to two thirds of reference capacity is a serious regression. That no project publishes that side of the curve does not mean it does not exist in theirs.

Sources

LiteLLM, Router - Load Balancing (estrategias, comprobaciones previas, afinidad): https://docs.litellm.ai/docs/routing.
LiteLLM, código: litellm/router.py, litellm/router_utils/prompt_caching_cache.py, litellm/router_utils/pre_call_checks/prompt_caching_deployment_check.py, litellm/router_utils/pre_call_checks/deployment_affinity_check.py, litellm/proxy/litellm_pre_call_utils.py, litellm/proxy/middleware/admission_control_middleware.py: https://github.com/BerriAI/litellm.
LiteLLM, incidencia 28427, TTL de afinidad por caché de prompts fijado a cinco minutos: https://github.com/BerriAI/litellm/issues/28427.
vLLM, código: vllm/v1/core/kv_cache_utils.py, vllm/v1/core/block_pool.py, vllm/v1/core/kv_cache_manager.py, vllm/distributed/kv_events.py, vllm/config/kv_events.py, vllm/v1/metrics/loggers.py, vllm/v1/metrics/stats.py: https://github.com/vllm-project/vllm.
vLLM, KV offloading usage (niveles CPU, filesystem y objeto; compartición entre procesos): https://docs.vllm.ai/en/latest/features/kv_offloading_usage.html.
vLLM, notas de la versión 0.29.0 (semilla determinista, seguimiento de propiedad en eventos, flags de admisión): https://github.com/vllm-project/vllm/releases/tag/v0.29.0.
vLLM, RFC 16669, publicación de bloques KV y métricas: https://github.com/vllm-project/vllm/issues/16669.
vLLM production-stack, código: src/vllm_router/routers/routing_logic.py, src/vllm_router/prefix/hashtrie.py, src/vllm_router/parsers/parser.py: https://github.com/vllm-project/production-stack.
LMCache Lab, resultados de production-stack frente a AIBrix (parte interesada): https://blog.lmcache.ai/en/2025/03/06/open-source-llm-inference-cluster-performing-10x-faster-than-sota-oss-solution/.
llm-d Router, código y documentación de operaciones (pesos por defecto, dimensionamiento del EPP, aviso de activo-activo): https://github.com/llm-d/llm-d-router.
llm-d, guía de enrutado preciso por prefijo y resultados publicados (parte interesada): https://github.com/llm-d/llm-d.
llm-d Router, incidencia 1290, estado de prefijo no compartido entre réplicas del EPP: https://github.com/llm-d/llm-d-router/issues/1290.
Red Hat, Same 16 GPUs, twice the users (parte interesada): https://www.redhat.com/en/blog/same-16-gpus-twice-users-inference-aware-routing-llm-clusters.
Kubernetes SIG Network, Gateway API Inference Extension (traslado del EPP y del BBR): https://github.com/kubernetes-sigs/gateway-api-inference-extension.
NVIDIA Dynamo, diseño y ajuste del router KV: https://github.com/ai-dynamo/dynamo.
SGLang, sgl-model-gateway: https://github.com/sgl-project/sglang/tree/main/sgl-model-gateway.
AIBrix, enrutado por caché de prefijo: https://github.com/vllm-project/aibrix/blob/main/pkg/plugins/gateway/algorithms/prefix_cache_readme.md.
Ray Serve LLM, enrutado consciente de prefijo (API en alfa): https://docs.ray.io/en/latest/serve/llm/user-guides/prefix-aware-routing.html.
Huang Cheng (Meta), CacheRoute, agosto de 2026: https://arxiv.org/html/2608.19677.
Yuan et al., DualMap, febrero de 2026: https://arxiv.org/abs/2602.06502.
Ricci Toniolo et al., GORGO, junio de 2026: https://arxiv.org/html/2602.11688.
Yoshimura, Chiba y van de Beek, Accuracy Is Speed (EuroMLSys ‘26), abril de 2026: https://arxiv.org/html/2604.15732v1.

Humans and agents on the same gateway: why `priority` does not prioritise, the 429 that brings down the whole pool, and the two ways to separate traffic

Thu, 10 Sep 2026 08:00:00 +0200

Fourth article in the operational track of the control layer. The pair with Langfuse covered observability, the proxy’s day 2 covered availability and virtual keys covered governance. Here comes the new tenant: agents, which speak the same protocol as humans and behave in a different way. Everything that follows is verified against LiteLLM 1.100.1 and vLLM 0.29.0, both from the first ten days of September 2026.

TL;DR

The day a team connects an agent to the corporate gateway, the platform changes regime without anyone touching a YAML. These are the seven conclusions.

LiteLLM’s priority parameter prioritises nothing under normal conditions. It is still marked as beta and its queue is only consulted when all the deployments in the group are in cooldown. With free capacity, the only thing it produces is a response header. On top of that, the success branch does not pop the item off the queue, so the list grows indefinitely; and on /chat/completions the value 0, which is the one used by all the documentation examples, is falsy in Python and disables queuing.

Separating traffic is done with two pools, not with priorities. The non-forgeable way is model_group_alias at key or team level: the client always asks for qwen-30b and the credential decides whether that resolves to qwen-30b-interactivo or to qwen-30b-agentes. The way that does not touch clients is tag_regex against the User-Agent, with a pattern like ^User-Agent: claude-code\/, which serves to classify load and not to control access, because that header is set by the client.

The useful admission control arrives in the 1.101 branch, with max_in_flight_requests_per_worker and a per-process queue that returns 503 with retry-after. It rejects before authenticating, so what it discards does not appear attributed to any key.

max_parallel_requests is only applied at virtual key level. The team, user, end customer and organisation endpoints accept the field and store it in the database, and nobody reads it. global_max_parallel_requests is a no-op unless the old limiter is reactivated, and general_settings.max_parallel_requests is read by no code path.

vLLM with --scheduling-policy priority does not push ahead anyone who is already generating. The only thing that changes is who it evicts when KV cache runs short. If the agents have filled max_num_seqs, a chat with the highest priority waits just the same. And the two knobs that in 2025 allowed short prompts to get ahead of long ones, max_num_partial_prefills and max_long_partial_prefills, disappeared from the code in 0.27.

The retry product reaches 45 calls per agent turn. Three from the client SDK times three from the router times up to five fallback groups. The proxy sets max_retries=0 on the SDK it uses itself, but it cannot touch the client’s. The timeout is per attempt, not per request, and LiteLLM’s Timeout comes out as a 408, which is retryable.

A single 429 takes a deployment out of the pool for five seconds, without going through the allowed_fails threshold. With an agentic burst against a fleet of four replicas, the whole pool enters cooldown and the gateway returns a 429 of its own to everybody, humans included.

You are here: the gateway layer with two classes of traffic

In the seven-layer stack this still lives in the gateway layer, with one difference compared to the previous three articles. There the variable was load: more requests, more teams, more budget to share out. Here the variable is the shape of the load.

A gateway sized for three hundred engineers using a chat works. The same gateway, with thirty of those engineers running coding agents, serves a fraction of the human traffic with the previous latency and the rest degrades. The GPUs stay just as busy, the utilisation metrics stay good, and the support team gets complaints from people waiting nine seconds for a paragraph.

The analogy: the registry desk and the courier with two hundred files

A registry office serves people in order of arrival. Citizens come in one at a time, present a piece of paper, and leave. The system is fair and the queue moves.

One Tuesday a courier turns up with a trolley of two hundred files. He joins the queue like anyone else, because rules are rules, and when his turn comes he occupies the desk for forty minutes. Behind him are twelve people with one piece of paper each. Nobody has broken any rule. The courier has the same right as everyone else and uses the same procedure. What fails is that the office was designed assuming a certain size of transaction and another one three orders of magnitude bigger has turned up.

The real solutions for an office are three, and all three have an exact equivalent in the platform. Open a desk for bulk deliveries, which is separating pools. Put up a capacity sign and send people back later, which is admission control with its 503. And say that files are handed in during the afternoon, which is the part that on an inference platform can almost never be applied, because the agent does not negotiate schedules.

What does not work in the office is giving the citizen a priority number when the courier is already at the desk. That number does not interrupt the transaction in progress. It is, literally, what priority does in the two layers of the platform, and it is worth keeping in mind while reading the next section.

The two kinds of traffic, measured

Before configuring anything, you have to be able to state the difference in numbers, because every later adjustment comes out of it.

Trait	Human chat	Agentic loop
Input per request	hundreds of tokens	tens of thousands, and it grows with each turn
Output per request	hundreds of tokens	from two tokens (a tool call) to several thousand
Concurrency per user	1	3 to 20 requests in flight
The metric that hurts	time to first token	total turn throughput
Tolerance for waiting	seconds	minutes
Cancelled requests	rare	routine, by the agent’s own decision
Retries	the browser’s	the SDK’s, plus the loop’s
Witness when it fails	the user	nobody

Two rows deserve comment because they contradict intuition.

The first is the metric row. An agent does not perceive time to first token, so on a shared engine it is the natural candidate to give up that metric. The operational conclusion is that the agent pool can be configured for aggressive throughput, with large batches and deep queues, while the interactive pool is configured for the opposite.

The second is the cancellations row. An agent that launches three speculative continuations and discards two generates two aborted streams per turn. At the gateway, that touches spend accounting, the release of concurrency counters and the open connection to the engine. LiteLLM 1.100 handles it reasonably well, and it did not six months ago.

What does not work: `priority` in LiteLLM

The first reaction of any operator is to look for a priority field. LiteLLM has one, documents it on its Request Prioritization page, and that page carries the beta label and the sentence saying it is for testing. The label is correct and has to be taken literally, because behind it there are four things.

The queue is only consulted when the whole group is in cooldown. The scheduler’s poll() method returns true if there is any healthy deployment, without touching the queue. With available capacity, which is the normal situation, the request goes straight through and priority orders nothing. The only thing that changes is that the response carries the x-litellm-request-prioritization-used header.

The queue does not empty. In that success branch there is no heappop and no call to the cleanup, which only happens in the expiry branch. The list under the key scheduler:queue:{model} grows without bound, is serialised whole into Redis on every insertion, and there is no maximum size and no discard policy. When the group does enter cooldown, the head of the heap is occupied by fossil entries from requests that finished hours ago, and new ones compare their identifier against that head, never match, and wait until the timeout runs out.

On /chat/completions, the highest priority disables the feature. The code reads the value with a kwargs.get("priority") or self.default_priority, and in Python 0 or None is None. Since in this scheduler the lowest number is the highest priority, priority: 0, which is the value in the documentation’s own examples, goes down the path without queuing. On /completions the check is done with is not None and it works; the two routes do not behave the same.

And default_priority in router_settings breaks the request. If the client does not send a priority but the global setting is in place, the value is truthy, the scheduling branch is entered and a function is invoked that requires the positional argument nobody has injected. The setting has no associated test and does not appear in the router_settings reference.

There is also a /queue/chat/completions, marked as experimental and hidden from the OpenAPI schema, which reads data["priority"] with no default value: a request without that field raises an exception that comes out as a 400 authentication error, with a message that has no relation to the cause.

There is a priority feature that is solid, and it is a different thing. The v3 dynamic limiter shares out token and request per minute quota between classes with weights, through litellm.priority_reservation with values like {"premium": 0.75, "standard": 0.25}. It shares out capacity, it does not order a queue: below the saturation threshold it lends free capacity and above it applies the weights. It never makes anyone wait, either it passes or it returns a 429. It requires Postgres and an enterprise licence, and it is still marked as beta.

What does work: two pools

Effective separation is done earlier, by deciding which set of replicas each class of traffic goes to. LiteLLM offers two mechanisms with very different security properties.

By credential: `model_group_alias` at key or team level

Router settings are resolved in the order key, team, global. Among those that accept that override is model_group_alias, and there lies the piece:

model_list:
 - model_name: qwen-30b-interactivo
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-chat.inferencia.svc:8000/v1
 model_info:
 id: chat-01
 max_input_tokens: 32768
 - model_name: qwen-30b-agentes
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-agentes.inferencia.svc:8000/v1
 model_info:
 id: agentes-01
 max_input_tokens: 131072

And on the agent team’s key:

curl -X POST "$PROXY/key/generate" \
 -H "Authorization: Bearer $MASTER_KEY" \
 -d '{
 "team_id": "plataforma-agentes",
 "models": ["qwen-30b"],
 "router_settings": {
 "model_group_alias": {"qwen-30b": "qwen-30b-agentes"}
 }
 }'

The client asks for qwen-30b and knows nothing about any of it. The resolution is tied to the credential, so the caller cannot get around it by changing a header. The same list allows routing_strategy, fallbacks, context_window_fallbacks, retry_policy, cooldown_time and allowed_fails to be set per key or team, which gives different reliability policies for each class of traffic on the same proxy instance.

It should not be confused with access groups (model_info.access_groups), which are permissions and not routing: they decide whether a key can invoke a model name, not which replica the request goes to.

By `User-Agent`: `tag_regex`

When separate credentials cannot be issued, because the agentic client uses the same personal key as that person’s chat, LiteLLM classifies by regular expression against the request headers:

model_list:
 - model_name: qwen-30b
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-agentes.inferencia.svc:8000/v1
 tag_regex: ["^User-Agent: claude-code\\/", "^User-Agent: .*codex"]
 model_info: {id: agentes-01}
 - model_name: qwen-30b
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-chat.inferencia.svc:8000/v1
 tags: ["default"]
 model_info: {id: chat-01}

router_settings:
 enable_tag_filtering: true
 tag_filtering_match_any: true

The proxy composes strings of the form User-Agent: <value> and tests them with re.search. Exact tag matching takes precedence over the regex. The documentation itself carries a warning that has to be respected: the User-Agent is written by the client, so this is traffic classification and not a security boundary. As load sharing it is enough, because an agent that lies about its identity to sneak into the interactive pool is a governance problem, not a routing one.

The behaviour when no deployment carries the requested tag has a nuance that comes as a surprise in production. If the tag is unknown to the group, the request falls through to the pool marked with tags: ["default"]. If the tag exists in the group but no healthy deployment carries it at that moment, the request fails with the tag configuration error, even though a default pool is available. To get deterministic fallthrough to the default pool you have to set model_info.allow_fail_open: true.

A different strategy per pool with `routing_groups`

router_settings:
 routing_strategy: simple-shuffle
 routing_groups:
 - group_name: interactivo
 models: [qwen-30b-interactivo]
 routing_strategy: simple-shuffle
 - group_name: agentes
 models: [qwen-30b-agentes]
 routing_strategy: simple-shuffle

The recommendation to leave simple-shuffle on both is not laziness, and sustaining it requires looking at what the alternatives do on a homogeneous vLLM fleet.

least-busy keeps a complete dictionary under a single key and updates it with non-atomic reads and writes, without using INCR, so with several proxy replicas increments are lost. It has no TTL, so a request that never completes leaves the counter raised forever and that replica is permanently excluded. Counters can go negative. And when the minimum corresponds to an identifier that is no longer in the healthy list, it falls back to random.choice without warning. The pattern of requests that do not complete is exactly that of an agent that cancels.

latency-based-routing measures seconds per output token, not wall-clock latency. A chat turn of twenty tokens looks extremely slow by that metric and an agent generation of two thousand tokens looks extremely fast, so mixing both kinds of traffic in the same group poisons the signal. It also seeds new deployments with zero latency, and with the buffer defaulting to zero, a freshly started replica takes all the traffic until its first success callback.

usage-based-routing-v2 uses minute windows that reset all at once, and it accounts for tokens when the request completes, so long generations in flight are invisible to the router. With long outputs, it systematically underestimates the real load.

cost-based-routing sorts by the sum of the input and output unit prices and takes the first, with no random tie-break. With replicas of the same model at the same price, all the traffic always goes to the same place. And a model that is not in the price map gets a default cost of 5.0 per token, a figure designed to deprioritise it.

Admission control: the new piece in the 1.101 branch

When the agent pool saturates, what you want is to reject fast and with a code the client understands, not to accumulate requests inside the proxy until memory runs out. That arrives with per-worker admission control, added in the 1.101 branch, in release candidate as this is written.

general_settings:
 max_in_flight_requests_per_worker: 64
 max_queued_requests_per_worker: 64
 admission_queue_timeout_seconds: 1.0

It is a semaphore with a per-uvicorn-process queue, without Redis. When the queue fills or the wait expires it returns 503 with retry-after: 1 and a body with "type": "overloaded_error". It is observed through /health/backlog, which exposes requests in flight, admitted, queued and rejected, and through the Prometheus litellm_admission_* metrics, whose rejection counter distinguishes queue_full from queue_timeout.

Two properties have to go into the runbook. The middleware rejects before authenticating, so what is discarded is not attributed to any key in the spend records, and the real number of requests rejected per team cannot be reconstructed from the spend table. And the limits are read on the first request: changing them requires a restart.

Concurrency per key, and the three settings that do nothing

max_parallel_requests is the right brake for a specific agent, and its configuration surface is full of dead ends.

The only level where it is applied is the virtual key. The v3 limiter, active by default since 1.94, has descriptors for user, team, team member, end customer, organisation, model per key, tag and agent, and all of them carry only request and token counters per window. The only reference to max_parallel_requests is the one in the key descriptor. The old limiter says so in a comment: supporting it for model, user and team is pending.

At the same time, /team/new, /user/new, /customer/new and the organisation endpoints accept the field, validate it and write it to Postgres, and the documentation shows it in their examples. Nobody reads it afterwards. A team with max_parallel_requests: 20 in its row has no concurrency limit at all.

global_max_parallel_requests is injected into the request metadata and is only consulted by the old limiter, which is disabled unless LEGACY_MULTI_INSTANCE_RATE_LIMITING=true is set. The server settings documentation still describes it as a global limit coordinated through Redis. And general_settings.max_parallel_requests does not appear in any execution path: its only two appearances are the copy from the database and the type schema of the interface.

What does work, in detail:

curl -X POST "$PROXY/key/generate" -H "Authorization: Bearer $MASTER_KEY" \
 -d '{"team_id":"plataforma-agentes","max_parallel_requests":8,"rpm_limit":600}'

Counting is done with a sorted set in Redis scored by Redis’s own clock, with Lua scripts to acquire and release. Each slot expires after 3,600 seconds, which is at once the maximum request duration the meter can track and the time a leaked slot takes to heal by itself. Release is hooked to success, to failure and to client disconnection during streaming, which was the classic leak with clients that cancel.

One live leak remains and it affects exactly the case in this article: every tool call through the MCP gateway acquires a slot and does not release it. The MCP route invokes the pre-call hook, which acquires, and none of the release functions is called from that module. With an agent that makes twenty tool calls per turn and a key with eight slots, the 429 arrives on the first turn and lasts an hour.

The order of evaluation matters for interpreting 429s. First the window counters, requests and tokens per minute, are checked, and only if they pass is the concurrency slot acquired. The code comment explains why: the other way round, every rate limit rejection would leave an orphan slot. In practice, with a low rpm you always see the requests type 429, and the max_parallel_requests one only appears when the relationship between concurrency and duration makes it binding first. The message distinguishes the two cases with the Limit type: field and the rate_limit_type header.

A minor detail with consequences on the client: the response to a concurrency rejection carries retry-after and reset_at computed from the window size, sixty seconds by default, when a concurrency meter has no window. A well-behaved client that respects that header will wait a minute when it might have had a free slot in two seconds.

Session affinity: the KV cache that gets thrown away

An agent turn resends the whole conversation context, system prompt and tools included. If that request lands on a different replica from the previous turn’s, vLLM’s prefix cache does not have the prefix and the prefill is recomputed in full. With twenty turns and four replicas balanced blindly, the long prefill is paid for fifteen times over.

LiteLLM has two mechanisms for this and they are not equally good.

The prompt_caching check computes a SHA-256 of the cacheable prefix, defined as everything up to the last block with cache_control, and pins the deployment with that hash. It has two problems. The TTL is hardcoded to 300 seconds in two places in the code, with no way to configure it, something which is reported and open; with one-hour caches, affinity is lost after five minutes of inactivity. And the key is an exact hash, so a client that moves the cut point forward on every turn, which is exactly what agents do to extend the cached prefix, generates a new key every time.

The session_affinity check is the one to use:

router_settings:
 optional_pre_call_checks: ["session_affinity"]
 deployment_affinity_ttl_seconds: 3600

It pins the deployment with an atomic Lua script in Redis, with a configurable TTL that defaults to one hour and that is refreshed on every request, so it bounds the idle time between turns and not the duration of the conversation. If the pinned deployment enters cooldown, the request goes out through the normal strategy and keeps the pin so it can come back when it heals.

The session identifier comes from a precedence chain that starts with x-litellm-trace-id, continues with x-litellm-session-id and then accepts any header of the form x-<something>-session-id whose value looks like an identifier. That includes the session header Claude Code already sends, with nothing configured on the client. Whatever matches populates both the session identifier and the trace identifier, which links this to the correlation from the first article in the track.

One dangerous interaction when combining it with tags: affinity is evaluated before tag-based routing. If the pin reduces the list to a deployment that does not satisfy the request’s tag, the request fails instead of falling through to the default pool, unless allow_fail_open has been enabled. Retagging a pool with live sessions breaks them.

And a clarification that saves a pointless search: LiteLLM does not do prefix routing. There is no prefix tree and nothing that directs the request to the vLLM replica that already has that prefix warm. What there is, is affinity by session or by key. For the other thing you need a specific router underneath the gateway, and that architecture has its own article pending.

The engine: what vLLM can and cannot do

Everything above shares out requests between engines. Inside one engine, the question is what happens when a twenty-token request and a sixty-thousand-token one live side by side.

Priority exists and does less than it looks like. --scheduling-policy priority turns the waiting queue into a heap ordered by priority, arrival time and identifier, with the lowest number first. The request can carry the priority field in the body, and there is also the X-Vllm-Priority header, which takes precedence over the body and is not documented anywhere.

What priority does not do is push ahead of whoever is already generating. In the running queue loop, when KV cache runs short, the policy only changes who is chosen as the victim of eviction: with priority the one with the worst priority and arrival time is evicted, and with fcfs the last in the queue. In the waiting queue loop, if there are no blocks available the iteration is cut short, without evicting anyone. And the max_num_seqs cap is a hard cut. If the agents have filled that cap, a chat with maximum priority waits just the same. There has been an open issue since April and an unmerged pull request about this.

That nuance about the victim is not negligible, because with fcfs the victims are the most recently admitted, that is, the chats that have just come in. Switching the priority policy on already improves the situation even if it pushes nobody ahead.

Eviction is always by recomputation. The swap-to-host-memory mode and the enumeration that controlled it disappeared from the code; all that is left is freeing blocks and returning the request to the queue with the computed token counter at zero. An evicted request loses all its prefill, and the prefix cache recovers part of it, but the perceived time to first token restarts.

Two admission settings help and do not appear in the narrative documentation. watermark reserves a fraction of free KV blocks on admission, with the declared purpose of avoiding repeated eviction, and it ships disabled. And scheduler_reserve_full_isl, active by default, checks that the full input length fits in the cache before admitting, instead of looking only at the first chunk.

Chunked prefill is active by default and prioritises decode. The scheduler walks the whole running request loop before touching the waiting queue, so a long agent prefill does not freeze a chat’s generation in progress. That protects the inter-token latency of what has already been admitted, and it does not protect the time to first token of what is waiting.

Here there is a breaking change that forces a review of the manifests. max_num_partial_prefills and max_long_partial_prefills were removed in 0.27. They were precisely the mechanism that allowed short prompts to get ahead of long ones. A deployment with those flags in its args does not start. long_prefill_token_threshold still exists, and its meaning changed: today it is a per-request chunk size cap, not a classification of long requests.

The partial replacement arrives in 0.29 with max_num_queued_tokens and max_num_queued_reqs, admission valves that return 503 when the pending prefill work exceeds the target. The code itself frames it as a quality of service mechanism for time to first token, with the formula of setting it to the latency target multiplied by prefill throughput. They are blind to the class of traffic: they reject a chat just the same as an agent.

And the default value of --max-num-batched-tokens depends on the hardware and on the start-up mode. With vllm serve on H100 or H200 it is 8,192, on cards of 160 GiB or more it is 16,384, and on the rest 2,048. The 2,048 that appears in the scheduler configuration only applies to tests.

The metrics for diagnosing coexistence, with the exact names from 0.29:

Metric	What for
`vllm:num_requests_waiting`	queue depth, the autoscaling signal
`vllm:num_requests_waiting_by_reason`	separates waiting for capacity from waiting in a blocked state
`vllm:num_preemptions_total`	evictions, the signal that the two classes are treading on each other
`vllm:kv_cache_usage_perc`	cache pressure, the cause of the evictions
`vllm:request_queue_time_seconds`	what a chat waits with the agents inside
`vllm:prefix_cache_hits_total` and `_queries_total`	measured in tokens, not in requests

A warning for anyone with old alerts. The log warning about evicted sequences that appears in the optimisation documentation belongs to the previous version of the engine and no longer exists in the code. Preemptions are only visible as a counter. An alert based on searching for that string in the logs has not fired for months, and not because there are no evictions.

On isolation between classes inside one engine, the short answer is that it does not exist. There are two queues and neither is per class of traffic. The conclusion, which is the one that holds up this article’s architecture: effective isolation means separate fleets, and on top of them the gateway deciding who goes to which.

Retries: the product nobody calculates

A human who gets an error sees it and decides. An agentic loop retries, and beneath it there are two more layers that also retry.

Layer	Default
Client SDK	2 retries, 3 attempts
LiteLLM router	2 retries, 3 attempts
Fallbacks	up to 5 groups

The product is about 45 calls to the engine for one agent turn, before counting the loop’s own retry. LiteLLM does what it can: when the call comes from the router, it forces max_retries=0 on the SDK it itself uses against the engine. What it cannot touch is the client’s SDK. The only layer that breaks the product is that one, and it is broken by setting max_retries=0 on the client and delegating the retry to the gateway.

The retryable errors are 408, 409, 429 and anything 500 or above. And there lies the problem with long reasoning: litellm.Timeout is a subclass of OpenAI’s timeout exception and comes out with code 408, which is retryable, so the most expensive request of all, the one that expired after ten minutes of generating, is launched again in full. Twice by the router, and as many times again by the client.

On timeouts you have to be precise because the code comment is: the configured timeout is per attempt, not per request. A request_timeout: 600 with two retries and two fallback groups gives a worst case of almost two hours before the client receives anything. The default value is still 6,000 seconds, as was already covered in the day 2 article.

The retry storm has two specific causes and both are default settings.

The first: a single 429 takes the deployment out of the pool, without going through the allowed_fails threshold. The cooldown logic has an explicit branch which, on a 429 in a group with more than one deployment, returns immediate cooldown. With an agentic burst against four replicas, all four leave the pool at once and the gateway answers everybody with its own 429 of “no deployments available”. The default cooldown is five seconds, which in a sustained burst gets renewed.

The second: backoff is zero while the pool looks healthy. The function that computes the wait returns 0 if the healthy deployment list is not empty. Exponential backoff with jitter, capped at eight seconds, only kicks in when there is nothing healthy left. Under partial degradation, the proxy hammers away without waiting and only slows down once everything has already fallen over.

On top of that, the engine’s Retry-After header is only respected if its value is between 1 and 60 seconds. A longer limit window is ignored and replaced by the proxy’s own backoff.

The settings that cut the problem off:

router_settings:
 num_retries: 1
 retry_after: 2 # real floor; the default 0 is hammering
 cooldown_time: 30
 allowed_fails_policy:
 RateLimitErrorAllowedFails: 3 # stops a 429 taking the deployment out
 retry_policy:
 TimeoutErrorRetries: 0 # do not relaunch the expensive request
 AuthenticationErrorRetries: 0
litellm_settings:
 request_timeout: 900 # per attempt
 drop_params: false

That last setting deserves an explanation, because it is a silent way of breaking an agent. With drop_params: true, LiteLLM removes from the body any parameter the destination provider does not support, and in that branch it writes no warning at all. The candidates for disappearing are tools, tool_choice, response_format and parallel_tool_calls. An agent that has tools taken away receives free text where it expected a function call, and interprets it as the model having decided not to use tools.

Worse still is prompt-based emulation. For a non-OpenAI-compatible provider that does not support tools, LiteLLM does not fail: it turns the tools into text inside the prompt, forces JSON format, discards tool_choice, and on the way switches on a process-global variable that makes any other tool-less provider in the same process go to emulation instead of erroring.

With that in mind, the warning about fallbacks is clear: LiteLLM does not check capabilities before falling back. The function that knows whether a model supports function calling exists and is only used in an informational endpoint. The pre-call checks filter by context window, request limit and region, and nothing else. A fallback list that goes from a model with tools to one without them degrades the agent silently. There are guards for provider-scoped resources, files and batches, so the absence of the capability guard is an omission and not a design oversight.

A month’s budget in twenty minutes

A runaway agent consumes in minutes what a team consumes in weeks, and the operational question is how long the gateway takes to find out.

The answer has changed, and the old mental model, the one where spend consolidates every minute and until then there is no brake, is no longer correct. Since 1.84 there are two new pieces. An optimistic reservation before the call, which estimates the maximum cost of the request and reserves it atomically in the counter before heading out to the engine, reconciling it with the real cost on completion. And a counter increment after the call that is explicitly awaited, with the comment that this way the counter is up to date before the next request goes through authentication. Budget checks read that counter, with Redis first and seeding from the database when cold.

Batch writing to Postgres still exists, every 10 to 15 seconds, and it affects persistence and what is shown in the interface, not the cut-off. The production documentation’s recommendation to raise that interval to 60 does not relax the control, and it does delay by a minute what /key/info shows, something which frequently generates operational panic. And the budget rescheduler’s ten minutes do not consolidate spend: they fire the job that resets expired budgets.

The overspend window then comes down to the cost of the requests in flight, provided there is Redis. Without Redis, the counters are per pod, and with N replicas you can spend up to N times the budget. That is the configuration failure that matters and it is the same Redis that the day 2 article declared non-optional for other reasons.

Two settings that do not appear in the public documentation, only in the data model descriptions, and that decide behaviour at the worst moment:

general_settings:
 fail_closed_budget_enforcement: true # false by default: if the counter fails, it lets things through
 # disable_budget_reservation: do not set it to true

With the default value, if the counter backend does not respond, the proxy fails open and carries on serving. For an agent in a loop that is exactly the opposite of what you want.

Time for the honest warning: this route is still unstable. There are open issues in both directions, one where the key and user budget limit stops being applied while spend is recorded, and another where a 429 is returned for stale spend in the reservation counters while the key query shows spend below the limit. Project budgets are not on the atomic reservation route and concurrency overshoots them. The practical conclusion is not to trust the hard cut-off to the budget alone.

On the manual cut-off, there is an asymmetry to know about before you need it. /key/block invalidates the local cache, the Redis one, and notifies all workers by publish, so with Redis the effect is almost immediate. /customer/block does the same. /team/block only writes to the database, without invalidating or notifying, so it takes however long the management object cache TTL takes, sixty seconds by default. To cut a team off dead you have to block its keys one by one, or lower user_api_key_cache_ttl. And /user/block no longer exists, even though the docstring of the function that replaced it still shows that route in its example.

On alerts, two numbers. The default thresholds warn at 85 % and 95 % of the budget. And deduplication of those alerts is 24 hours per event and identifier, so crossing 85 % generates one warning and then silence for a day. For agentic traffic, where 85 % and 100 % can be twenty minutes apart, the second warning arrives when it is no longer any use.

Attribution: who the user is when the caller is an agent

With agents, the question of who the spend is charged to has two answers and both are needed: the agent that executes and the human on whose behalf it acts.

The chain that extracts the end customer identifier tries the standard headers first, then the configurable header mapping, and only then the user field of the body, the litellm_metadata.user of the Anthropic format, the metadata.user_id and the safety_identifier. The last four are body fields and are therefore written by the client. The code itself warns about it: for untrusted callers the identifier has to be set from headers or from a middleware on the server, to avoid impersonation.

Out of that comes the correct pattern, which is the inverse of the one people put together out of inertia: the virtual key identifies the agent and is the non-forgeable part, and the header or the user field identify the human and serve for attribution, not for control. An agent that lies about the human behind it is still bounded by the budget and the limits of its key.

The spend table stores user, end_user, team_id, organization_id, request_tags, session_id and agent_id, with daily aggregation per agent, so the detail is available once it is sent. The x-litellm-agent-id header is what populates that last field.

A note about the MCP gateway, which on a platform with agents ends up being the other entry door. Tool calls do appear in the spend records, with call types of their own and a metadata block that stores tool name, arguments, result and server. What is missing is token counting: the cost is 0.0 by default and is only populated if the administrator configures a price per query, per server or per tool. It is the same pattern of silent zero cost that already came up with unpriced models in the first article in the track.

And the permission setting to review on day one: access control for MCP servers is open by default. If no level of the hierarchy defines a list, the request reaches every configured server. general_settings.require_key_mcp_access_defined: true inverts that inheritance, and it is not the default value.

A reference `config.yaml`

All of the above, together, on a fleet split into two pools:

model_list:
 - model_name: qwen-30b-interactivo
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-chat.inferencia.svc:8000/v1
 input_cost_per_token: 0.00000018 # without this, silent zero cost
 output_cost_per_token: 0.00000072
 model_info:
 id: chat-01
 max_input_tokens: 32768 # without this there is no context filter

 - model_name: qwen-30b-agentes
 litellm_params:
 model: hosted_vllm/Qwen3-30B
 api_base: http://vllm-agentes.inferencia.svc:8000/v1
 input_cost_per_token: 0.00000018
 output_cost_per_token: 0.00000072
 tag_regex: ["^User-Agent: claude-code\\/"]
 model_info:
 id: agentes-01
 max_input_tokens: 131072
 allow_fail_open: true

router_settings:
 routing_strategy: simple-shuffle
 enable_pre_call_checks: true
 enable_tag_filtering: true
 optional_pre_call_checks: ["session_affinity"]
 deployment_affinity_ttl_seconds: 3600
 num_retries: 1
 retry_after: 2
 cooldown_time: 30
 allowed_fails_policy:
 RateLimitErrorAllowedFails: 3
 retry_policy:
 TimeoutErrorRetries: 0
 AuthenticationErrorRetries: 0

general_settings:
 fail_closed_budget_enforcement: true
 require_key_mcp_access_defined: true
 user_api_key_cache_ttl: 10
 proxy_batch_write_at: 10
 # 1.101 branch onwards:
 max_in_flight_requests_per_worker: 64
 max_queued_requests_per_worker: 64
 admission_queue_timeout_seconds: 1.0

litellm_settings:
 request_timeout: 900 # per attempt
 drop_params: false
 cache: true # Redis, not optional

And the start-up of the two engines, with the deliberate asymmetry:

# interactive pool: protect the first token
vllm serve Qwen/Qwen3-30B \
 --scheduling-policy priority \
 --max-num-seqs 64 \
 --max-model-len 32768 \
 --max-num-batched-tokens 4096

# agent pool: throughput and deep queues
vllm serve Qwen/Qwen3-30B \
 --max-num-seqs 256 \
 --max-model-len 131072 \
 --max-num-batched-tokens 8192

Checklist

Issue separate keys for agents, with model_group_alias at key or team level. It is the only separation the client cannot evade.
Add tag_regex over the User-Agent as a safety net for agents that use personal credentials, with a default pool and allow_fail_open: true.
Leave simple-shuffle on both pools. The other four strategies have verifiable defects on homogeneous replicas.
Enable session_affinity with a one-hour TTL. Do not use prompt_caching for this while its TTL stays hardcoded.
Declare max_input_tokens on every deployment. Without that value, the context window filter is not applied and it fails silently for models that are not in the price map.
Declare per-token prices on every self-hosted model, or the agents’ spend will be zero and their budget will never run out.
max_parallel_requests on the agent’s key, never on its team. On the team it is stored and not applied.
Lower num_retries to 1, put a floor on retry_after, raise cooldown_time and disable retry on timeout. On the client, max_retries=0.
fail_closed_budget_enforcement: true and mandatory Redis. Without Redis, N replicas are N times the budget.
Start the interactive engine with --scheduling-policy priority even though it pushes nobody ahead: it changes who it evicts.
Alert on vllm:num_preemptions_total and vllm:num_requests_waiting_by_reason, not on log strings that are no longer emitted.
Prepare the cut-off: block by key, never by team, and verify the real propagation time in your own deployment.

Traps and things that are not what they seem

priority: 0 on /chat/completions disables prioritisation. It is the value in the documentation examples and it is falsy in Python.
max_parallel_requests on a team is stored and not applied. The field exists in the API, in the database and in the documentation; the limiter only reads it from the key descriptor.
global_max_parallel_requests is a no-op with the default limiter, and the server settings documentation still describes it as active.
Every MCP call leaks a concurrency slot, which only heals when its hour expires. With agents calling tools in a loop, the 429 appears early and lasts a long time.
The TTL of prompt cache affinity is 300 hardcoded seconds. For agent conversations, session affinity is the right option.
Affinity is evaluated before tag-based routing. A pin to a deployment that stops satisfying the tag makes the request fail instead of falling through to the default pool.
LiteLLM’s timeout is per attempt. The worst case is the timeout multiplied by attempts and by fallback groups.
A 429 takes the deployment out of the pool without going through allowed_fails, and backoff is zero while anything healthy remains.
The engine’s Retry-After is ignored if it asks for more than 60 seconds.
drop_params: true removes tools and response_format without leaving a warning, and the fallback does not check whether the destination model supports function calling.
/team/block does not invalidate the cache. It takes the full TTL; /key/block does propagate by publish in Redis.
Access to MCP servers is open by default if no level defines a list.
max_num_partial_prefills and max_long_partial_prefills no longer exist in vLLM. A manifest with those flags does not start from 0.27 onwards.
vLLM’s priority does not push ahead of running requests, and with max_num_seqs full it pushes nobody ahead.
The eviction log warning that appears in the vLLM documentation is no longer emitted. Only the Prometheus counter is left.

Closing

The coexistence of humans and agents on the same platform is not solved with a priority field, and that is the conclusion that takes the most effort to accept because the field exists and looks made for this. Neither the gateway nor the engine has a usable notion of class of service today: one orders a queue that is almost never consulted, the other picks who to evict. What they do have is the ability to direct each request to a different place, and that is where the win is.

The architecture that comes out of all of the above fits in one sentence. Two vLLM fleets with opposite parameters, a credential that decides which one each request goes to, session affinity so as not to throw away the prefill work, concurrency limits on the agent’s key and not on its team, retries trimmed back on the three layers where they multiply, and budgets that fail closed.

The rest is measurement. If vllm:num_preemptions_total rises in the interactive pool, the two classes are treading on each other despite the separation. If chat queue time grows while total throughput holds, the partition is badly sized. And if the agents’ spend comes out as zero, the problem is not coexistence but undeclared prices, which is where this track started.

Sources

LiteLLM, [BETA] Request Prioritization: https://docs.litellm.ai/docs/scheduler.
LiteLLM, Router - Load Balancing (estrategias, afinidad de sesión, comprobaciones previas): https://docs.litellm.ai/docs/routing.
LiteLLM, Tag Based Routing (incluye tag_regex y el ejemplo con clientes agénticos): https://docs.litellm.ai/docs/proxy/tag_routing.
LiteLLM, Router Settings for Keys and Teams (resolución clave, equipo, global): https://docs.litellm.ai/docs/proxy/keys_teams_router_settings.
LiteLLM, Server Tuning (control de admisión por worker): https://docs.litellm.ai/docs/proxy/server_tuning.
LiteLLM, Dynamic TPM/RPM Allocation (reserva de prioridad, beta y enterprise): https://docs.litellm.ai/docs/proxy/dynamic_rate_limit.
LiteLLM, Budgets, Rate Limits: https://docs.litellm.ai/docs/proxy/users.
LiteLLM, Production Best Practices: https://docs.litellm.ai/docs/proxy/prod.
LiteLLM, Reliability (fallbacks, ventana de contexto, enfriamientos): https://docs.litellm.ai/docs/proxy/reliability.
LiteLLM, código: litellm/scheduler.py, litellm/router_strategy/tag_based_routing.py, litellm/router_utils/pre_call_checks/deployment_affinity_check.py, litellm/proxy/hooks/parallel_request_limiter_v3.py, litellm/proxy/middleware/admission_control_middleware.py, litellm/router_utils/cooldown_handlers.py: https://github.com/BerriAI/litellm.
LiteLLM, incidencia 28427, TTL de la afinidad por caché de prompts fijado a cinco minutos: https://github.com/BerriAI/litellm/issues/28427.
LiteLLM, incidencia 34534, fuga de slot de concurrencia en cada llamada MCP: https://github.com/BerriAI/litellm/issues/34534.
LiteLLM, incidencia 26672, presupuesto de clave y usuario no aplicado: https://github.com/BerriAI/litellm/issues/26672.
LiteLLM, incidencia 872, comprobación previa de soporte de llamada a funciones, cerrada sin implementar: https://github.com/BerriAI/litellm/issues/872.
vLLM, Optimization and Tuning (prefill troceado, prioridad de decodificación): https://docs.vllm.ai/en/v0.29.0/configuration/optimization.html.
vLLM, Metrics design (nombres, convención _total, autoescalado como problema abierto): https://docs.vllm.ai/en/v0.29.0/design/metrics.html.
vLLM, Data Parallel Deployment (caché KV independiente por réplica): https://docs.vllm.ai/en/v0.29.0/serving/data_parallel_deployment.html.
vLLM, código: vllm/v1/core/sched/scheduler.py, vllm/v1/core/sched/request_queue.py, vllm/config/scheduler.py, vllm/engine/arg_utils.py, vllm/v1/metrics/loggers.py: https://github.com/vllm-project/vllm.
vLLM, incidencia 40004, la prioridad no desaloja peticiones en ejecución: https://github.com/vllm-project/vllm/issues/40004.

Virtual keys, budgets and limits in LiteLLM: the layer that decides who gets the GPU, and the four things the documentation gets wrong

Tue, 08 Sep 2026 07:30:00 +0200

Third article in the operational track of the control layer. The pair with Langfuse covered observability, and day 2 of the proxy covered availability. Here it is governance: who can call, which model, how much and on what budget. The economic model behind those figures is in FinOps and multi-tenancy with LiteLLM.

TL;DR

On a platform running on your own GPUs there is no provider invoice to put consumption in order. Capacity is fixed, known and shared, so someone has to impose the allocation, and that someone is the gateway.

The hierarchy has six levels: organisation, team, team member, internal user, key and end customer, plus the project and tag scopes. Each level’s budgets are evaluated independently and all of them have to pass. They neither add up nor cancel each other out, with one exception.

The exception is in the code and it is the most important governance trap in the system. A key that belongs to a team suppresses the check of its owner’s personal budget, unless apply_user_budget_to_team_keys is enabled. A user with a hundred-euro cap who works with their team’s key spends whatever the team allows.

Rotating a key preserves its spend. /key/regenerate does an UPDATE on the same row changing the hash, so spend, max_budget and the budget window remain intact. It accepts a grace period, stored in a table of its own, and if the period’s format is wrong it is ignored without blocking anything.

The v3 limiter is already the default in 1.100.0, and you go back to the previous one with an environment variable. Its window is sliding, sixty seconds, anchored to the first request, and the counters are incremented with Lua scripts inside Redis using Redis’s own timestamp to close the race between replicas.

And four points where the official documentation does not match the code, verified against tag v1.100.0: an exhausted budget returns 429, not 400; the audited actions are six, not three; /key/info and /key/list are GET, not POST; and the v3 limiter carries a docstring saying it is not production-ready while it is the one loaded by default.

You are here: the gateway as an administrative boundary

Without this layer, an in-house inference platform has a single mode of operation: whoever knows the URL consumes. It works while there is one team. It stops working the day there are three, or the day someone connects an agent that fires twenty calls per interaction and wrecks everyone else’s time to first token.

The two questions this layer answers are different and have to be kept apart. The capacity one is instantaneous: how many requests and how many tokens per minute each consumer can push, so the fleet does not saturate and latency stays within the service agreement. The cost one is cumulative: how much each consumer has spent this month against its budget. They are configured in the same place, applied at different moments and fail in different ways.

The analogy: the building’s key ring

An office building hands out keys at three levels. The master opens everything and the building manager has it. The floor keys open a whole floor and each tenant company has one. And the office keys open one office.

Three properties of that system are the ones that matter here. A key is replaced without changing the lock of the whole building, and the tenant keeps their parking space and their electricity meter: that is rotation preserving the spend. A lost key is cancelled from the central panel without collecting anything physical: that is blocking from the interface. And the record of who took out each key and when lives in a separate book, which is what you show an inspector: that is the audit table.

The analogy breaks at one point, and that point is where this system’s governance flaw lives. In the building, having a floor key does not cancel the office’s consumption limit. In LiteLLM, it does.

The hierarchy, and the line that breaks it

The levels and their management endpoints:

Level	Endpoints	Budget
Organisation	`/organization/new`, `/organization/member_add`, …	`max_budget`
Team	`/team/new`, `/team/member_add`, `/team/block`, …	`max_budget`
Team member	`/team/member_add` with `max_budget_in_team`	Its own row in the budgets table
Internal user	`/user/new`, `/user/update`, …	`max_budget`
Key	`/key/generate`, `/key/update`, …	`max_budget`, `soft_budget`
End customer	`/customer/new`, and the parallel `/end_user/*` set	`max_budget`

The default behaviour is the right one for a multi-tenant platform: each scope carries its own counter and the check of each one is independent of the rest. A request passes if all of them pass.

And then there is this condition, which lives in the authorisation checks:

is_team_key = team_object is not None and team_object.team_id is not None
if is_team_key and general_settings.get("apply_user_budget_to_team_keys") is not True:
 return

Translated: when the key belongs to a team, the check of the budget of the user who owns that key returns without evaluating anything. The personal cap exists in the database, is visible in the interface, and is not applied.

For a deployment where personal budgets are decorative, it makes no difference. For one where the per-person cap is the control that was shown to someone as a guarantee that a user cannot overrun the platform, it is a hole. The line that closes it is one:

general_settings:
 apply_user_budget_to_team_keys: true

Virtual keys: the complete life cycle

The endpoints all exist and some of them are not where you expect:

Route	Method	What for
`/key/generate`	POST	Create
`/key/update`	POST	Modify
`/key/info`	GET	Query one
`/key/list`	GET	List
`/key/delete`	POST	Delete
`/key/regenerate`	POST	Rotate
`/key/block`, `/key/unblock`	POST	Suspend and reactivate
`/key/health`	POST	Check that key’s callbacks

The two marked in bold are GET, and several examples in the documentation suggest otherwise. /key/health is not what its name suggests either: it does not validate the key or its budget, it checks the logging callbacks associated with it.

There are many creation parameters, and these are the ones that govern:

curl -X POST 'https://gateway.interno/key/generate' \
 -H 'Authorization: Bearer sk-...' \
 -d '{
 "key_alias": "equipo-datos-notebooks",
 "team_id": "t-datos",
 "models": ["llama-70b", "qwen-30b"],
 "max_budget": 250,
 "budget_duration": "30d",
 "tpm_limit": 400000,
 "rpm_limit": 600,
 "max_parallel_requests": 8,
 "duration": "90d",
 "tags": ["produccion", "notebooks"]
 }'

Alongside those there are controls that are rarely used and solve specific problems: model_max_budget to set a per-model cap within the same key, model_tpm_limit and model_rpm_limit for the same thing with capacity, enforced_params to require certain keys to always arrive with given fields, allowed_routes so an application key cannot touch the management endpoints, and blocked to be born suspended.

How they are stored

A key’s hash is a single-pass hexadecimal SHA-256, with no salt and no derivation function:

hashed_token = hashlib.sha256(token.encode()).hexdigest()

The row lives in LiteLLM_VerificationToken, with the hash as primary key, and it contains the accumulated spend, the limits, the expiry, the allowed models and the rest of the configuration.

Two consequences to be clear about before an audit. The first is that this hash is not designed to resist a dictionary attack against the database, and the code itself acknowledges it in a comment about encryption key derivation, where it admits that a single-pass unsalted SHA-256 is not a derivation function and that moving to HKDF would be more defensible in an audit. The real mitigation is that virtual keys are generated with enough entropy, not the strength of the hash. The second is that LITELLM_SALT_KEY plays no part here: that variable encrypts the stored provider credentials, not the virtual keys.

There is a trap around that variable that takes down a whole deployment. If it is not defined, it falls back to the master key. Rotating LITELLM_MASTER_KEY without having first set your own LITELLM_SALT_KEY leaves every stored provider credential unreadable, and the failure does not block: it logs an error and returns null, so the symptom shows up as models that stop authenticating with no explanation. Setting LITELLM_SALT_KEY on day one, and never touching it, is one of the things that saves the most grief.

Rotation

/key/regenerate does not create a new key and delete the old one. It updates the same row changing the hash, so the accumulated spend, the maximum budget and the reset window are preserved. For a monthly chargeback this is exactly what you want: rotating mid-month does not zero the counter.

It also accepts a grace period, so the old key remains valid while consumers update:

curl -X POST 'https://gateway.interno/key/sk-vieja/regenerate' \
 -H 'Authorization: Bearer sk-...' \
 -d '{"grace_period": "24h"}'

The old hash is stored in a separate table with its revocation date. Without that parameter, revocation is immediate, and the default value can be set with LITELLM_KEY_ROTATION_GRACE_PERIOD.

A detail with teeth: if the period’s format is wrong, the system logs a warning and continues with no grace period. The rotation completes, the old key dies on the spot, and all that is left is a log line. In an automated rotation procedure, that combination of “carry on” and “warn in the log” is what produces the outage at three in the morning.

What exactly happens when a budget runs out

The code raises BudgetExceededError with status 429. The documentation says 400 in one place, and there the code wins. Its being a 429 has a practical implication: OpenAI clients retry automatically on a 429, so an exhausted budget turns into a retry storm against a gateway that is going to keep rejecting. Internal clients should distinguish the reason, which travels in the message.

The messages are templates and state the scope, the spend and the cap:

Budget has been exceeded! Key=<clave> Current cost: <gasto>, Max budget: <tope>
Budget has been exceeded! Team=<equipo> Current cost: <gasto>, Max budget: <tope>
ExceededBudget: User=<usuario> over budget. Spend=<gasto>, Budget=<tope>
ExceededBudget: End User=<cliente> over budget. Spend=<gasto>, Budget=<tope>
Budget has been exceeded! Organization=<org> ...
Budget has been exceeded! Tag=<etiqueta> ...

Two clarifications about the comparison. It is spend >= cap, with the equals sign included, so landing exactly on the cap already blocks. And the soft budget, soft_budget, blocks nothing: it only writes a log line when it is crossed. It serves to warn, not to contain, and building an alert on top of it is the right way to use it.

The rest of the rejections, with their codes:

Situation	Code	Behaviour
Budget exhausted	429	Message with scope, spend and cap
Expired key	401	It also deletes the cache entry, so the expiry takes effect across the whole fleet without waiting for the TTL
Model not allowed	403	Different error type depending on the scope: key, team, user, organisation or project
Blocked key	Generic exception	`Key is blocked. Update via /key/unblock if you're an admin.`

The last row is an inconsistency in the project worth knowing about: blocking raises an exception with no explicit type or code, while expiry and model access do carry them. Any alert that discriminates by error type will not see blocks the way it sees the rest.

A useful note about allowed models: the check accepts both the alias and the underlying model, so a key authorised for llama-70b can call it by its alias or by its real name with no additional configuration.

Capacity limits: the v3 limiter

This is the most significant silent change of the year in this layer. In 1.100.0, the limiter loaded by default is the third version. Going back to the previous one has to be asked for explicitly:

LEGACY_MULTI_INSTANCE_RATE_LIMITING=true

There is no flag to turn it on, only to turn it off. And it carries a contradiction worth knowing before trusting it: the file itself still opens with a comment saying it is under development and not production-ready, while it is the one running. The design is avowedly inspired by Envoy’s.

The window is sliding and lasts sixty seconds, configurable with LITELLM_RATE_LIMIT_WINDOW_SIZE. It is not an exact rolling log nor a clock-aligned window: it anchors to the first request and resets once the window size has elapsed from that moment. For one-minute limits, the difference from a calendar window shows up at the edges and rarely matters.

What does matter is Redis. The counters are incremented with Lua scripts executed inside Redis, using Redis’s own timestamp and not the pod’s, precisely so window resets are deterministic across replicas and to close the race between checking and counting. Without Redis, each pod counts on its own and the effective limit is multiplied by the number of pods. A limit of 600 requests per minute with five replicas is a limit of 3,000.

There is an inherent imprecision in the token limits that Redis does not solve. The number of output tokens is not known until the response finishes, so the limiter counts with an estimate before the call and reconciles afterwards. The possible excess is, per in-flight request, the difference between the estimate and the real figure. With high concurrency and long responses, that allows the cap to be exceeded within a window. To protect the fleet, the concurrent request limit (max_parallel_requests) is a more direct instrument than tokens per minute.

What the client sees

When limited, a 429 with the message giving the scope, the current limit, what is left and when it resets. And three headers: retry-after with the window size, that is 60 by default, plus rate_limit_type and reset_at.

On the success path, the informational headers use a nested format of their own:

x-ratelimit-api_key-remaining-requests
x-ratelimit-api_key-limit-tokens
x-ratelimit-team-remaining-requests

They are not the standard names x-ratelimit-limit-requests that OpenAI and compatibles emit, so a client expecting the usual format will find nothing. There is also an open issue about these headers being lost on streaming responses, which is most of an assistant’s traffic.

Identity: beyond the static key

Handing out keys by hand works with three teams and stops working with thirty. The two routes for connecting the gateway to corporate identity:

JWT. It is enabled with enable_jwt_auth and configured with the litellm_jwtauth block, which maps token fields to LiteLLM entities: the team comes from team_id_jwt_field, the user from user_id_jwt_field, the organisation from org_id_jwt_field. It supports dot notation for nested claims, automatic creation of users and teams with user_id_upsert and team_id_upsert, role-based access control with enforce_rbac, per-team model restriction with enforce_team_based_model_access, and several issuers at once. The issuer’s public key is cached with a TTL of 600 seconds by default.

general_settings:
 enable_jwt_auth: true
 litellm_jwtauth:
 team_id_jwt_field: "groups"
 user_id_jwt_field: "sub"
 user_email_jwt_field: "email"
 enforce_rbac: true
 enforce_team_based_model_access: true
 team_id_upsert: true

OIDC for the admin interface. With Keycloak you use the generic provider, and the variables are PROXY_BASE_URL, GENERIC_CLIENT_ID, GENERIC_CLIENT_SECRET, GENERIC_AUTHORIZATION_ENDPOINT, GENERIC_TOKEN_ENDPOINT, GENERIC_USERINFO_ENDPOINT, plus a set of attributes to map the identifier, the email, the name and the role, and GENERIC_ROLE_MAPPINGS_* to translate Keycloak groups into LiteLLM roles. Login is rate-limited and sessions have a TTL of their own.

For automatic provisioning there is SCIM in the enterprise version, with base /scim/v2, and deprovisioning a user blocks their keys and takes them out of the authentication cache. That part is not in the open tree, so it cannot be verified against the code.

This fits with the identity work already covered in hardening and secrets of the sovereign stack and in the article on MCP authentication with Keycloak, which uses the same issuer.

Audit: what gets recorded

The table is LiteLLM_AuditLog and its columns are what you would expect from a change log:

id, updated_at, changed_by, changed_by_api_key,
action, table_name, object_id, before_value, updated_values

It stores the previous value and the new one, and attributes the change to an actor and to the key it was made with.

Three clarifications the documentation does not capture well.

The actions are six, in the past tense: created, updated, deleted, blocked, unblocked and rotated. The documentation mentions three. That blocking, unblocking and rotation are actions in their own right is exactly what is needed to reconstruct a credential’s life cycle for an auditor.

The scope is wider than advertised. Besides teams, users, keys and models, changes to the proxy configuration and to the SSO configuration are audited. A change of identity issuer is recorded.

The default depends on the licence. The resolution order is litellm_settings.store_audit_logs, then the variable LITELLM_STORE_AUDIT_LOGS, and if neither is set, it stays enabled in enterprise and disabled in the open version. An OSS deployment that assumes there is an audit trail does not have one:

litellm_settings:
 store_audit_logs: true

Two more pieces for a serious chain of evidence. The actor attribution can be passed with the LiteLLM-Changed-By header, and its use is restricted by the key’s or the team’s configuration, so any caller cannot forge it. And the events can be shipped out with audit_log_callbacks and its S3 parameters, which is what allows depositing them in an immutable store with object lock.

That last part is what turns the record into evidence. The Postgres table is mutable by anyone with database access, and as of September 2026 it has no indexes, so querying it by date range over a long history will not be comfortable either.

What this demonstrates to an auditor, and what it does not

This is the part that decides whether the setup is good enough for a regulated customer. With everything above configured, the platform sustains without difficulty:

Access control via corporate identity, with onboarding and offboarding in the identity provider and not in the gateway, and with the role determining which models are reachable. Segregation between tenants, with independent counters per organisation, team and person. Traceability of administrative changes, with actor, time, previous value and new value, exportable to an immutable store. And demonstrable consumption limits, with the evidence of the rejection in the spend log.

What it does not cover, and has to be solved elsewhere:

The record of the interactions is not this. The audit table records configuration changes, not model calls. The calls are in the spend log and in the traces, and traces are best-effort, which is the discussion in the first article in the series.

The integrity of the record is not something Postgres provides. Without export to immutable storage, the evidence is only as strong as the database permissions.

The content of the prompts is a separate problem, with its own personal-data treatment and its own decision about whether it is stored, masked or not logged at all.

And a warning about the control that tends to be shown first: while apply_user_budget_to_team_keys is not enabled, the per-person cap is not being applied to team keys. Showing that field in the interface as proof of a control that is not executed is the kind of finding a competent auditor discovers. The complete control mapping is in ENS, ISO 42001 and the EU AI Act.

Checklist

apply_user_budget_to_team_keys: true if personal budgets have to be enforced.
LITELLM_SALT_KEY set on day one, different from the master key and never rotated lightly.
store_audit_logs: true explicitly, without trusting the default.
audit_log_callbacks towards an immutable store if an audit has to be sustained.
Redis mandatory if there is more than one replica, or the limits get multiplied by the number of pods.
max_parallel_requests per key, on top of the per-minute limits, to protect the fleet from a runaway agent.
allowed_routes on application keys, so they cannot reach the management endpoints.
soft_budget with an alert on top, understanding that it does not block.
Rotation with a verified grace_period, because an invalid format revokes on the spot.
JWT against Keycloak as soon as there are more than a handful of teams, with enforce_rbac and enforce_team_based_model_access.

Traps and things that are not what they look like

A user’s cap is not enforced. They are using a team key and apply_user_budget_to_team_keys is missing.

Clients retry in a loop against an exhausted budget. The code is 429 and the OpenAI libraries retry 429s automatically.

The soft budget has stopped nothing. It does not block, it only logs.

The per-minute limits are enforced at double or triple. There is no Redis, and each replica keeps its own count.

The client does not see the limit headers. The names are nested and proprietary, and in streaming there is an open issue where they get lost.

There is no audit log. It is an OSS deployment and the default there is disabled.

The models stop authenticating after rotating the master key. There was no fixed LITELLM_SALT_KEY, and the encrypted provider credentials can no longer be decrypted.

The rotation cut the service on the spot. The grace_period carried a format the system did not understand and it carried on without it.

Blocking a key does not fire the alert. It is raised as a generic exception, without the type that expiry and model access do carry.

Closing

The keys and budgets layer is the one that turns a shared URL into a platform with tenants. It is well solved in LiteLLM and has more depth than the documentation suggests, with per-model budgets inside a key, multiple budget windows, per-tag limits and rotation that preserves the history.

Out of the whole article, two things take the attention. The apply_user_budget_to_team_keys line, because it is the difference between having a control and believing you have one. And setting LITELLM_SALT_KEY, because its default turns a routine rotation of the master key into an outage that is hard to diagnose. Both are one line of configuration and both are discovered late.

Sources

LiteLLM, Virtual Keys (parámetros de creación, ciclo de vida, rotación): https://docs.litellm.ai/docs/proxy/virtual_keys.
LiteLLM, Budgets, Rate Limits (jerarquía, presupuestos por nivel, clientes finales): https://docs.litellm.ai/docs/proxy/users.
LiteLLM, JWT Token Auth (bloque litellm_jwtauth y todos sus campos): https://docs.litellm.ai/docs/proxy/token_auth.
LiteLLM, Audit Logs y administración múltiple (la cabecera LiteLLM-Changed-By): https://docs.litellm.ai/docs/proxy/multiple_admins.
LiteLLM, SCIM (aprovisionamiento automático, versión enterprise): https://docs.litellm.ai/docs/tutorials/scim_litellm.
LiteLLM, litellm/proxy/auth/auth_checks.py en la etiqueta v1.100.0 (la condición de apply_user_budget_to_team_keys y los mensajes de presupuesto agotado): https://github.com/BerriAI/litellm/blob/v1.100.0/litellm/proxy/auth/auth_checks.py.
LiteLLM, litellm/proxy/management_endpoints/key_management_endpoints.py en v1.100.0 (rutas, rotación sobre la misma fila, periodo de gracia): https://github.com/BerriAI/litellm/blob/v1.100.0/litellm/proxy/management_endpoints/key_management_endpoints.py.
LiteLLM, litellm/proxy/hooks/parallel_request_limiter_v3.py y litellm/proxy/hooks/__init__.py en v1.100.0 (el limitador v3 como valor por defecto, la ventana, los scripts Lua): https://github.com/BerriAI/litellm/blob/v1.100.0/litellm/proxy/hooks/parallel_request_limiter_v3.py.
LiteLLM, litellm/exceptions.py en v1.100.0 (BudgetExceededError con código 429): https://github.com/BerriAI/litellm/blob/v1.100.0/litellm/exceptions.py.
LiteLLM, litellm/proxy/common_utils/encrypt_decrypt_utils.py en v1.100.0 (LITELLM_SALT_KEY cayendo en la clave maestra, y el comentario sobre la derivación de claves): https://github.com/BerriAI/litellm/blob/v1.100.0/litellm/proxy/common_utils/encrypt_decrypt_utils.py.
LiteLLM, schema.prisma en v1.100.0 (LiteLLM_VerificationToken, LiteLLM_AuditLog, la tabla de claves deprecadas): https://github.com/BerriAI/litellm/blob/v1.100.0/schema.prisma.
LiteLLM, issue #27748 — cabeceras x-ratelimit-* perdidas en respuestas en streaming con el limitador v3: https://github.com/BerriAI/litellm/issues/27748.

LiteLLM on day 2: one worker per pod, six thousand seconds of timeout and the other five things to change before opening the traffic

Tue, 08 Sep 2026 07:00:00 +0200

Second article in the operational track of the control layer. The operational pair with Langfuse covered the seam between the gateway and observability; here everything stays inside the gateway. The prior decisions, which gateway and why, are in choosing the OSS gateway and in the L7 inference router.

TL;DR

LiteLLM Proxy starts from a twenty-line config.yaml and serves traffic that same afternoon. The operational work consists of correcting seven defaults designed for a scenario other than an in-house inference factory.

One uvicorn worker per pod. The project’s production guide asks for --num_workers 1 on Kubernetes and horizontal scaling, with 1 vCPU and 4 GiB per worker set at the same time as requests and as limits. The 4 GiB are a floor, not a target: the Prisma query engine marks a resident memory high-water mark that grows up to the largest statement it has executed and that glibc does not return to the system.

Autoscale on CPU, never on memory. A direct consequence of the above. Memory goes up and never down, so an HPA on memory scales and never scales back. The recommended target is targetCPUUtilizationPercentage: 60.

The Postgres connection arithmetic. database_connection_pool_limit is 10, and the number of connections is instances times workers times the pool. The Helm charts ship maxReplicas: 100, on the order of a thousand connections, far above what a stock Postgres accepts.

Redis stops being optional from around a thousand requests per second or ten instances. Without it, each instance applies its limits on its own, caches are not shared, and spend updates against the same rows produce locks and exhaust the connections with FATAL: sorry, too many clients already.

Three health endpoints and only two are usable as probes. /health/liveliness and /health/readiness require no authentication and answer fast. Bare /health demands a key and fires a real request against every model in the catalogue, with its cost in tokens.

The resilience numbers are 2, 3 and 5: two retries, three allowed failures and five seconds of cooldown, the last one despite the function’s own documentation saying the default is 1.

request_timeout is 6,000 seconds. A hundred minutes holding a connection against an engine that does not answer.

And a warning about the performance figures: the project publishes 8 ms of p95 overhead on one benchmark and 257.7 ms of p99 on another. Both are theirs, and the difference is in the shape of the load.

You are here: the gateway layer, the day after

In the seven-layer stack, this article lives entirely in the gateway layer. The difference with the selection article is the one that separates buying from maintaining: there the question was which piece to put in front of the fleet, and here it is what happens when that piece has been serving for six months, the team has tripled the traffic and someone has added an agent that fires twenty calls per interaction.

The gateway has an uncomfortable property worth keeping in mind: it sits on the critical path of every request and contributes not a single token. All it does is coordination, and any coordination that fails turns into an outage of the whole inference service, even with the GPUs perfectly healthy.

The analogy: the switchboard and its notebook

A hospital switchboard does three jobs at once. It puts calls through, which is the urgent part. It writes down in a notebook who called, to which extension and for how long, which is what sustains the billing. And it consults a list of extensions to know where to put each call through.

The three jobs compete. If the operator stops to write each line in the notebook before putting the next call through, the waiting queue grows. If there are three operators and each keeps her own notebook, the totals do not add up. And if the list of extensions is consulted in a filing cabinet on another floor, every call costs the trip.

The three tensions have an exact equivalent in LiteLLM. The notebook is the SpendLogs in Postgres, which is why the transaction buffer in Redis exists. The three operators with three notebooks are the pod replicas applying limits independently. And the filing cabinet on another floor is usage-based routing, which adds a Redis lookup inside the request path.

The process model: one worker per pod

The production documentation is explicit: on Kubernetes, --num_workers 1 and scale with replicas. On a virtual machine with no orchestrator, NUM_WORKERS equal to the number of vCPUs.

The reason is not ideological. An additional worker inside the same pod shares the container’s memory limit with the others, multiplies the database connections by the same factor, and registers a copy of every periodic job. With separate replicas, each one has its own resource budget, its own connection quota and a place where the scheduler can put it.

The recommended sizing is 1 vCPU and 4 GiB per worker, and the part that gets overlooked is that these go as requests and as limits at the same time. Setting less than 4 GiB produces an OOM restart loop that shows up late, once some large request has grown the query engine’s footprint.

resources:
 requests: { cpu: "1", memory: "4Gi" }
 limits: { cpu: "1", memory: "4Gi" }

That footprint needs to be described precisely, because it determines the autoscaling policy. The Prisma query engine keeps a resident memory that behaves as a high-water mark: it grows up to the largest statement the process has executed and does not come back down, because glibc does not return that freed memory to the system. A pod that served a spike yesterday still shows the footprint of that spike today.

Two rules come out of this. The first: autoscale on CPU and never on memory. An HPA on memory sees a metric that only goes up, scales at the peak and never scales back. The recommended target is targetCPUUtilizationPercentage: 60, lower than the 80 the charts ship by default, because the startup probe allows up to 300 seconds before passing the first readiness check and there needs to be headroom for the new pod to be ready before the saturated one goes down.

The second: bound the process lifetime. --max_requests_before_restart 10000 recycles the worker before the high-water mark matters.

Postgres: the arithmetic that blows up when you scale

database_connection_pool_limit is 10 by default. The total number of connections the deployment opens is replicas times workers times pool, and it has to be compared against the Postgres max_connections.

The formula the documentation gives is the inverse, and it is the one to apply when sizing:

database_connection_pool_limit = MAX_DB_CONNECTIONS / (instances × workers)

The detail that hurts in production is in the Helm charts, which ship autoscaling.maxReplicas and keda.maxReplicas at 100. With the default pool, that is on the order of a thousand connections at full scale-out, far above what an untouched Postgres accepts. And the moment the HPA reaches those replicas is exactly the moment of highest load, so connection exhaustion arrives when it can least be afforded.

There are two more valves on the same path. Provider errors are written to the database by default, and under a sustained engine failure that inflates the spend table without contributing anything that is not already in the metrics. They are switched off like this:

general_settings:
 disable_error_logs: true # stops writing provider errors
 proxy_batch_write_at: 60 # batches the spend writes

disable_spend_logs: true is the radical version, which removes the per-request detail from the interface and leaves the cost in Prometheus and in the tracing backend. Before going that far you have to decide whether that table is the system’s audit record, which is the discussion in the previous article: if it is, it cannot be switched off, and the way out is the Redis buffer.

Redis: when it stops being optional

Without Redis, LiteLLM works. Each instance keeps its own in-memory cache and applies its own counters. The consequences are three, and all of them show up as you grow.

Requests-per-minute limits are applied per instance, so a limit of 100 with five replicas is in practice a limit of 500. Cache hits are local, so the same repeated question hits once in five. And there is no leader election, so periodic jobs run in every process at once.

The threshold the documentation gives for enabling the transaction buffer is a thousand requests per second or ten instances. Below that, each instance updates the key, user and team rows directly; above it, they all write to the same rows, locks appear and Postgres starts rejecting with FATAL: sorry, too many clients already.

general_settings:
 use_redis_transaction_buffer: true

The metrics to watch when that is enabled are litellm_in_memory_spend_update_queue_size, litellm_redis_spend_update_queue_size and litellm_pod_lock_manager_size. If the first grows without the second coming down, the flush to Postgres is not keeping up.

A note about the routing strategy, which interacts with this. simple-shuffle consults nothing external. Usage-based routing does, and it adds a round trip to Redis inside the request path. On a homogeneous vLLM fleet behind the same model, the improvement it brings rarely pays for that latency; on a heterogeneous fleet, the conversation is different and the router article covered it.

Background jobs register per worker

This detail appears in no getting-started guide and produces considerable bewilderment when it is discovered. The proxy’s periodic jobs register per uvicorn worker, not per pod. With --num_workers 4 and ten replicas that is forty copies of every job, all of them running with no coordination if there is no Redis for leader election.

The variable that separates them is LITELLM_JOB_ROLE: value serving on the pods that serve traffic and a separate replica with value worker for the jobs. With the one-worker-per-pod recommendation the problem shrinks, but it does not go away while there are several replicas.

Probes: three endpoints and only two are usable

The three endpoints exist and do different things:

Endpoint	Authentication	What it does	What it is for
`/health/liveliness`	No	Returns `I'm alive!`, or 503 during shutdown	`livenessProbe`
`/health/readiness`	No	State of the process and the database. 503 if the configured Postgres does not answer	`readinessProbe`
`/health`	Yes	Fires a real request against every model in the catalogue	Manual diagnosis, never a probe

The last row deserves insisting on. /health spends tokens on every call, one per configured model. Set as a probe, with the Kubernetes default interval and a fleet of eight models, that is thousands of inference requests a day that serve nobody.

There are more endpoints useful for diagnosis: /health/readiness/details, /health/services?service=langfuse to check one specific callback, /health/history and /health/latest for the history, and /health/backlog, which returns the in-flight requests and shows up in the latency section. /health/drain exists but returns 404 unless it is enabled with enable_drain_endpoint: true, and it is protected with X-Drain-Token.

The periodic model checks are configured separately and not in the probe:

general_settings:
 background_health_checks: true
 health_check_interval: 300 # seconds
 background_health_check_model_groups: ["llama-70b", "qwen-30b"]

And one setting with consequences in deployments with no Internet egress: allow_requests_on_db_unavailable: true allows serving traffic with the database down, but leaves /health/readiness always returning 200. With that set, the readiness probe stops detecting the very failure it was designed to detect, and budget and model errors keep blocking just the same.

Retries, cooldowns and fallbacks: the real numbers

The defaults, read from the project’s code and not from the documentation:

Constant	Value	What it governs
`DEFAULT_MAX_RETRIES`	2	Retries before moving on to fallbacks
`DEFAULT_ALLOWED_FAILS`	3	Failures before cooling down a deployment
`DEFAULT_COOLDOWN_TIME_SECONDS`	5	Seconds a deployment stays out
`SINGLE_DEPLOYMENT_TRAFFIC_FAILURE_THRESHOLD`	1000	Minimum requests before applying the cooldown logic with a single deployment
`DEFAULT_REQUEST_TIMEOUT_SECONDS`	6000.0	Request timeout

Besides the failure counter, there is a proportional criterion: a deployment enters cooldown if 50% of its requests fail in any given minute.

On the cooldown there is a discrepancy worth knowing about before debugging blind. The docstring of the router function says cooldown_time defaults to 1, and the code uses the constant of 5. The code wins.

The fallbacks are three separate lists and are applied after the retries are exhausted:

router_settings:
 num_retries: 2
 timeout: 120
 fallbacks: [{"llama-70b": ["qwen-30b"]}]
 context_window_fallbacks: [{"llama-70b": ["llama-70b-128k"]}]
 content_policy_fallbacks: [{"llama-70b": ["llama-70b-sin-guardrail"]}]

A note on the behaviour when everything is in cooldown: if no deployment is left available in the group, LiteLLM falls back onto a specific model_info.id, skipping the cooldown check. It is a deliberate degradation, serve something rather than return 503, and it is worth knowing it exists because it masks the real state of the fleet.

And a capability lost along the way: since version 1.85.0, the mock_testing_fallbacks parameter and its two siblings are stripped from requests arriving through the proxy and have no effect. Fallbacks can no longer be tested against the proxy with a doctored request; it has to be done against litellm.Router directly, in a test. Anyone who had a smoke test built on that has it broken without warning.

The six thousand second timeout

It deserves its own section because it is the setting with the worst damage-to-effort ratio in the whole system. request_timeout is 6,000 seconds. A hundred minutes. An inference engine that stops answering without closing the connection holds a worker for that long, and with one worker per pod that is an entire pod out of service per trapped request.

general_settings:
 request_timeout: 600

Ten minutes is still generous for a long generation, and it bounds the damage.

Overhead: the project’s two figures

LiteLLM publishes two measurements of its own overhead that differ by a factor of thirty, and both are legitimate because they measure different loads.

The benchmarks page, with Locust, a thousand users with think time and a fake endpoint, gives for four instances an x-litellm-overhead-duration-ms of median 2 ms, p95 8 ms and p99 13 ms at 1,170 requests per second. With two instances, median 12 ms, p95 29 ms and p99 43 ms. The page itself warns that those figures correspond to around 130 in-flight requests, and that a closed-loop client with no think time keeps a thousand in flight and sees on the order of eight times that latency at the same rate, by Little’s law.

The AIGatewayBench benchmark, from July 2026, measures the Python proxy at 257.7 ms of added p99 and 329.5 MB of peak memory, with no callbacks, no spend tracking and no persistence. The Rust variant, in beta, gives 0.7 ms.

The reading for an in-house platform: the first figure is the best case with human clients, and the second is the order of magnitude to plan for agentic load, which arrives in a closed loop against fast responses. There is one reported case that resembles this scenario closely: an in-house OpenAI-compatible engine served around 16 requests per second directly and around 9 through LiteLLM, with the degradation growing with concurrency, on a 4 vCPU, 8 GB pod.

The latency gap the metrics cannot see

LiteLLM’s timers start when its handler starts. The time the request spends queued in the uvicorn event loop, before getting there, appears in none of its metrics. The example in its own troubleshooting guide is a case in which LiteLLM records 10 seconds and the user experiences 20.

It is detected by comparing two sources: GET /health/backlog or the litellm_in_flight_requests gauge against the response time measured by the load balancer in front. If they diverge, the gap is in the input queue and the answer is more replicas, not more tuning.

Two headers are always on and serve to watch it without instrumenting anything: x-litellm-overhead-duration-ms and x-litellm-callback-duration-ms. For the second, the documentation sets the suspicion threshold at 100 ms, above which the diagnosis is that the payloads are too large.

And the first cause of latency that guide mentions is none of the above: LITELLM_LOG=DEBUG serialises the payload with json.dumps(indent=4) synchronously, and with payloads over 2 MB that alone can cost between 2 and 5 seconds per request. A log level set to debug a problem and forgotten there produces exactly the problem it was meant to debug.

The failure mode to put in the runbook

There is an open issue describing a cascade with a characteristic shape. Under sustained 429s from the engine upstream, the pods stopped answering readiness probes during startup, Kubernetes killed them on failed probes, and the restart put more pressure on an upstream that was already saturated. Peaks of 500 requests per second, average of 60, between two and five replicas with 1.3 CPU and 4 GB.

The shape of the failure is what matters more than the specific case: the engine’s saturation turns into a restart loop of the gateway, and from the outside it looks like the problem is the gateway. The defences are the ones already in this article, applied together: startupProbe with a wide margin so readiness does not compete with startup, request_timeout bounded so trapped requests get released, cooldowns that take the deployment returning 429 out of rotation, and CPU autoscaling at 60% so the capacity exists before it is needed. It fits with what was covered in incident response runbooks.

Upgrading without cutting traffic

The deployment is a normal Deployment with RollingUpdate, maxUnavailable: 0 and maxSurge: 1, and there is no further mystery to it except on one point: schema migrations. LiteLLM uses Prisma and applies migrations at startup. With several replicas coming up at once after a version change, the migration must be run by a single process, which is another reason for the dedicated replica with LITELLM_JOB_ROLE: worker.

Before a version bump, two checks that pay for themselves: read the release notes looking for changes in the metadata emitted to the callbacks, because a change like that breaks saved dashboards without touching the service, and verify that the new configuration starts with --detailed_debug on a separate pod before applying it to the fleet.

Go-live checklist

--num_workers 1 and scaling by replicas.
requests equal to limits, 1 vCPU and 4 GiB per worker.
HPA on CPU at 60%. Never on memory.
database_connection_pool_limit computed from max_connections, and the chart’s maxReplicas lowered to something the database can take.
Redis as soon as there is more than one replica, and use_redis_transaction_buffer from ten instances or a thousand requests per second.
livenessProbe on /health/liveliness, readinessProbe on /health/readiness, and /health out of the probes.
startupProbe with a margin of up to 300 seconds.
request_timeout: 600.
LITELLM_LOG out of DEBUG, and disable_error_logs: true.
A dashboard with x-litellm-overhead-duration-ms, x-litellm-callback-duration-ms, litellm_in_flight_requests and the response time of the load balancer in front.

Traps and things that are not what they look like

The HPA scales up and never back down. It is measuring memory, which in this process only goes up.

The pods die in a loop right after deploying. Less than 4 GiB of limit. The footprint appears with the first large request, not at startup.

Postgres starts rejecting connections at the peak. The chart has maxReplicas: 100 and the pool defaults to 10.

The per-minute limits are enforced badly. There is no Redis, and each replica keeps its own count.

The health check consumes tokens. Someone set /health as the readiness probe.

The readiness probe returns 200 with the database down. allow_requests_on_db_unavailable is set.

A deployment keeps receiving traffic after failing. The cooldown is 5 seconds, not minutes, and with a single deployment in the group you have to pass a thousand requests for the logic to apply.

The fallback test no longer tests anything. The simulation parameters are stripped from requests arriving through the proxy since 1.85.0.

The p99 is ten times worse than the published benchmark. The benchmark has think time and the real load is agentic.

Closing

None of these settings is complicated and all of them are one line. What unites them is that their defaults describe a small deployment, with one instance, a human client on the other side and a commercial provider taking care of capacity. An in-house inference factory is the opposite on all four axes: several replicas, automated clients, and an engine whose capacity is finite and known.

Of the ten points in the checklist, three decide almost everything in an on-premise deployment: the process model with its sizing, the database connection arithmetic, and the request timeout. The other seven are the ones that avoid the Saturday phone call.

Sources

LiteLLM, Production best practices (topología, workers, memoria, pool de conexiones, LITELLM_JOB_ROLE, buffer de transacciones en Redis, request_timeout): https://docs.litellm.ai/docs/proxy/prod.
LiteLLM, Health checks (los tres endpoints, comprobaciones de fondo, enable_drain_endpoint): https://docs.litellm.ai/docs/proxy/health.
LiteLLM, Reliability (fallbacks, reintentos, deprecación de los parámetros de simulación en la 1.85.0): https://docs.litellm.ai/docs/proxy/reliability.
LiteLLM, Benchmarks y Load test advanced (las cifras de sobrecarga por número de instancias y la advertencia de la ley de Little): https://docs.litellm.ai/docs/benchmarks · https://docs.litellm.ai/docs/load_test_advanced.
LiteLLM, Rust AI Gateway benchmarks (los 257,7 ms de p99 del proxy en Python): https://docs.litellm.ai/blog/rust-ai-gateway-benchmarks.
LiteLLM, Latency overhead (el hueco de la cola de uvicorn, las cabeceras de diagnóstico, el coste de LITELLM_LOG=DEBUG): https://docs.litellm.ai/docs/troubleshoot/latency_overhead.
LiteLLM, DB info (disable_spend_logs, disable_error_logs y qué se pierde con cada uno): https://docs.litellm.ai/docs/proxy/db_info.
LiteLLM, litellm/constants.py (los valores por defecto de reintentos, fallos permitidos, enfriamiento y timeout): https://github.com/BerriAI/litellm/blob/main/litellm/constants.py.
LiteLLM, litellm/router.py (la discrepancia entre el docstring de cooldown_time y la constante que usa el código): https://github.com/BerriAI/litellm/blob/main/litellm/router.py.
LiteLLM, issue #15526 — cascada de 429 sostenidos, sondas de readiness fallidas y bucle de reinicios: https://github.com/BerriAI/litellm/issues/15526.
LiteLLM, issue #21046 — degradación de throughput contra un motor propio compatible con OpenAI: https://github.com/BerriAI/litellm/issues/21046.

LiteLLM and Langfuse: the operational pair, and the four places where the trace is lost between the gateway and the dashboard

Mon, 07 Sep 2026 18:00:00 +0200

This post opens the operational track of the control layer. The previous articles left the pieces chosen and assembled: choosing the OSS gateway and the L7 inference router explain why LiteLLM ends up in front of the fleet, and Langfuse from the inside takes apart the backend that receives the traces. Here we deal with the seam between the two, which is where the work of the next three years lives.

TL;DR

The LiteLLM plus Langfuse pair is configured with a list of callbacks and two API keys, and from there four matters decide whether the dashboard is good for operating or only for showing in a demo.

There are three integration routes and they are not equivalent. The native langfuse callback is tied to the Python SDK v2 (langfuse>=2.59.7,<3.0 in LiteLLM’s pyproject.toml), which writes against the legacy ingestion that Langfuse Cloud retires on 16 November 2026. The langfuse_otel preset exports over OTLP and is the path recommended in the current documentation of both projects. The OTel v2 route, behind the LITELLM_OTEL_V2=true flag, changes span names, leaves the success status at UNSET and, above all, its Langfuse mapper does not read the request metadata: session_id, trace_user_id, tags and trace_id stop arriving.

The cost of a self-hosted model comes out as zero unless a price is registered. get_model_info() raises an exception for a model that is not in the pricing map, completion_cost() propagates it, the logger catches it, stores it in a debugging field and returns None, and the payload turns it into 0.0. There is no warning at the default log level. A team’s budget is never exhausted because its spend is always zero.

Client-gateway-engine correlation does not come out of the box. LiteLLM does continue the client’s trace if a traceparent arrives, and it also extracts from it the 32-hex trace-id to use as litellm_trace_id, which makes the Langfuse identifier equal to the W3C one. Upstream the story changes: forward_traceparent_to_llm_provider is False by default, the function that applies it returns early if no otel logger is loaded, and when it does act it forwards the client’s original header, so the vLLM span ends up a sibling of LiteLLM’s rather than a child. Add to that the fact that vLLM exports OTLP over gRPC by default and Langfuse does not accept gRPC.

Between the request and the dashboard there are four bounded queues that drop events. LiteLLM’s logging thread pool (100 threads, 10,000 pending tasks, dropping once full), the asynchronous LoggingWorker (50,000, and once full it aggressively flushes half), OTel’s BatchSpanProcessor (2,048 spans by default) and Langfuse’s ClickhouseWriter, which drops rows after exhausting the retries with no dead-letter queue. None of the four applies back pressure on the request, which is exactly what you want, and none of them guarantees delivery, which is exactly what an auditor asks about.

The rest of the article develops the four points with configuration applicable to a generic 4×H100 cluster with vLLM, the overhead figures the project itself publishes, and the honest limit of what this data sustains before ENS or ISO/IEC 42001.

You are here: the seam between DEPLOY and OBSERVE

In the seven-layer stack, LiteLLM lives in the gateway layer and Langfuse in the observability one. The separation is clean on the diagram and false in operation: the gateway is the only point in the system that sees the complete request before chopping it up and the complete response after reassembling it, so it is also the only reasonable place from which to emit the trace. The inference engine sees tokens and latencies; it does not see who is asking or how much that team has spent this month.

That turns LiteLLM into a telemetry producer as well as a traffic distributor, and moves onto the gateway a responsibility that was not in the original design: if the callback fails, the request is served all the same and nobody notices that the dashboard has been incomplete for two days.

The analogy: the toll booth that issues the ticket

A motorway toll booth does two independent jobs. It raises the barrier, which is its visible function, and it issues a ticket with number plate, time, section and amount, which is the function the concession lives off. The two share a booth and do not share a destination: the barrier responds in a hundred milliseconds because there is a car waiting, and the ticket travels to a processing centre over a slower channel that batches thousands of records before sending them.

The design is well thought out as long as its consequence is understood. If the processing centre goes down, cars keep going through. If the ticket queue fills up, tickets are thrown away, traffic is not stopped. The concession accepts losing records rather than causing a jam, and charges on a statistical sample it knows to be imperfect.

LiteLLM and Langfuse work the same way. The four queues in the corresponding section are the slow channel, and all of them prefer to drop rather than block. Whoever builds the pair inheriting the complete analogy gets the operation right. Whoever thinks the ticket is an accounting entry ends up explaining to an auditor why traces from a Tuesday afternoon are missing.

Three routes, and one with an expiry date

As of September 2026 three paths coexist for getting a request from LiteLLM to Langfuse. LiteLLM’s documentation already labels the first as legacy.

Route	How it is enabled	Transport	Status
Native `langfuse` callback	`success_callback: ["langfuse"]`	Langfuse Python SDK v2, legacy ingestion	Maintained for compatibility. Tied to `langfuse<3.0`
`langfuse_otel` preset (v1)	`callbacks: ["langfuse_otel"]`	OTLP/HTTP against `/api/public/otel`	Recommended by both projects
OTel v2 route	`callbacks: ["langfuse_otel"]` + `LITELLM_OTEL_V2=true`	OTLP/HTTP, new mapper	Opt-in. Changes semantics and loses metadata

The date that orders the decision is 16 November 2026. Langfuse’s compatibility matrix marks the legacy batch ingestion of the Python SDK v2 as retired on that date for Cloud, at which point trace ingestion through that route stops working. On self-hosted the endpoint survives, but a v4 server in events_only mode returns 400 for every event type except score-create and sdk-log.

Since LiteLLM’s native callback is pinned to langfuse>=2.59.7,<3.0 in its own pyproject.toml, and its code calls APIs that only exist in SDK v2 (self.Langfuse.trace(...), trace.generation(...)), there is no way to move up an SDK without changing route. Anyone who today has success_callback: ["langfuse"] against Langfuse Cloud has a migration with a date on it. Anyone who has it against a self-hosted Langfuse has one with some slack, because v3 receives security patches until the end of January 2027, but they have one all the same.

One detail of the OTLP route worth checking in older deployments: the x-langfuse-ingestion-version: 4 header is the one that enables real-time ingestion. Without it the traces arrive, but with a delay that Langfuse’s documentation puts at around ten minutes (the compatibility page says fifteen, and that discrepancy remains unresolved in their own documents). LiteLLM started sending it in version 1.95.0, of August 2026. A proxy older than that version against a Langfuse v4 produces the classic symptom: the dashboard works, but it is always behind and nobody knows why.

The configuration that works today

For a generic 4×H100 cluster with vLLM behind it and Langfuse self-hosted in the same Kubernetes, the OTLP route in its v1 variant is the one that gives the most functionality with the fewest surprises.

model_list:
 - model_name: llama-70b
 litellm_params:
 model: hosted_vllm/meta-llama/Llama-3.3-70B-Instruct
 api_base: http://vllm-llama70b.inference.svc.cluster.local:8000/v1
 api_key: os.environ/VLLM_API_KEY
 # Price derived from the cluster's €/GPU-hour. Without this, spend = 0.0
 input_cost_per_token: 0.00000031
 output_cost_per_token: 0.00000089

litellm_settings:
 callbacks: ["langfuse_otel"]
 turn_off_message_logging: false # a conscious decision, see the PII section
 langfuse_default_tags: ["model_group", "user_api_key_team_alias", "cache_hit"]

general_settings:
 disable_error_logs: true # provider errors inflate SpendLogs
 proxy_batch_write_at: 60

The credentials travel through the environment. LANGFUSE_PUBLIC_KEY, LANGFUSE_SECRET_KEY and LANGFUSE_HOST are enough: the preset builds the Authorization: Basic base64(public:secret) header and adds the ingestion version one, so there is no need to touch OTEL_EXPORTER_OTLP_HEADERS by hand. The preset’s default host points to the US region of Langfuse Cloud, a relevant detail for anyone building a platform with data sovereignty requirements: if the variable is not defined, the telemetry leaves Spain without anyone having decided it.

For real multi-tenancy, LiteLLM supports per-team Langfuse credentials (POST /team/<id>/callback with callback_vars) and per-key ones (metadata.logging[] when generating the key), and the key’s wins over the team’s without mixing. There is a limitation that breaks the architecture of anyone who planned to separate tenants across different Langfuse instances: under OTel v2, a per-key or per-team langfuse_host does not move that tenant to another host, because the exporter fixes the destination at startup and the only thing that varies per tenant are the headers.

And one operational warning with years of history behind it: MAX_LANGFUSE_INITIALIZED_CLIENTS is 50 by default because each Langfuse client starts a thread. The comment in LiteLLM’s code is explicit about the incident that motivated it, with the CPU at 100% from repeated initialisations. A deployment with per-team credentials and many teams hits that ceiling.

What is lost when moving to OTel v2

The LITELLM_OTEL_V2=true flag brings a step up in quality in the proxy’s general instrumentation, and at the same time a concrete regression on the path to Langfuse that the documentation acknowledges without underlining it.

The Langfuse mapper of the v2 route emits langfuse.observation.type, langfuse.observation.model.name, the observation identifier, the model parameters, input and output, usage_details and cost_details, plus langfuse.trace.metadata.team_id and team_alias. It does not emit session.id, nor user.id, nor langfuse.trace.tags, nor langfuse.trace.id, nor the generation name, nor parent_observation_id. LiteLLM’s documents put it this way: those attributes are set by the preset from the request and the response, not from metadata supplied by the client.

The practical consequence is that an application that today groups conversations by sending metadata: {"session_id": "...", "trace_user_id": "..."} in the request body stops grouping them when the flag is enabled, with no error and no warning. The traces keep arriving, loose.

The other v2 changes are semantic and affect alerts already written. The root span goes from being called Received Proxy Server Request to being named after the route (POST /v1/chat/completions), the inference span goes from litellm_request to {operation} {model}, the guardrail spans stop hanging off the inference span and hang off the root instead, and the success status stays at UNSET instead of OK. Any rule that fires on status == OK stops firing. The way back is to remove the flag and restart.

Cost: why the trace says zero

This is the part that takes the most time to diagnose and the least to fix.

LiteLLM computes the cost of a request from a per-model pricing map. For commercial provider models the map comes filled in. For hosted_vllm/whatever, which is all there is in an in-house inference factory, the map is empty, and the failure path is as follows:

litellm.get_model_info() raises an exception with the message This model isn't mapped yet.
completion_cost() and response_cost_calculator() propagate the exception.
Logging._response_cost_calculator() catches it, writes a StandardLoggingModelCostFailureDebugInformation into response_cost_failure_debug_information, logs it at debug level and returns None.
The standard logging payload does llm_response_cost = raw_response_cost or 0.0.

The system delivers a spend of zero, with no visible error at the default log level, and writes it into the spend column of LiteLLM_SpendLogs. The per-key, per-team and per-organisation budgets are evaluated against that zero and are never exhausted. The Langfuse dashboard shows a total cost of zero for everything self-hosted, which is precisely the part you wanted to measure.

Registering the price

The pricing keys are accepted both in litellm_params and in model_info, and those in litellm_params win when they are in both places. The complete set has 91 fields, but for a deployment with vLLM two are enough, and there are three things to know about them.

The first is that the price is per individual token, in USD. There are no per-thousand variants: searching for per_1k or per_1000 in the pricing model returns nothing. A three-orders-of-magnitude error here is easy to make and hard to see, because it produces plausible figures.

The second is that input_cost_per_request exists in the schema and is not applied in chat calls: a model registered with that field returns a cost of 0.0. To charge per request it has to be converted into a cost per token.

The third is that input_cost_per_second needs the response time and is computed as price times response_time_ms / 1000, so outside the proxy’s request path it silently returns zero.

The number that goes into those two keys comes from the article on FinOps and multi-tenancy with LiteLLM and the one on cost per token and per request: amortisation plus energy divided by tokens served. The example arithmetic, to set the order of magnitude: a cluster whose fully loaded cost is 9.60 USD per hour sustaining 3,000 output tokens per second produces 10.8 million tokens an hour, that is 0.00000089 USD per output token. That is the value that goes into output_cost_per_token. If the cluster’s average utilisation is 30 % and not 100 %, the divisor changes and the price triples, which is the same lever that article develops.

Before sending traffic, the /spend/calculate endpoint accepts the same parameters as completion_cost and returns the computed cost. It is the cheap way to check that the price resolves.

How that number reaches Langfuse, and who wins

The two integration routes write the cost into different fields. The native callback sends it as usage.total_cost of the v2 schema, and duplicates it in the trace metadata as litellm_response_cost. The OTLP preset emits langfuse.observation.cost_details as a JSON string of the form {"total": cost}, and langfuse.observation.usage_details with input, output and total.

The rule on the Langfuse side is documented and is the right one for this case: when there is ingested cost and inferred cost, the ingested one takes priority. Langfuse only computes from its own pricing table when the emitter sends no cost, and changes to a model definition apply to new generations, with no retroactive recalculation. A self-hosted model therefore needs no pricing definition in Langfuse: it is enough for LiteLLM to send cost_details.

One detail remains that explains a puzzling symptom. Langfuse stores the ingested cost and the computed one in separate columns, totalCost and calculatedTotalCost, and the session header totals add up only the first. A mixed deployment where part of the traffic brings cost and part lets Langfuse compute it shows sessions at zero while the individual generations show cost. There is an open issue about it.

The tokens of a streaming response

Almost all of an assistant’s traffic is streamed, and there the OpenAI usage object only appears if the client asks for stream_options.include_usage. LiteLLM solves this by injecting the option on its own: on the asynchronous completion routes with stream: true, if the caller did not set it and all the candidate deployments support it, it adds it and marks _litellm_strip_stream_usage so the usage block is used in the cost calculation and not returned to the client. The general_settings.always_include_stream_usage flag forces or disables that behaviour, and is worth setting to True when capability detection does not recognise the vLLM deployment.

When the usage is missing anyway, LiteLLM reassembles the response with stream_chunk_builder() and counts tokens with the tokeniser. It is an approximation, and it produces the second path towards a cost of zero. The two paths to zero are told apart by looking at total_tokens in LiteLLM_SpendLogs: zero tokens with a correct price is a usage problem, correct tokens with zero cost is a price problem.

Correlation: from the client to the engine

The operational question that justifies the whole setup is being able to take a user complaint, find their request and see where the time went. That requires the trace identifier to survive three hops.

What does work

If the client sends traceparent, LiteLLM’s spans hang off the existing trace. In addition, and independently of the OTel subsystem, litellm_pre_call_utils.py extracts by regular expression the 32-character hexadecimal trace-id and assigns it to data["litellm_trace_id"] as a last resort, and takes the session_id from the W3C baggage header. That makes the Langfuse trace identifier equal to the W3C trace identifier, which is the most robust correlation key available: with it, an identifier copied from the application log opens the trace in Langfuse with no translation.

For the routes that are not OTel v2, moreover, headers prefixed with langfuse_ inject per-request metadata. langfuse_trace_id, langfuse_trace_user_id and langfuse_trace_metadata are documented, but the code strips the prefix and writes the rest into the metadata, so any body key works as a header. Two warnings about this: header values arrive as strings and are not interpreted as JSON, and Nginx drops headers with underscores by default (underscores_in_headers off), which produces the symptom that the metadata works locally and disappears behind the ingress.

What does not work without effort

Towards the engine, propagation is off. The flag is litellm.forward_traceparent_to_llm_provider, it is False, and its implementation has three behaviours to know about before enabling it:

The function returns immediately if open_telemetry_logger is None. A deployment with callbacks: ["langfuse_otel"] and without the generic otel callback propagates nothing, whatever the flag is set to.
When it does propagate, it copies the client’s traceparent header as is, instead of injecting the context of LiteLLM’s active span. The vLLM span ends up hanging off the client’s span-id, a sibling of the gateway call’s span. The trace is one, the hierarchy is wrong.
If the client sends no traceparent, none is created.

Add to that the fact that LiteLLM’s general policy is not to forward client headers to the provider, and that the forward_client_headers_to_llm_api allowlist covers headers prefixed with x-, among which traceparent is not. This is the only route.

The vLLM side, and why the Collector stops being optional

vLLM enables traces with --otlp-traces-endpoint, and --collect-detailed-traces with model, worker or all adds forward and execute times, with a performance impact its own documentation warns about. The attributes it emits are in vllm/tracing/utils.py and are the most useful part of the setup: gen_ai.latency.time_in_queue, time_to_first_token, time_in_scheduler, time_in_model_prefill, time_in_model_decode, time_in_model_forward, time_in_model_execute, time_in_model_inference and e2e, plus the token counters. That breakdown is what turns a “it’s slow” into “the time goes in the queue, not in decode”, which is the conversation you want to have during an incident. It fits with what was already covered in instrumenting vLLM with OTel and with the metrics in DCGM and vLLM.

There are three constraints that decide the architecture:

vLLM exports over gRPC unless OTEL_EXPORTER_OTLP_TRACES_PROTOCOL=http/protobuf is set, and its gRPC exporter is built with insecure=True hard-coded. Langfuse, for its part, does not support gRPC. And vLLM emits the old convention names (gen_ai.usage.prompt_tokens, gen_ai.usage.completion_tokens), renamed in the specification to input_tokens and output_tokens, plus a gen_ai.latency.* family that was never standard and that the file itself documents as its own attributes until they are normalised.

An OpenTelemetry Collector in between solves all three at once: it receives gRPC, exports HTTP with the Basic header towards /api/public/otel/v1/traces, and renames attributes with a transform. It is also the only place where tail sampling can be done, which is the next point. And add the warning from the llm-d community, which holds for any deployment with Envoy or sidecars in between: OTEL_SERVICE_NAME has to be set per engine, because vLLM sends no service name and without it every span arrives as unknown_service.

The four serial queues

A request’s telemetry crosses four bounded buffers before it is queryable. All four drop once full and none blocks the request.

Queue	Capacity	What it does once full
LiteLLM logging thread pool	100 threads, 10,000 pending tasks	Drops the new ones with a rate-limited warning
Asynchronous `LoggingWorker`	50,000 events, concurrency 100	Aggressively flushes 50 % of the queue; `enqueue()` documents that it never blocks
OTel’s `BatchSpanProcessor`	2,048 spans (`OTEL_BSP_MAX_QUEUE_SIZE`), batch 512, sent every 5 s	Drops spans silently
Langfuse’s `ClickhouseWriter`	Bounded retries	Drops rows after exhausting retries, with no dead-letter queue

The first is documented in LiteLLM’s own code with a sentence that saves half an hour of discussion: logging is best-effort and, once the cap is reached, new entries are dropped with a rate-limited warning instead of being queued forever. The third is a barely visible consequence: LiteLLM builds the BatchSpanProcessor without adjusting max_queue_size, so the OTel SDK defaults govern, and no LiteLLM documentation mentions the OTEL_BSP_* variables that change them. The Langfuse SDK does the same on its own account: it sets batch size and interval, and leaves the queue size at the default.

On top of this comes a fifth loss point that is not a queue: LiteLLM does not call flush() per request on the native route, so a SIGKILL on the pod takes the last batch with it. On Kubernetes, this is an argument for a generous terminationGracePeriodSeconds and a preStop that gives some slack.

None of this is a defect. It is the right design decision for a piece that sits on the critical path of inference. What changes is what can be asserted with the resulting data.

What cannot be promised to the auditor

A dashboard built on this pipeline answers statistical questions well: what is the p95 of time to first token for the model group, which team consumes the most, what proportion of requests fails, how cost per request evolves. That is what it is designed for, and that is what it is good for.

It does not answer completeness questions well, which are the ones an ENS auditor or an ISO/IEC 42001 management system auditor asks. Faced with “show me that I have a record of every interaction with the AI system during March”, the honest answer is that the traces are a best-effort record with four documented points of silent dropping, with no dead-letter queue on the last hop and no durability guarantee on any of them.

The clean separation is the one already proposed in technical controls ENS, 42001 and the AI Act: the regulatory audit record is a different artefact, with a different write path, durability guarantees and retention of its own. In the pair this article deals with, the LiteLLM_SpendLogs table in Postgres is a much better candidate than Langfuse for that function, because it is a transactional write against a relational database, and because it already contains request_id, api_key in hashed form, team_id, organization_id, end_user, model, tokens, startTime, endTime, status and requester_ip_address. Writing the audit event from the same place and in the same transaction as the spend is the sensible route; delegating it to the tracing pipeline is not.

Here an operational tension appears that has to be resolved deliberately. LiteLLM’s performance recommendation for production includes disable_error_logs: true and, in very loaded deployments, disable_spend_logs: true, because provider errors inflate the table and concurrent writes against the same rows produce locks and exhaust the Postgres connections. Disabling spend_logs to gain performance and at the same time leaning on that table for the audit record are incompatible decisions. Above a thousand requests per second or more than ten instances, the way out is use_redis_transaction_buffer: true, which batches the spend updates in Redis before pushing them down to Postgres, watching the litellm_in_memory_spend_update_queue_size and litellm_redis_spend_update_queue_size metrics.

PII: switching off the content before it leaves

An LLM trace contains, by definition, the user’s prompt. In an on-premise deployment for a regulated customer, that content may be exactly what cannot leave the application’s domain, not even towards the observability backend in the same cluster.

The controls on the proxy side, from most global to most fine-grained:

litellm_settings.turn_off_message_logging: true cuts messages and responses in every callback, and keeps the metadata and the spend. The redaction replaces the content with the REDACTED_BY_LITELLM sentinel in messages, choices, reasoning_content, streaming deltas, tool call arguments and in the standard logging object.
The x-litellm-enable-message-redaction: true header enables redaction per request, and LiteLLM-Disable-Message-Redaction: true disables it. The second is checked first and wins over everything else, including the global configuration, which makes it a control to block at the ingress if the policy is that content is never logged.
redact_user_api_key_info: true removes the hashed token, the user_id and the team_id from the log.
Per request and per callback, metadata: {"mask_input": true, "mask_output": true}.
The body parameter "no-log": true disables logging of that call. It is a body parameter, not a header, and it can be disabled globally with global_disable_no_log_param: true so a client cannot make itself invisible.

On the OTel v2 route the policy is inverted towards the safe side: content is off by default and is opened with OTEL_INSTRUMENTATION_GENAI_CAPTURE_MESSAGE_CONTENT, with values no_content, span_only, event_only and span_and_event. The documentation underlines that the gate is applied centrally and that a user’s request cannot force their prompt towards the backend while capture is disabled. It is also the stance the OpenTelemetry specification recommends, whose preferred pattern for production is to store the content in external storage and leave only the reference in the span.

One specific version to check in the inventory: below LiteLLM 1.98.0, with per-team Langfuse credentials, the team’s own credentials ended up inside its traces. That version changed the origin of the metadata emitted to the StandardLoggingPayload to cut it off. The same change removed a couple of dozen metadata fields (model_group, deployment, queue_time_seconds, attempted_retries and others), so it breaks saved dashboards and alerts that filter on them. It is an upgrade with work attached, and it has to be done anyway. The same applies to the general hardening of the stack, covered in hardening and secrets of the sovereign stack.

The price in latency

LiteLLM publishes two overhead figures that differ by a factor of thirty, and both are theirs.

The benchmarks page, with Locust, a thousand users with think time and a fake OpenAI endpoint, gives for four instances an x-litellm-overhead-duration-ms of median 2 ms, p95 8 ms and p99 13 ms, at 1,170 requests per second. For two instances, median 12 ms, p95 29 ms and p99 43 ms. The page itself warns that those figures hold with around 130 in-flight requests, and that a closed-loop client with no think time keeps a thousand in flight and sees on the order of eight times that latency at the same rate, by Little’s law.

The AIGatewayBench benchmark, of July 2026, measures the Python proxy at 257.7 ms of added p99 with 329.5 MB of peak memory, against 0.7 ms for the Rust variant in beta. That measurement has no callbacks, no spend tracking and no persistence.

The operational reading is that the first figure is the best case and the second the order of magnitude to plan for agentic load, which is the one that arrives in a closed loop against fast responses. To watch it there are two always-on headers: x-litellm-overhead-duration-ms and x-litellm-callback-duration-ms, and the documentation sets the useful threshold of the second at 100 ms, above which the diagnosis is that the payloads are too large.

Three production settings that appear in LiteLLM’s documentation and that are wrong by default:

request_timeout is 6,000 seconds. A hundred minutes holding a connection against a hung upstream. The recommendation is to lower it to 600.

LITELLM_LOG=DEBUG serialises the payload with json.dumps(indent=4) synchronously, and the documentation puts the cost of that serialisation at 2 to 5 seconds with payloads over 2 MB. It is the first cause of latency their troubleshooting guide mentions.

Background jobs register per uvicorn worker, not per pod. With --num_workers 4 and ten replicas that is forty copies of every periodic job. The LITELLM_JOB_ROLE variable separates the pods that serve traffic from the replica that runs jobs.

And one failure mode with an open issue that deserves to be in the runbook: under sustained 429s from the upstream, pods have been reported unable to answer readiness probes during startup, with the ensuing restart loop. It fits with what was covered in incident response runbooks.

Start-up checklist

Ten checks before signing off the installation of the pair:

callbacks: ["langfuse_otel"] instead of success_callback: ["langfuse"], unless you depend on prompt management, which the OTLP preset does not support.
LANGFUSE_HOST defined explicitly. The preset’s default value points to the US region.
LiteLLM at 1.98.0 or higher, because of the team credential leak into the traces, and at 1.95.0 or higher because of the v4 ingestion header.
A price registered for every self-hosted model, per token and in USD, verified with /spend/calculate before opening traffic.
A test request with a known traceparent, checking that the same identifier appears in Langfuse.
An OpenTelemetry Collector between vLLM and Langfuse, with OTEL_SERVICE_NAME per engine and renaming of the old token attributes.
request_timeout: 600 and LITELLM_LOG out of DEBUG.
Tail sampling in the Collector, with a latency and status code policy, to keep 100 % of what is slow and what failed.
A written decision about prompt content, and blocking of the LiteLLM-Disable-Message-Redaction header at the ingress if the policy is not to log.
A dashboard with x-litellm-overhead-duration-ms, x-litellm-callback-duration-ms and Langfuse’s ingestion queue depth (langfuse.queue.ingestion.depth), which is the autoscaling signal for its workers.

Traps and things that are not what they look like

The trace arrives, but ten minutes late. The x-langfuse-ingestion-version: 4 header is missing, which LiteLLM sends from 1.95.0. It is not a ClickHouse capacity problem.

The metadata works in development and disappears in production. Nginx drops headers with underscores by default, and all of this integration’s have one.

Session cost comes out as zero and generation cost does not. Langfuse adds up only the ingested cost in the session totals, not the one it computes itself.

Enabling OTel v2 breaks nothing visible and ungroups the conversations. The v2 mapper does not read session_id or trace_user_id from the request.

A team’s budget is never exhausted. Its spend is zero because the model has no price, and the calculation failure is only logged at debug level.

The alerts stop firing after the migration to v2. The success status becomes UNSET; no rule on OK is satisfied.

Forwarding the traceparent is enabled and it does not reach the engine. The generic otel callback is missing, without which the function that applies it returns before looking at the flag.

All the engines’ spans appear under unknown_service. vLLM emits no service name; OTEL_SERVICE_NAME has to be set per deployment.

Closing

The gateway and the tracing backend are the two pieces of the inference stack that take the least time to install and the longest to operate well. The installation fits in a ConfigMap. The operation consists of four decisions no getting-started guide raises: which integration route to take knowing that one expires in November, where the price per token of a model nobody invoices comes from, what to do with the traceparent at each hop, and what to tell an auditor about data the system drops by design when things get tight.

The answer to the fourth is the one that orders the other three. This pipeline is operational instrumentation, with statistical precision and best-effort guarantees, and separating it from day one from the regulatory audit record avoids the awkward conversation two years from now. With that separation made, the LiteLLM and Langfuse pair is the pair of pieces that turns a fleet of GPUs into a platform with an owner, a known cost and explainable latency.

Sources

LiteLLM, Langfuse integration (rutas de integración, metadata aceptada, cabeceras langfuse_*): https://docs.litellm.ai/docs/observability/langfuse_integration.
LiteLLM, OpenTelemetry v2 y Migración a OTel v2 (preset, captura de contenido, cambios de semántica): https://docs.litellm.ai/docs/observability/opentelemetry_v2 · https://docs.litellm.ai/docs/observability/opentelemetry_v2_migration.
LiteLLM, Logging y Config settings (redacción, turn_off_message_logging, variables de entorno): https://docs.litellm.ai/docs/proxy/logging · https://docs.litellm.ai/docs/proxy/config_settings.
LiteLLM, Custom pricing y Spend tracking (claves de precio, SpendLogs, endpoints de gasto): https://docs.litellm.ai/docs/proxy/custom_pricing · https://docs.litellm.ai/docs/proxy/cost_tracking.
LiteLLM, Production best practices (topología, request_timeout, buffer de transacciones en Redis, LITELLM_JOB_ROLE): https://docs.litellm.ai/docs/proxy/prod.
LiteLLM, Benchmarks y Latency overhead (cifras de sobrecarga y cabeceras de diagnóstico): https://docs.litellm.ai/docs/benchmarks · https://docs.litellm.ai/docs/troubleshoot/latency_overhead.
LiteLLM, Rust AI Gateway benchmarks (la cifra de 257,7 ms de p99 del proxy Python): https://docs.litellm.ai/blog/rust-ai-gateway-benchmarks.
LiteLLM, notas de versión v1.95.0, v1.98.0 y v1.100.0: https://docs.litellm.ai/release_notes/v1.95.0/v1-95-0 · https://docs.litellm.ai/release_notes/v1.98.0/v1-98-0 · https://docs.litellm.ai/release_notes/v1.100.0/v1-100-0.
Langfuse, OpenTelemetry (endpoint, autenticación, tabla de mapeo de atributos, ausencia de gRPC): https://langfuse.com/integrations/native/opentelemetry.
Langfuse, Compatibility y Migrate v3 to v4 (la fecha del 16 de noviembre de 2026 y los modos de escritura): https://langfuse.com/docs/compatibility · https://langfuse.com/self-hosting/upgrade/upgrade-guides/upgrade-v3-to-v4.
Langfuse, Token and cost tracking (prioridad del coste ingerido sobre el inferido): https://langfuse.com/docs/observability/features/token-and-cost-tracking.
Langfuse, Scaling y Sampling (dimensionado, señal de autoescalado, muestreo de cabecera): https://langfuse.com/self-hosting/configuration/scaling · https://langfuse.com/docs/observability/features/sampling.
Langfuse, issue #13468 — descarte de filas en ClickhouseWriter sin cola de mensajes muertos: https://github.com/langfuse/langfuse/issues/13468. Issue #15109 — coste de sesión a cero: https://github.com/langfuse/langfuse/issues/15109. Issue #5173 — Nginx y cabeceras con guión bajo: https://github.com/langfuse/langfuse/issues/5173.
vLLM, vllm/tracing/utils.py y vllm/tracing/otel.py en la etiqueta v0.28.0 (atributos emitidos, protocolo por defecto, insecure=True): https://github.com/vllm-project/vllm/blob/v0.28.0/vllm/tracing/utils.py.
Langfuse, vLLM integration (la advertencia de que vLLM solo exporta contadores y latencias): https://langfuse.com/integrations/model-providers/vllm.
LiteLLM, litellm/constants.py y litellm/litellm_core_utils/logging_worker.py (capacidades de las colas y política de descarte): https://github.com/BerriAI/litellm/blob/main/litellm/constants.py.
OpenTelemetry, Semantic Conventions for Generative AI y el repositorio semantic-conventions-genai (estado de desarrollo, renombrados, captura de contenido): https://github.com/open-telemetry/semantic-conventions-genai.
OpenTelemetry Collector, tailsamplingprocessor (políticas de muestreo por cola y la restricción de instancia única por traza): https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/processor/tailsamplingprocessor/README.md.
llm-d, End-to-end and fine-grained tracing in llm-d (OTEL_SERVICE_NAME y los spans huérfanos tras Envoy): https://llm-d.ai/blog/end-to-end-and-fine-grained-tracing-in-llm-d.

Knative and scale-to-zero for LLM inference: when switching off the GPU saves money and when it costs you the SLO

Mon, 31 Aug 2026 10:00:00 +0200

Closing the platform and self-service run, after Backstage and Kubeflow. The question here is purely economic with a technical trap inside: switching off the GPU when nobody is using it looks like free money, until the first user of the day waits two minutes.

TL;DR

A GPU costs the same serving as it does idle. An H100 on dedicated cloud runs at 2 to 4 euros an hour whether it is used or not, so switching the service off when there is no traffic looks like money given away. Knative is the piece that does it: its autoscaler scales down to zero replicas and brings them back when a request arrives, measuring concurrency instead of CPU, which is what the Kubernetes autoscaler cannot do.

The trap is in the cold start. Knative brings up a pod in seconds, but that is only the beginning: an LLM has to load its weights from disk into HBM and compile its graphs, and there you go from tens of seconds to minutes depending on model size and storage. One reference study measures vLLM startup at around 20 seconds even for a 3 billion parameter model, and weight loading grows linearly with size. The first user of every traffic trough pays that whole wait.

Hence the rule. Switching off the GPU pays off with sporadic traffic, many lightly used models or development environments, and it is a trap with a strict latency SLO, sustained traffic or huge models whose startup breaks any interactive budget. Between all and nothing there are three middle grounds that almost always win: keep one replica warm, share the GPU with model-swap, or speed up weight loading so that startup fits inside the autoscaler’s cycle.

The analogy: the restaurant that switches off the kitchen

A restaurant with the kitchen running at four in the afternoon, without a single customer, burns gas for nothing. The temptation is to switch it off and light it again when somebody walks in. The problem is the time it takes to heat up. If the kitchen is an induction hob, it heats in seconds and switching it off between services is obvious. If it is a wood-fired oven that takes two hours to come up to temperature, switching it off means the first customer of the night eats cold or leaves.

Serving a small model is an induction hob. Serving a 70 billion parameter model is the wood-fired oven: the weights that have to go up into HBM are the wood you have to burn until you reach temperature, and wishing does not speed it up. The decision to switch off the kitchen, that is, to scale to zero, does not depend on whether you want to save gas, but on how long your kitchen takes to heat and on how long your first customer is willing to wait. This whole article is that calculation.

How Knative scales

Knative graduated in the CNCF in October 2025 and has three components; the one that concerns us is Serving, the serverless application component with autoscaling. Its object model has four pieces: the high-level Service, the Configuration that describes the desired state, the immutable Revision which is what actually scales, and the Route that splits traffic between revisions.

What makes scaling to zero possible are two data plane components. The queue-proxy is a sidecar in every pod that measures concurrency and applies the simultaneous request limit. The activator is the key piece: when the service is at zero replicas, requests are routed to it, and it parks them in a queue, tells the autoscaler that capacity is needed, and forwards the parked request once a pod is ready. It is the waiter who takes your order and tells you the kitchen is warming up, instead of shutting the door on you.

KPA versus HPA

Knative’s own autoscaler, the KPA, measures concurrency and requests per second, not CPU or memory. That is the difference that matters: the Kubernetes autoscaler, the HPA, scales on CPU and does not know how to go down to zero, its operational minimum is one. To switch off completely you need the KPA.

The KPA works with two windows. The stable one, 60 seconds by default, averages normal traffic. The panic one, much shorter, reacts to sharp spikes: when demand exceeds twice the capacity of the current replicas, it enters panic mode and scales in one go. The default concurrency target per replica is 100, but for LLM inference the useful value is usually 1 to 4, because each request occupies the GPU heavily; the official KServe example uses a target of 1.

Two timers govern the shutdown. The scale-to-zero-grace-period, 30 seconds by default, is how long the system waits to have the restart machinery ready before withdrawing the last replica. The scale-to-zero-pod-retention-period, zero by default, is the minimum time the last pod survives after the decision to switch off. Raising this second value is one of the levers for not switching off as soon as there is a micro-trough in traffic.

Why an LLM cold start is not starting a pod

Here is the core of the article, and there is a nuance that most discussions skip. Knative bringing up a pod in seconds does not mean the model is ready in seconds. Starting an inference service has several phases, and the two costs that dominate are not the ones you would expect.

A recent study measuring vLLM cold start breaks it down step by step. For a 3 billion parameter model, total startup is around 20 seconds, and the counterintuitive finding is that most of it is CPU-bound, not GPU-bound: process initialisation, library imports, engine construction. Inside that there are two costs that grow, and they are the ones the acceleration techniques attack.

The first is weight loading, which scales linearly with model size. In that same study it goes from half a second for a small model to almost five for a 16 billion one. Projected onto real sizes, the on-disk weight rules: an 8 billion model takes around 16 GB in FP16 and 8 in FP8; a 70 billion one, 140 and 70; a 405 billion one, 810 and 405. Loading 140 GB from disk into HBM does not happen in a blink, and if storage is slow or sits on the network, the minute arrives on its own.

The second is graph compilation. Graph transformation and capture with torch.compile costs 11 to 21 seconds if there is no compilation cache, and drops to 3 to 6 with one. It is half a battle that has nothing to do with the weights, and it is won by keeping the compilation cache between startups.

The operational conclusion is that the enemy is not only HBM: it is weight loading plus compilation. For small models the weights are only a fraction of startup, and switching off the GPU is almost painless. For large models the weights dominate, and the cold start becomes the problem. Loading a 70 billion model with the naive loader of the usual library can run to several minutes, and that is exactly the time your first user waits.

The techniques that shorten the wait

The good news is that the cold start can be attacked on both fronts. On the weights side, streaming loaders such as NVIDIA’s transfer the weights to the GPU concurrently instead of sequentially: the published data takes an 8 billion model from around 48 seconds with the standard loader to around 14 with streaming from SSD, and to under 8 with fast storage. The most graphic case is a 122 billion model, 233 GB on disk, which goes from three and a half minutes to around 37 seconds. That number has a direct consequence: 37 seconds fit inside an autoscaler polling cycle of 30 to 60 seconds; three and a half minutes do not. It is the difference between scaling to zero being viable with large models or not being viable at all.

On the compilation side, the technique is cheaper still: keep the torch.compile cache between startups, and with it the 11 to 21 seconds become 3 to 6. Tools such as Tensorizer and OCI images with the weights packaged inside attack the same problem from another angle, as we saw in the article on speeding up cold start.

Knative and KServe: the two modes

KServe leans on Knative for its serverless mode, and this is what connects everything above to a real platform. KServe has two deployment modes.

The Serverless mode uses Knative Serving and the KPA, supports scaling to zero and measures concurrency and requests per second. It is enabled by setting minReplicas: 0 in the InferenceService. It is the natural path for serving many intermittent models.

The Standard or raw mode uses the Kubernetes HPA, does not scale to zero, and measures CPU and memory. You pick it when you need something Knative restricts, such as mounting several volumes, and you do not need to switch off completely.

There is a third path of particular interest to anyone who already has KEDA deployed: in Standard mode, KServe recommends KEDA for generative inference, and KEDA can scale to zero on Prometheus metrics such as queue length or tokens per second. It is a route to scaling to zero on your own metrics, outside Knative, and it fits better with autoscaling guided by the model’s real saturation signal rather than by generic concurrency.

For anyone serving many lightweight predictive models there is also ModelMesh, the high-density pattern that packs many models into the same pods and rotates them in memory, without leaning on Knative. It is the equivalent of model-swap: instead of switching off and on per model, it shares one GPU between many. It is aimed at classic short-response inference, not at generative LLMs.

The economics

The decision comes down to comparing what you save by switching off with what you risk at startup. The saving is easy to estimate. If a GPU costs $ C $ euros per hour and your service is idle $ H $ hours a day, switching it off saves on the order of

$$\text{daily saving} \approx C \cdot H$$

With an H100 at 3 euros an hour and 16 idle hours a day, that is 48 euros a day per GPU, on the order of 1,400 a month. Across a fleet of several lightly used models the figure multiplies and stops being negligible.

The cost is on the other side, and it is not measured in euros: it is measured in SLO. Scaling to zero only pays off if the cold start fits inside what your worst-case latency tolerates. With a small model and fast storage, tens of seconds at most, and for an internal assistant used in bursts that may be acceptable. With a 70 billion model and the naive loader, minutes, and no interactive SLO survives that.

Between switching off completely and never switching off there are three middle grounds, and almost always one of them is the right answer:

Warm replica, with minReplicas: 1. You remove the first user’s cold start in exchange for paying a continuous GPU-hour. The break-even point is direct: keeping it warm pays off when the cost of that GPU running hot is lower than the reputational or SLO cost of the first user waiting. For a customer-facing service, almost always.

Model-swap or sleep on a shared GPU, as we saw in serving several models on one GPU. The process stays alive and the model is unloaded and reloaded in seconds instead of starting from scratch. It is the middle ground for several models that do not justify a GPU each but do not tolerate a full startup either.

Speed up loading so that the cold start fits inside the autoscaler’s cycle, with weight streaming and a compilation cache. It is what makes real scaling to zero viable with large models, and what turns “three and a half minutes” into “37 seconds”.

Operational traps and honest scepticism

The pod starts in seconds, the model does not. Do not confuse Knative’s startup time with the time until the model answers. The difference is the whole article.

The default concurrency target is 100, and for an LLM that is a disaster. Each request occupies the GPU; a target of 1 to 4 is the reasonable one. Leaving the default 100 means accepting a hundred simultaneous requests on a replica that handles four.

The cold start is not just weight loading. Graph compilation costs what it costs, and it is attacked separately, with a cache. Optimising only storage leaves half the problem untouched.

Cilium and Gateway API are not a supported path yet. Knative supports Gateway API in beta and tests with Istio, Contour and Envoy Gateway; integration with Cilium does not appear as tested and has documented friction. If your ingress is Cilium, verify it before counting on it.

KEDA may be a better signal than the KPA. Scaling on generic concurrency is worse than scaling on the model’s real saturation metric. If you already have KEDA, Standard mode with scaling to zero on your own metrics deserves a test against the default Serverless.

For an inference factory

The decision to switch off the GPU is not ideological, it is arithmetic, and it depends on two numbers: how long your model takes to start and how long your worst user waits. Three rules close the matter.

First, switch off what is intermittent and cheap to start: development environments, small models used sporadically, the long tail of models almost nobody touches. There, scaling to zero is direct money with no appreciable SLO cost.

Second, do not switch off what is critical and expensive to start: the customer-facing model with a strict SLO, and the huge models whose cold start is measured in minutes. There a warm replica costs less than a user waiting.

Third, before deciding, measure your real startup and fix it. Many services are ruled out of scaling to zero over a three-minute startup that, with weight streaming and a compilation cache, drops to thirty seconds and changes the answer. The most profitable lever is not choosing well between switching off or not: it is making switching off cheap.

With this we close the platform and self-service run. The portal shows what is there, the ML toolbox gives what is missing without duplicating what you already have, and scaling to zero decides how much of all that is switched on when nobody is looking. All three answer the same question from different angles: how to put an expensive platform at the disposal of its users without going broke keeping it switched on.

Sources

CNCF, Cloud Native Computing Foundation announces Knative’s graduation — https://www.cncf.io/announcements/2025/10/08/cloud-native-computing-foundation-announces-knatives-graduation/
Knative Docs, Serving architecture — https://knative.dev/docs/serving/architecture/
Knative Docs, Request flow — https://knative.dev/docs/serving/request-flow/
Knative Docs, Configuring KPA-specific autoscaling — https://knative.dev/docs/serving/autoscaling/kpa-specific/
Knative Docs, Configuring concurrency — https://knative.dev/docs/serving/autoscaling/concurrency/
Knative Docs, Configuring scale-to-zero — https://knative.dev/docs/serving/autoscaling/scale-to-zero/
Knative Blog, Managing Gateway API ingress with the Knative Operator — https://knative.dev/blog/articles/gateway-api-ingress-with-knative-operator/
CNCF, KServe becomes a CNCF incubating project — https://www.cncf.io/blog/2025/11/11/kserve-becomes-a-cncf-incubating-project/
KServe Docs, Autoscaling with Kubernetes HPA (Serverless vs Standard) — https://kserve.github.io/website/docs/model-serving/predictive-inference/autoscaling/hpa-autoscaler
KServe Docs, Autoscaler for generative inference (KEDA) — https://kserve.github.io/website/docs/model-serving/generative-inference/autoscaling
KServe GitHub, docs/samples/autoscaling (InferenceService minReplicas 0) — https://github.com/kserve/kserve/blob/master/docs/samples/autoscaling/README.md
Wang et al. (MLSys 2026), Breaking the ice: analyzing cold start latency in vLLM — https://arxiv.org/abs/2606.07362
NVIDIA Developer Blog, Reducing cold start latency for LLM inference with NVIDIA Run:ai Model Streamer — https://developer.nvidia.com/blog/reducing-cold-start-latency-for-llm-inference-with-nvidia-runai-model-streamer
Microsoft Azure SDK Blog, Eliminate LLM cold starts: load models up to 6x faster — https://devblogs.microsoft.com/azure-sdk/eliminate-llm-cold-starts-load-models-up-to-6x-faster-with-azure-blob-storage-and-runai-model-streamer/
Hugging Face Blog, Llama 3.1 (VRAM and precision tables) — https://huggingface.co/blog/llama31
CloudZero, H100 GPU cost 2026 — https://www.cloudzero.com/blog/h100-gpu-cost/

KubeVela and Score: handing over the platform by CLI without teaching the developer Kubernetes

Mon, 31 Aug 2026 09:45:00 +0200

Third and last article in the batch about rebuilding the cloud on-premise. The general map placed the gaps for you and Crossplane covered the infrastructure control plane. Here comes the layer above, the one your developer touches every day: application-centric self-service by CLI.

TL;DR

Crossplane answers “give me a database”, from the infrastructure side. KubeVela and Score answer “deploy my application”, from the workload side, and they do it by CLI, which is precisely the friendly experience Crossplane deliberately does not give you. The two share an article because they cover the same layer from different angles. KubeVela is a complete delivery platform: it uses the OAM model, where your platform team writes some definitions in the CUE language once and the developer deploys their whole application with a fifteen-line file and a CLI (vela up, vela status, vela logs) that feels like Heroku on the outside and is GitOps on the inside. Score is more modest and more portable: a workload specification in a single file that gets translated by CLI into Docker Compose locally and into Kubernetes in production, without running anything in the cluster.

The honest part up front is the limits. Neither of the two appears in the “Adopt” section of the CNCF technology radar from early 2026, where Helm, Backstage and Argo CD do: they are niche against the mainstream. KubeVela version 1.11 is still in alpha, so the stable one is 1.10; its community is concentrated, with around 33 people contributing in 2025. Score is young, its Kubernetes implementation has few stars and its sponsor, Humanitec, was acquired by Akamai. And KubeVela’s power lives in CUE, an uncommon language that is hard to debug. The abstraction is real, but when it breaks, your developer ends up looking at Kubernetes YAML all the same.

The analogy: the order form against the shopping list

Deploying on raw Kubernetes is going to the wholesaler with a technical shopping list: you have to know that you want a Deployment, a Service, a HorizontalPodAutoscaler, an Ingress, a ConfigMap and a ResourceQuota, and how each one gets filled in. It works if you know the warehouse. But your data scientist, who just wants to serve their model, should not have to learn the warehouse.

KubeVela is an order form. Instead of the technical list, you tick “I want to serve this application, with these replicas, with this gateway”, and behind it somebody who does know the warehouse (your platform team) has defined exactly what gets ordered for that to work. The developer fills in the form; the translation into the technical list was done by somebody else, once, for everyone.

Score is the same form but designed so that it works for you in two different shops. You describe your order once and the same sheet serves the market next door (Docker Compose, your laptop) and the wholesaler (Kubernetes, production), without rewriting it. Whatever changes between one shop and the other is supplied by whoever translates, not by you. Both analogies point at the same thing: separating what the developer wants from how it materialises, so that they do not have to learn the warehouse.

KubeVela: the application as the unit

What it is and what state it reaches 2026 in

KubeVela is a CNCF project at incubation level since February 2023. It was born as an evolution of the OAM runtime on Kubernetes, with initial contributions from Alibaba Cloud, Microsoft and Upbound, and it rests on the Open Application Model, the specification Alibaba and Microsoft co-created to describe applications independently of the infrastructure.

Here is a realism check on its community size. The official 2025 retrospective declares seven releases in the year, 124 commits and around 33 people contributing, on a little over 7,600 GitHub stars. It is not a project in decline, but it is a concentrated community, with a growth curve that has flattened compared to its early stage under Alibaba. And an important version warning: the stable branch is 1.10, with regular patches through 2026; 1.11 is still in alpha, so do not rely on its features as if they were available.

The OAM model in five pieces

Your whole application is described in a single YAML file of type Application, with four sections you need to know by name. The components are the artefact you deploy, be it an image, a Helm chart or a service; the traits are operational requirements you hang off each component, such as scaling, the gateway or storage; the policies are global application strategy, such as multi-cluster topology or SLOs; and the workflow is the step-by-step delivery process, which can include a manual approval pause or a notification.

The key to the model is that component and trait types are not fixed: they are programmable modules maintained by your platform team, expressed as ComponentDefinition and TraitDefinition. The developer only consumes a type and its properties; they never see the Kubernetes YAML underneath. That is exactly the “do not expose raw Kubernetes” this is all about: your platform operator writes the definition once, your data scientist writes fifteen lines of Application.

CUE, the power and the toll

The definitions are written in CUE, and here you have KubeVela’s strength and weakness at the same time. CUE lets you parameterise seriously: the definition declares which inputs it exposes to the developer and which Kubernetes resources it renders from them, with access to context variables such as the application name. The real flow starts from an existing YAML, which vela def init turns into a CUE skeleton; you edit it, validate it with vela def vet and publish it with vela def apply, and it becomes available to all your developers.

The toll is that CUE is an uncommon language, with few experts and hard to debug. The cost falls on your platform team, not on the developer, which is the correct distribution, but it is a real cost you have to sign up for before adopting. Nobody arrives already knowing CUE.

The vela CLI: Heroku on the outside, GitOps on the inside

Here is the answer to the question this blog has been dragging along about CLI provisioning. Day to day with KubeVela looks like Heroku: vela up deploys the application from local files, vela status shows you its state, vela ls lists the applications, vela logs follows the logs, vela exec runs a command inside the container without you knowing the pod name, and vela port-forward opens a tunnel for you. You even have vela dry-run, which renders the Kubernetes resources without applying them, so you can see what is going to happen.

The honest qualification is that it is “imperative on the surface, declarative underneath”. When you run vela up you are not executing an imperative command like heroku scale, you are applying a declarative Application object that a controller reconciles. The CLI imitates Heroku’s comfortable flow, but the underlying model is GitOps, which means it fits with your Flux stack through the corresponding addon, instead of replacing it.

Multi-cluster, which is what matters to you

If you have a platform with several sites (site01 and site02, in my case) the multi-cluster part is the deciding one. KubeVela orchestrates from a central cluster and only the already rendered resources reach the managed clusters, with centralised governance. A topology policy decides the destination by cluster list or by label selector (by region, for example); an override policy adjusts images, replicas or traits per cluster without touching the component; and the deployment step in the workflow distributes, with the option of a phased rollout: first to the local cluster, then a manual approval gate, and only then to production. It is the kind of governance you have to build on-premise and that the cloud served you ready-made.

VelaUX, the lightweight portal thrown in

KubeVela comes with an optional web console, VelaUX, which you switch on as an addon and which gives you a dashboard, project management, environments and application lifecycle. If you want CLI first it is optional, more useful for giving operators visibility and access control than as the main route. It is a lightweight portal “thrown in” that does not aim to compete with Backstage, but it can save you from setting one up if your needs are modest.

Score: describe the workload once

What it is and what it solves

Score is an open workload specification, agnostic of platform and environment. The idea, in its own formulation, is that you define the workload once and use a Score implementation to translate it to several platforms: Docker Compose, Kubernetes, Fly.io or Cloud Run. The problem it attacks is concrete and familiar: the divergence between what you run on your laptop with Docker Compose and what gets deployed in production with Kubernetes, two descriptions that drift apart and cause you the classic “it worked on my machine”.

In a score.yaml file the developer declares their containers, the service ports and the resources they need as dependencies (a database, a DNS record, a route), and the environment-specific parameters are injected at deploy time. Score entered the CNCF sandbox in July 2024. It was created and donated by Humanitec, a developer platform company, some eighteen months before it joined the foundation.

The two implementations

Score does nothing on its own: it is a specification that needs an implementation to translate. You have two official ones. score-compose generates Docker Compose for local development, with init and generate commands, and provisions on the order of fourteen resource types, from Postgres or Redis to S3 storage or queues. score-k8s generates Kubernetes manifests ready for kubectl apply, with flags for the image, the namespace and the provisioners. Humanitec’s commercial CLI, humctl, also supports Score natively.

The most important architectural fact is that Score runs nothing in the target cluster. It is purely CLI translation: it generates manifests for you and steps aside. There is no controller, no CRD, no reconciliation. That is at once its virtue (it is lightweight, portable and without runtime lock-in) and its limit (it neither orchestrates nor deploys, it only translates).

Score against KubeVela and Helm

Score’s own official positioning clarifies the layers, and I will pin it down for you because they are easy to confuse. Score is the workload specification layer, developer-centric: it describes a workload and its dependencies and gets translated to N platforms by CLI, but it does not orchestrate or deploy. KubeVela is a complete continuous delivery platform, application-centric: it orchestrates multiple components, applies policies and workflows, does multi-cluster, and for that it needs its controller in the cluster; it is, in its own words, far richer in features than Score. Helm is the packaging and rendering of manifests, without the developer-centric abstraction or the portability to Compose.

The clean synthesis is that they are complementary layers, not substitutes: Score generates, Helm or the manifests package, KubeVela orchestrates and deploys. If they are presented to you as rivals, somebody has confused the layers.

Where each one fits: the cross-cutting comparison

With Crossplane already in hand from the previous article, the comparison of the three self-service pieces sorts itself by the question each one answers.

Score and KubeVela answer at the application level: the workload a developer deploys, “deploy this app”. Crossplane answers at the infrastructure level: the resource a developer asks for, “give me a managed Postgres”. And Backstage answers at the portal level: the web catalogue and the golden paths, the discovery layer that neither deploys nor provisions by itself, but orchestrates the ones below.

The interesting part is that they combine. A score.yaml can declare its resources and delegate the provisioning of those resources to Crossplane through a provisioner, while KubeVela orchestrates the application deployment and Backstage acts as the façade. It is not an either-or; it is a stack where each layer does its job. If your team is small, you pick a single piece and that is that; the complete stack is for mature platforms.

The fit with inference: a pattern, not an official recipe

Just as with Crossplane, honesty is called for here: there is no official, maintained KServe definition in KubeVela’s default catalogue, nor an inference resource type in Score. What follows is a design pattern with pieces confirmed separately, not a documented recipe.

The KubeVela pattern would be that your platform operator defines once, in CUE, a ComponentDefinition called for example inference-service, which underneath renders a KServe inference service with vLLM runtime, with the GPU limits, the namespace quotas and the autoscaling already wired in. Your data scientist then writes a fifteen-line Application with the inference-service type and its properties (the model, the replicas), without touching KServe, without knowing the GPU quotas or the runtime class, and operates with vela up, vela status and vela logs. Multi-site is solved for you by the topology policy, which decides which GPU cluster it lands on, and the override policy, which adjusts replicas or model per site, with a manual approval in the workflow before production.

The Score alternative would be a score.yaml with a resource of type inference-endpoint that score-k8s, with a custom provisioner, emits as a KServe inference service. More portable, but today it requires that provisioner of your own, because KServe is not a default type. In both cases, the abstraction layer rests on pieces that are already standard (KServe moved to CNCF incubation at the end of 2025 and vLLM is its reference generative runtime), but the inference abstraction is built by you. Take it as a proposal, not as a proven case.

A note for you: in 2026 KServe introduced a type dedicated to large models, LLMInferenceService, still in alpha and with vLLM as its runtime, which is the brick a KubeVela definition or a Score provisioner would render underneath. And at the infrastructure layer, not this one, the Crossplane team published Modelplane, an inference control plane made of compositions that serves vLLM directly. Neither of the two is a KubeVela or a Score type, but both confirm for you that the brick underneath is standardising, which is what makes putting a self-service façade on top viable.

Caveats, no frills

This is the section that separates the article from the brochure, and with these two projects you need it especially.

Adoption is niche, and there is a hard figure. The CNCF technology radar from early 2026, covering more than 400 developers, places Helm, Backstage and Argo CD in “Adopt”. Neither KubeVela, nor Score, nor Crossplane get a mention. They are niche tools against the mainstream, and choosing them means betting on an adoption curve that may not take off.

Score is young and its sponsor changed hands. Its Kubernetes implementation has few stars, the specification is at 0.x versions, and Humanitec, which sponsors it, was acquired by Akamai in 2025, so you have to watch the continuity of the investment in the open source part.

KubeVela has a concentrated community and v1.11 in alpha. The 33 people contributing in 2025 and the flat star curve are a sign of a plateau. Do not build on 1.11 features until it is stable.

The CUE curve is real. KubeVela’s power lives in CUE definitions, and CUE is hard. The cost falls on the platform, but it exists.

And the usual risk with abstractions: the leak. Both KubeVela and Score add an indirection over Kubernetes and KServe. When something fails underneath, the abstraction cracks and your developer ends up looking at the KServe YAML anyway, plus lock-in on your own definitions. An abstraction that does not hold up under failure hands you back exactly the complexity it promised to hide.

For your sovereign inference factory

The practical decision sorts itself by the profile of your users. If your people live in the CLI and think in applications, KubeVela gives you the Heroku-style experience on your own iron, with real multi-cluster, in exchange for your platform team learning CUE. If what hurts most is the divergence between local and production, Score solves that with the bare minimum, without putting one more runtime in the cluster. And if you want both plus a portal, the Score, Crossplane, KubeVela and Backstage stack exists, but it is for a platform that already serves many teams, not for the start.

The sobriety criterion is the same as throughout this batch: these tools plug a real gap left by the cloud you walked away from, but each one is one more system to operate. For two teams who know each other, good Helm and a tidy GitOps repository do the job without CUE or definitions of your own. Application-centric self-service starts paying you back when the number of people deploying exceeds the number of people who know Kubernetes, which is precisely the moment an inference factory stops being a project and starts being a platform.

That closes the batch. The map drew the gaps of leaving the cloud, Crossplane covered the infrastructure control plane, and KubeVela with Score cover the face the developer touches. Your own house now looks quite a lot like the hotel, and the data stayed inside.

Sources

KubeVela, State of KubeVela 2025 — https://kubevela.io/blog/2025/12/20/state-of-kubevela-2025/
CNCF, KubeVela brings software delivery control plane capabilities to the CNCF incubator — https://www.cncf.io/blog/2023/02/27/kubevela-brings-software-delivery-control-plane-capabilities-to-cncf-incubator/
KubeVela Docs, Core Concept — https://kubevela.io/docs/getting-started/core-concept/
KubeVela Docs, vela CLI reference — https://kubevela.io/docs/cli/vela/
KubeVela Docs, Custom Component with CUE — https://kubevela.io/docs/platform-engineers/components/custom-component/
KubeVela Docs, Multi-cluster delivery — https://kubevela.io/docs/case-studies/multi-cluster/
KubeVela Docs, VelaUX addon — https://kubevela.io/docs/reference/addons/velaux/
CNCF, Score accepted as a CNCF Sandbox Project — https://www.cncf.io/blog/2024/08/08/score-accepted-as-a-cncf-sandbox-project/
Score Docs, Overview — https://docs.score.dev/docs/
Score, Score vs the Open Application Model and KubeVela — https://score.dev/blog/score-vs-open-application-model-kubevela/
GitHub, score-compose — https://github.com/score-spec/score-compose
GitHub, score-k8s — https://github.com/score-spec/score-k8s
CNCF, KServe becomes a CNCF Incubating Project — https://www.cncf.io/blog/2025/11/11/kserve-becomes-a-cncf-incubating-project/
KServe Docs, Understanding LLMInferenceService — https://kserve.github.io/website/docs/model-serving/generative-inference/llmisvc/llmisvc-overview
Crossplane Blog, Building Modelplane on Crossplane — https://blog.crossplane.io/building-modelplane/ · https://modelplane.ai
CNCF y SlashData, Technology Radar Q1 2026: Platform Engineering — https://www.cncf.io/announcements/2026/03/24/cncf-and-slashdata-report-finds-platform-engineering-tools-maturing-as-organizations-prepare-for-ai-driven-infrastructure/

Crossplane: the control plane that turns your cluster into a private cloud API

Mon, 31 Aug 2026 09:30:00 +0200

Second article in the batch about rebuilding the cloud experience on-premise. The general map placed the gaps for you; this one goes into the provisioning control plane, that API which on-premise does not come for free. The third moves up a layer, to application-centric self-service with KubeVela and Score.

TL;DR

Crossplane solves the “open a ticket and wait” gap for you. In the cloud you ask for a database through an API and it appears with permissions, backups and monitoring; on-premise you do that provisioning by hand or with scripts. Crossplane brings the Kubernetes model of declarative APIs to infrastructure: your platform team defines an API of its own (a database, an inference endpoint) and the developer consumes it with a kubectl apply of a few fields, while controllers inside the cluster reconcile the desired state against reality and correct drift continuously. It graduated from the CNCF in November 2025, with more than three thousand contributors and adopters such as Nike, SAP, IBM or NASA’s science cloud.

The difference from Terraform is one of nature, not of features: Terraform applies a plan on demand and leaves; Crossplane is a control plane that lives inside the cluster and never stops reconciling. That gives you drift self-correction and native GitOps, and in exchange it takes away the safety net of plan and forces you to operate the control plane itself, with providers that create hundreds of resource definitions and state that lives in etcd. And a warning about something awkward up front: Crossplane is not an imperative aws ... style CLI, it is declarative first. On the fit with GPU inference, which until recently was only a pattern with no public case, you now have a real reference, Modelplane, from the Crossplane team itself, although at version 0.1 and composing vLLM directly without going through KServe. I go into the detail at the end.

The analogy: the thermostat against the light switch

Think of it this way: Terraform is a light switch. You flip it, the light comes on, and it stays there until you touch it again. If somebody else changes it, the light obeys whoever touched it last, and you do not find out until you go and look. It is provisioning by one-off action: you run it, it applies, it finishes.

Crossplane is a thermostat. You do not tell it “turn the heating on”, you tell it “I want twenty-one degrees”, and from then on a mechanism watches the temperature and acts on its own, continuously, to hold it. If somebody opens a window and the thermometer drops, the thermostat reacts without you intervening. That is continuous reconciliation: you declare the desired state once and a controller takes care of reality not straying from it.

The analogy also warns you about the cost. A thermostat is more convenient than a switch, but it is one more device you have to install, power and fix when it fails, and if you program it badly it adjusts the temperature to something you did not want, without asking. A switch almost never breaks and you always know what state it is in. Crossplane gives you the convenience of the thermostat in exchange for you operating the thermostat.

What Crossplane is and what state it reaches 2026 in

Crossplane was born at Upbound in 2018 and followed the CNCF maturity path without shortcuts: sandbox in 2020, incubation in 2021 and graduation on 6 November 2025. Graduation matters because it is the seal the CNCF reserves for projects it considers production-ready, the same level as Kubernetes or Prometheus. The figures that came with the announcement are solid: more than three thousand contributors from more than 450 organisations, more than a thousand pull request authors (which puts it in the top 10 % of the foundation’s projects by that metric), more than a hundred releases and two completed security audits. Among the more than seventy public adopters you have Nike, Nokia, Grafana, NASA’s science cloud, SAP and IBM.

The project’s official description is “the cloud native control plane”, and that phrase sums up exactly what it does. The main vendor behind it is still Upbound, which at graduation framed the project’s future towards “AI-native” infrastructure, a point I come back to when talking about inference, with the caveat that it is a vendor’s positioning, not a proven case.

The release cadence is quarterly, with a nine-month support window and three actively maintained versions at any time. As of writing, the v2 branch is the current one, with v1.20 as the last close of the v1 branch.

The v2 turn (which changes how you think about it)

Crossplane 2.0 reached general availability in August 2025 and brings four structural changes you need to know about, because a good deal of the old material describes a model that is no longer the recommended one.

The first is that resources are now namespaced by default. Both composite and managed resources, which used to be cluster-scoped, now live in a namespace, which enables fine-grained access control and natural multi-tenancy, exactly what you need in a sovereign platform with several teams.

The second is that a composition can include any Kubernetes resource, not only Crossplane resources. Previously a composition was limited to Crossplane’s own managed resources; now it can include Deployments, Services or third-party CRDs, so you can compose abstractions mixing application and infrastructure in a single object. This change is what makes the inference pattern you will see viable.

The third is that Claims disappear. In the v1 model there was a duality between the composite resource, cluster-scoped, and the Claim, its namespaced face which the developer consumed. Now that the composite resource is itself namespaced, that duality is redundant: the developer consumes the composite resource with no intermediary. If you come across documentation talking about Claims, it belongs to the earlier model.

The fourth is a new type, Operations, for one-off or scheduled operational tasks that run a pipeline of functions to completion, like a Kubernetes Job but inside the Crossplane model.

There is one removal that deserves its own paragraph, because it is the most surprising: in v2 native patch-and-transform was withdrawn, the classic patching mechanism of compositions. It was not relegated, it was removed. Everything now goes through composition functions, and I get to that in a moment. Migration from v1 to v2 is mostly non-breaking and the v1 APIs still work, so adopting v2 is optional, but the new model is the one you have to learn.

The model, piece by piece

Crossplane has dense jargon and I will take it apart calmly, because once the pieces are placed the whole thing is coherent.

A provider is a package that installs controllers and resource definitions to talk to an external API, be it AWS, GCP, a remote Kubernetes cluster, Helm or Terraform. It is installed declaratively, by applying a Provider object. The providers for the big cloud providers are generated with Upjet, a framework that produces the code from the Terraform providers, which explains why there are so many of them and also why some are enormous.

A managed resource is the one-to-one representation, inside Kubernetes, of a resource from the external provider: a bucket, an instance, a database. It is the unit the controller reconciles. When you create a managed resource of type bucket, the provider’s controller creates the real bucket and then watches that it keeps existing with the configuration you declared.

On those bricks you build the abstraction, which is where the real value is, with three concepts:

The CompositeResourceDefinition, or XRD, defines the schema of your own API: which fields the user can ask for. It is what turns “a database” into a resource type with its parameters.

The Composition is the template and the logic that, given one of those resources, materialises the N managed resources needed underneath. The documentation says it bluntly: a composition is a pipeline of composition functions.

The Composite Resource, or XR, is the concrete instance: the object the developer creates with kubectl apply, with the few fields the XRD exposes.

The complete pattern is this: your platform operator publishes the abstraction (the XRD plus the Composition, which say “this is what a database means here”), and the developer only applies a composite resource with four fields. How it is built underneath stops mattering to them, just as in the cloud you do not care how AWS assembles your managed database.

Composition functions: the real logic

The composition mechanism in v2, and the only one, is composition functions. A composition is a pipeline of functions that Crossplane invokes over gRPC; each function receives the observed state, the desired state, its input and a shared context, and chains its output to the next one. The YAML in the examples is illustrative, but underneath it is not processed as YAML: it is executed as functions.

The available functions cover several styles. You have function-patch-and-transform, which reimplements the old model now as just another function. There is function-go-templating for Helm-style templates, function-kcl for the KCL language, function-python for logic in Python and function-cue for CUE.

The reason for the change is what you want to understand. Classic patch-and-transform had no loops or conditionals, so an operation as simple as transforming an array of strings turned into dozens of lines of patching, and the community went as far as calling it outright bad. Functions give you full programming logic (loops, conditionals, external calls) in a real language. The honest flip side is that the complexity does not disappear: it shifts from “writing verbose patches” to “writing and operating functions”, which is a different kind of work, not less work.

Crossplane against Terraform, OpenTofu and Pulumi

This is the comparison everybody makes, and I will make it properly because the difference is one of category. According to Pulumi’s own documentation, which compares honestly, Crossplane is a declarative control plane with continuous reconciliation inside Kubernetes, while Terraform and Pulumi are infrastructure as code executed one-off, client-side and on demand.

The practical consequences line up along a few axes. Faced with configuration drift, Crossplane corrects it continuously and automatically, while Terraform and Pulumi only detect it when you run a refresh. Crossplane needs a cluster to run on, because it is controllers; Terraform and Pulumi do not. Crossplane is GitOps-native, applied with kubectl and fitting with Flux without an adapter; Pulumi needs its Kubernetes operator for the same thing. And as for language, Crossplane uses YAML and CRDs with logic in the functions, while Pulumi uses general-purpose programming languages.

The point most often forgotten is that you do not have to choose one or the other. There is a Terraform provider (and an OpenTofu one, recommended for new versions because the Terraform one was frozen at the last freely licensed version) that runs HCL inside Crossplane, wrapping your existing modules as a managed resource. You can reuse your whole investment in Terraform and still gain continuous reconciliation for whatever needs it. If you already have Terraform in production, this hybrid pattern is usually your sensible route, not the rewrite.

On comparative adoption you have to be sober: there is no official, reliable “Crossplane against Terraform” figure, and the most repeated documented criticism is that Crossplane lacks the mature module ecosystem Terraform has been accumulating for years. That is a real qualitative difference, not a detail.

The awkward question: is it an AWS-style CLI?

Since on this blog I have been pulling on the thread of CLI provisioning, I will answer directly: no. Crossplane does not offer you an imperative aws s3 create-bucket style experience. It is declarative first. You provision with a kubectl apply of a resource, and the crossplane binary is there to build, package, test and debug, not to “create a bucket” in one go.

The subcommands make that clear. crossplane render renders a composition and its functions locally, so you can test without a cluster. The crossplane xpkg build, push, install family manages the OCI packages of providers, configurations and functions. And crossplane beta validate, trace and top validate schemas, draw the resource tree and show consumption inside the cluster. It is the toolbox of whoever builds the platform, not of whoever consumes it day to day. Upbound’s up CLI adds more, but oriented towards its commercial product.

The conclusion matters for setting expectations: the imperative AWS-console-style experience, if you want it, gets built on top of Crossplane, with Backstage, a portal or your own wrapper. Crossplane gives you the declarative API; the friendly face you put on separately. If what you are after is an imperative provisioning CLI, you fit better with the model of KubeVela and Score, which is precisely the subject of the next article.

On-premise providers: provisioning without touching AWS

For a sovereign platform your question is what you can provision without setting foot in a public cloud, and the answer is more than it seems, with uneven maturity.

The most versatile is provider-kubernetes, which manages any Kubernetes object in remote clusters through an Object resource. It is the basis of the multi-cluster on-premise pattern: from a central cluster you materialise resources in your secondary RKE2s. Alongside it, provider-helm installs and manages Helm charts declaratively through a Release resource, letting you deploy things like vLLM or KServe as part of a composition.

For the identity and storage part you have concrete pieces: provider-keycloak configures Keycloak realms, clients and roles declaratively, useful for the sovereign IAM we already set up in the article on authentication with Keycloak; and provider-minio manages buckets and policies of on-premise S3-compatible storage. For vSphere you do not have a first-class native provider, but you cover it by generating one from the Terraform provider, the usual pattern for VMware infrastructure.

An honest warning: for KubeVirt, Harvester or CloudNativePG there are no mature, first-class native providers; in practice you manage them by applying their CRDs through provider-kubernetes or by wrapping their Terraform. It works, but it is not the polished experience of a dedicated provider. With the combination of provider-kubernetes, provider-helm, provider-minio and provider-keycloak, plus provider-terraform for whatever is missing, you can assemble a one hundred per cent on-premise self-service catalogue, with the caveat that maturity varies piece by piece.

The fit with GPU inference: from pattern to Modelplane

This is the section that in the first version of this text carried the most caution, because there was no public case of Crossplane orchestrating GPU and inference and it all remained a design pattern. Halfway through 2026 that changed, and the case was published by the very team that makes Crossplane.

Modelplane is an open source control plane for AI inference, Apache-licensed and with a first version 0.1 from June 2026, and the interesting thing is how it is built: entirely on Crossplane compositions and composition functions in Python, with no controllers of its own. It composes vLLM directly, with the vllm/vllm-openai image, without going through KServe, and its self-service API for the ML teams is a ModelDeployment resource where you declare the model, the engine, the topology and the hardware. Underneath, a ModelReplica resource represents each replica, and the platform team declares its GPU pools with InferenceCluster and InferenceClass resources, using an attribute model and CEL expressions taken from Kubernetes dynamic resource allocation. Its most advanced piece is a two-level fleet scheduler written as a pure function, which places replicas across the whole infrastructure and delegates to each cluster’s scheduler. It is proof that the pattern works, with the warning that it is a version 0.1 under open development, not something to put into production blindly, and that adopting it ties your platform to its resource types.

The other recent enabler is in KServe, which introduced a type dedicated to large models, LLMInferenceService, still in alpha and built on the llm-d project, with prefill and decode separation and multi-node execution, and which runs vLLM as its runtime. It is different from the classic InferenceService, reserved for predictive ML. That new type is your ideal brick if you prefer composing KServe instead of bare vLLM.

With that, your own golden path stops being a hypothesis. You define an XRD, for example InferenceEndpoint, with fields such as the model, the number and type of GPU and the replicas. Its composition, taking advantage of the fact that in v2 it can include any Kubernetes resource, materialises in a single pipeline the team’s namespace, the quota for nvidia.com/gpu and a KServe LLMInferenceService pointing at the model with vLLM runtime, and the developer only applies the InferenceEndpoint with four fields, knowing nothing about KServe or the quotas underneath. KServe supports vLLM and GPU natively, as we saw in the article on KServe.

The honest reading is that you are at the frontier, not on well-trodden ground. Outside Modelplane, made by Upbound, there is still no body of independent cases assembling Crossplane with vLLM, and the KServe type is in alpha. It is a coherent architecture, with a real reference backing it and plenty of room for you to be among the first to document it on your own iron.

What it costs you to operate your own control plane

No tool that gives you back control is free, and Crossplane has well-documented costs you need to budget for beforehand, not discover afterwards.

The learning curve is steep, acknowledged even in the coverage of its graduation. Adopters report debugging difficulties when a composition or a provider misbehaves, and they miss Terraform’s module ecosystem.

Compositions tend towards complexity. There are practitioners describing definitions spanning thousands of lines with poor navigation, and the absence of an integrated testing framework. Functions improve this, but as I said, they shift the complexity rather than remove it.

There is no plan. Unlike Terraform’s plan and apply cycle, Crossplane applies changes immediately. You lose the safety net of reviewing what is going to happen before it happens, with the risk of modifications you did not want.

Operating the control plane itself costs. Big providers create hundreds of resource definitions (the GCP one reached 347), with a risk of API server unavailability and of leaving your cluster hanging. State lives in etcd, which complicates disaster recovery and migrating control between clusters. And there is one specific danger: accidental deletion of cloud resources when upgrading XRDs, providers or compositions, because of the ownership Crossplane keeps over what it created.

Some have decided not to adopt it. The Masterpoint team ruled it out for the lack of data sources and ignore_changes, for incomplete coverage of AWS services, and for the cost of abandoning their accumulated Terraform experience. It is a legitimate decision, and quoting it to you is part of an honest article.

How it lives alongside what you already have

If you already run a platform, your question is not Crossplane yes or no, but Crossplane with what. Three relationships matter to you.

With Backstage they are complementary, not rivals. Backstage is the portal and the templates that generate manifests or pull requests, day zero; Crossplane provisions and reconciles the real state, day two. The usual pattern, which some call the golden triangle, is Backstage as the interface, Crossplane as the control plane and Argo CD or Flux as GitOps. Each one does a layer.

With Flux, which you probably already use, the relationship is additive. Flux reconciles manifests towards the clusters, but it does not give you a platform abstraction API: it does not know what “a database” is, it only applies the YAML you hand it. Crossplane adds exactly that abstraction layer, and the two coexist, with Flux deploying the composite resources and Crossplane materialising them. If you already have Flux, Crossplane’s value is precisely that self-service API layer, in exchange for the cost of operating the control plane I described above.

With KubeVela and Score, the subject of the next article, the boundary is one of layer: Crossplane answers “give me a Postgres”, from the infrastructure side; KubeVela and Score answer “deploy my application”, from the workload side. They do not compete, they stack.

For your sovereign inference factory

The practical decision boils down to this. Crossplane makes sense when you want to expose a self-service API of your own over your on-premise infrastructure and you are willing to operate a control plane to get it. If your platform is used by two teams who know each other and you provision with a tidy Flux repository, Crossplane is cost without return for now. The trigger is scale: when manual provisioning turns into a ticket bottleneck, the declarative API starts paying back its cost.

Your sensible entry route is not rewriting everything. It is starting with a provider that already serves you (provider-kubernetes and provider-helm cover a lot), wrapping your existing Terraform with the corresponding provider instead of throwing it away, and modelling a single abstraction of clear value, probably the InferenceEndpoint we discussed, as a golden path. From there, you grow on demand. And always with a clear account of having traded the safety net of plan and a handful of scripts for a thermostat you have to keep switched on.

The next article moves up a layer, from the infrastructure resource to the application workload, with KubeVela and Score, which is where the friendly CLI that Crossplane deliberately does not give you finally appears.

Sources

CNCF, Announces Graduation of Crossplane (6-nov-2025) — https://www.cncf.io/announcements/2025/11/06/cloud-native-computing-foundation-announces-graduation-of-crossplane/
Crossplane Blog, Announcing Crossplane’s CNCF Graduation — https://blog.crossplane.io/crossplane-cncf-graduation/
Crossplane Blog, Announcing Crossplane 2.0 — https://blog.crossplane.io/announcing-crossplane-2-0/
Crossplane Docs, What’s New in v2 — https://docs.crossplane.io/latest/whats-new/
Crossplane Docs, Compositions — https://docs.crossplane.io/latest/composition/compositions/
Crossplane Docs, Release Cycle — https://docs.crossplane.io/latest/learn/release-cycle/
Crossplane Docs, Crossplane CLI — https://docs.crossplane.io/latest/cli/
Upbound, Crossplane Graduates From CNCF, Upbound Redefines AI-Native Infrastructure — https://www.upbound.io/blog/crossplane-graduates-from-cncf-upbound-redefines-ai-native-infrastructure
InfoQ, Crossplane Reaches Production Maturity by Graduating CNCF — https://www.infoq.com/news/2025/11/crossplane-grad/
Pulumi Docs, Pulumi vs. Crossplane — https://www.pulumi.com/docs/iac/comparisons/crossplane/
Upbound Marketplace, provider-terraform — https://marketplace.upbound.io/providers/upbound/provider-terraform/latest
GitHub, provider-kubernetes — https://github.com/crossplane-contrib/provider-kubernetes
GitHub, provider-helm — https://github.com/crossplane-contrib/provider-helm
GitHub, provider-keycloak — https://github.com/crossplane-contrib/provider-keycloak
VSHN, provider-minio — https://github.com/vshn/provider-minio
Masterpoint, Crossplane: Why it Didn’t Work for Us — https://masterpoint.io/blog/passing-on-crossplane/
CECG, Crossplane: the good, the bad and the ugly — https://www.cecg.io/blog/crossplane-the-good-the-bad-the-ugly
Taloflow, Backstage vs Crossplane for Platform Engineering — https://www.taloflow.ai/guides/comparisons/backstage-vs-crossplane-platform-engineering
Crossplane Blog, Building Modelplane on Crossplane — https://blog.crossplane.io/building-modelplane/
Modelplane — https://modelplane.ai · https://github.com/modelplaneai/modelplane
KServe Docs, Understanding LLMInferenceService — https://kserve.github.io/website/docs/model-serving/generative-inference/llmisvc/llmisvc-overview

Kubeflow in depth: which pieces are worth it in an on-premise LLM platform and which ones you already have

Mon, 31 Aug 2026 09:30:00 +0200

Second post in the platform and self-service run, after Backstage and before Knative. The question here is not what Kubeflow is, but which parts of Kubeflow still make sense once you have built half a platform on your own.

TL;DR

Kubeflow graduated in the CNCF on 17 August 2026, and that headline hides the important one: it is no longer a monolithic platform, it is an umbrella of independent subprojects, each with its own repository, its own version and its own life. That changes how you have to evaluate it entirely. The 2020 question, should I install Kubeflow?, has become the 2026 one, which piece of Kubeflow do I install, and at what cost?

For a platform that already has RKE2, vLLM, KServe, Flux and Volcano or Kueue, the honest answer is that Kubeflow as a whole duplicates almost everything you already have. KServe no longer lives under Kubeflow: it left, and since November 2025 it has been a CNCF project in its own right, so anyone serving models with KServe does not need Kubeflow to serve. The Trainer leans on Volcano and Kueue, it does not replace them. And the full manifest drags in Istio, Dex and cert-manager as hard dependencies that clash with what you already operate. The only thing that may be worth installing on its own is Pipelines, as an orchestrator for data and training flows, and perhaps the Trainer for its fine-tuning templates. The rest you either already have, or do not need.

The analogy: the hypermarket and the specialist shop

Kubeflow was born as a hypermarket: a single building with everything an ML team might need, from notebooks to the inference server, taking in pipelines and hyperparameter tuning along the way. It made sense in 2019, when assembling each piece separately was a project in itself and there were no standards.

By 2026 the hypermarket has been converted into a street of specialist shops. The butcher’s counter, KServe, went independent, opened its own premises and is doing better alone. The fishmonger’s, the Trainer, now depends on the central market next door, Volcano and Kueue, instead of having its own cold store. And the shared building still demands that you put in its wiring, its plumbing and its security system (Istio, cert-manager, Dex) even though you already have your own.

Anyone arriving today with a half-full fridge, which is the case for anyone with a platform already running, does not need the whole hypermarket. They need to know which shop to walk into for what they are missing, and to avoid buying twice what they already have at home.

What Kubeflow is in 2026

Kubeflow was created by Google in 2017 and graduated in the CNCF on 17 August 2026, reaching the top level alongside Kubernetes or Prometheus. The graduation figures are serious: more than 6,600 contributors from over a thousand organisations, and adopters such as Bloomberg, NVIDIA, Red Hat or Spotify.

The structural change is the one that matters for the decision. The full distribution, the Kubeflow Community Distribution, is numbered by year and month (the current one is 26.03, from March 2026, on a six-monthly cadence), but each component is by now a subproject with its own repository and its own cycle, usable independently. The distribution only packages them together. That means “installing Kubeflow” is no longer an atomic decision: it is a shopping list.

Component by component

Kubeflow Pipelines: the piece you may actually want

Pipelines orchestrates containerised ML workflows. They are written in Python with its SDK, which compiles to an intermediate YAML representation so they are portable, and underneath they run on Argo Workflows. It is the piece most likely to add value to an existing platform: a flow of “ingest, embeddings, fine-tuning, evaluation, deployment” is exactly what you need to operate RAG and adapters repeatably.

The small print has two parts. The first is that if you already use plain Argo Workflows, Pipelines adds a layer of SDK, interface and metadata on top of the same engine: it does not bring a new scheduler, it brings the typed component model and the runs interface. The second is that its metadata and lineage system, MLMD, is being removed from Pipelines itself, so it is not wise to build the platform’s traceability on that particular piece.

Kubeflow Trainer: it leans on your scheduler, it does not replace it

The old Training Operator, the one with PyTorchJob and TFJob, has been rewritten as Kubeflow Trainer, with a single TrainJob API that unifies all frameworks. Version 2.2 positions itself explicitly for distributed training and LLM fine-tuning, with support for PyTorch, DeepSpeed, HuggingFace and company.

A common expectation needs correcting here. The Trainer does not replace Volcano or Kueue: it leans on them. The TrainJob carries a podGroupPolicy field that automatically creates the Volcano PodGroup objects for gang scheduling, and it integrates with Kueue’s queues. If you already built that scheduler, as we saw in its post, the Trainer adds the TrainJob abstraction and some fine-tuning templates on top of what you already have, not a new scheduling layer. It may be worth it for the templates; not for the scheduler.

Katib: only half relevant

Katib does Kubernetes-native AutoML: hyperparameter optimisation with algorithms such as Bayesian optimisation, TPE or Hyperband, plus architecture search. In the LLM era it has repositioned itself, with official documentation for tuning fine-tuning hyperparameters and even for tuning RAG pipelines.

The practical reading is lukewarm. Architecture search is irrelevant for anyone doing LoRA and RAG. Hyperparameter optimisation can be useful for learning-rate or LoRA-rank sweeps, but it competes with doing it from the SDK of the QLoRA runbook itself or with Ray Tune, and it rarely justifies dragging in Katib for that alone.

KServe: the piece that is no longer in Kubeflow

This is the fact that settles half the evaluation. KServe left Kubeflow. It was born inside the project in 2019, was donated to the Linux Foundation in 2022, was renamed from KFServing to KServe and graduated from Kubeflow that same year, and since November 2025 it has been a CNCF incubating project in its own right. In the 2026 taxonomy it appears as an ecosystem project, external, not as a core component.

The consequence is direct: anyone already serving with KServe does not need Kubeflow to serve. And KServe is exactly where the LLM action is: its 0.15 release strengthened the vLLM backend and introduced an LLMInferenceService resource with disaggregated serving, prefix caching, per-variant autoscaling and OpenAI-compatible APIs. If your serving layer is vLLM on KServe, you already have that layer complete and Kubeflow adds nothing to it.

Model Registry, Notebooks and the rest

The Model Registry, now reorganised under the name Kubeflow Hub, is an index of models, versions and metadata. It stores pointers and states, not the bytes. Against MLflow, it lacks the experiment and metric tracking that is MLflow’s strength, to the point that the guides recommend combining them rather than substituting one for the other; and against an OCI registry, it does not package the binary, which still lives wherever you put it. It is still a young component, below version 1.0.

The Notebooks live in two versions, the stable 1 and a 2 redesigned on CRDs that is still in alpha. The Spark Operator remains in the core; Feast, by contrast, has left the core and is now an ecosystem project, a nuance worth being clear about before counting on it.

Multi-tenancy: powerful and tied to Istio

Isolation between teams is done with Profiles, a CRD that wraps a namespace and gives it RoleBindings, ServiceAccounts and Istio authorisation policies that validate an identity header derived from OIDC. It is a complete model, but it has an Achilles heel: it depends entirely on the Istio sidecar and on that header. If traffic bypasses the mesh or someone forges the header, isolation breaks, and all multi-tenant security ends up coupled to Istio, which clashes head-on if you already operate another mesh or no service mesh at all.

The cost of installing the whole thing

Here is the main reason not to install the full distribution on top of an existing platform. The 26.03 manifest drags in as hard dependencies: Istio, cert-manager, Dex for OIDC, OAuth2-Proxy, and Knative Serving and Eventing for KServe. It recommends 16 GB of RAM and 8 vCPU as a minimum, and the aggregate of all components comes to around 4.4 CPU cores and 12 GB of memory for the control plane alone.

The project itself acknowledges the problem. There is an open thread in its repository, bluntly titled “community feedback on Kubeflow’s complexity”, where it is admitted that Kubeflow practically forces a dedicated cluster because of the assumptions it makes about what is installed, that Istio and Dex should be swappable and not fixed dependencies, and that a lighter Helm that installs only what is needed is missing. For anyone already operating cert-manager, an ingress and GitOps with Flux, the full manifest does not coexist: it duplicates and tramples the base infrastructure.

Kubeflow against the alternatives

The field is wide and almost all the comparative literature is written by vendors with a product of their own, so it has to be read with tweezers. With that caution:

Plain Argo Workflows is the engine Pipelines uses underneath; if all you need is container graphs, Pipelines is overhead. MLflow is strong in experiment tracking and lightweight registry, installs in minutes, and does not orchestrate distributed training: it is complementary, and in fact we use it in the post on prompt versioning. Flyte is a typed orchestrator that many perceive as lighter to operate than full Kubeflow. And Ray on Kubernetes is the most serious competitor for LLM workloads, because with a single runtime it covers training, serving and tuning, overlapping at once with the Trainer, Katib and KServe.

Where Kubeflow wins is in breadth and in governance: a CNCF-graduated umbrella with adoption by large companies. Where it loses is in operational weight, in hard dependencies, and in the fact that its best pieces are already independent projects that do not require the rest.

The real fit with LLM workloads

The project has moved towards GenAI, and that has to be acknowledged: there is an SDK with LLM fine-tuning templates, the Trainer adds support for reinforcement post-training methods, and new agent-oriented pieces have appeared. It is not empty marketing, there is real work there.

But the question for a specific platform is not whether Kubeflow does LLM: it is which part I have not already duplicated. On a stack of RKE2, vLLM, KServe, Flux and Volcano or Kueue, the balance looks like this:

Component	Does it add anything new to your stack?
KServe	No: you already have it, and it is no longer Kubeflow’s
Trainer	Only the fine-tuning templates; you already have the scheduler
Katib	Marginal: LoRA HP sweeps, little more
Pipelines	Possibly yes, as a flow orchestrator, but it runs on Argo
Model Registry	Optional: competes with MLflow and with your OCI registry
Notebooks	Depends on what you use for notebooks today
Multi-tenancy	No, if you already isolate by namespace without tying everything to Istio

Operational traps and honest scepticism

Do not install the full manifest on a platform that already works. You are going to duplicate Istio, cert-manager and the rest, and fight a dependency war with no prize at the end. If something in Kubeflow interests you, install it as a standalone subproject.

KServe is not Kubeflow. It is the most common category error of 2026. If someone justifies building all of Kubeflow “to serve models”, you already know they have not checked that KServe left long ago and lives on its own.

The Trainer does not save you the scheduler. It needs Volcano or Kueue underneath. If you expected it to solve GPU gang scheduling by itself, it does not.

Multi-tenancy ties you to Istio. Before adopting Profiles, ask yourself whether you want security between teams to depend on one specific mesh. If you already isolate well by namespace and RBAC, you may not need that layer.

MLMD is on its way out. Do not build your pipelines’ traceability on the metadata system the project itself is retiring.

For an inference factory

The conclusion is comfortable to apply. Kubeflow in 2026 is not a yes-or-no decision, it is a menu you order from à la carte. On a platform that already serves with KServe, schedules with Volcano and deploys with Flux, the menu shrinks a great deal.

If you need to orchestrate data and training flows repeatably, and you do not want to build plain Argo Workflows with its ergonomics, Pipelines is the piece worth evaluating, installed standalone. If you do distributed fine-tuning frequently and want a uniform abstraction on top of your scheduler, the Trainer may save you some templates, knowing that it leans on the Volcano you already have. Everything else you either have, or a specialist piece covers better.

The expensive mistake would be installing the whole hypermarket to buy bread. Kubeflow stopped being that; treat it as what it is, a street of shops, and walk only into the one you are missing. The next piece in this run, Knative, is precisely one of those specialist shops, the one that decides whether switching off the GPU when nobody is using it pays off or costs you dear.

Sources

CNCF, CNCF announces Kubeflow’s graduation — https://www.cncf.io/announcements/2026/08/17/cncf-announces-kubeflows-graduation-solidifying-the-standard-for-cloud-native-ai-operations/
CNCF, Kubeflow unveils new cloud native innovations to supercharge AI — https://www.cncf.io/blog/2026/07/28/kubeflow-unveils-new-cloud-native-innovations-to-supercharge-ai/
CNCF, KServe becomes a CNCF incubating project — https://www.cncf.io/blog/2025/11/11/kserve-becomes-a-cncf-incubating-project/
Kubeflow Docs, Introduction / components — https://www.kubeflow.org/docs/started/introduction/
Kubeflow Docs, Pipelines overview — https://www.kubeflow.org/docs/components/pipelines/overview/
Kubeflow Trainer Docs, Volcano gang scheduling and Kueue — https://trainer.kubeflow.org/en/latest/operator-guides/job-scheduling/volcano.html
Kubeflow Docs, Migrating to Kubeflow Trainer v2 — https://www.kubeflow.org/docs/components/trainer/operator-guides/migration/
Kubeflow Docs, Katib overview — https://www.kubeflow.org/docs/components/katib/overview/
Kubeflow Docs, Hyperparameter optimization for LLM fine-tuning — https://www.kubeflow.org/docs/components/katib/user-guides/llm-hp-optimization/
Kubeflow Docs, Model Registry overview — https://www.kubeflow.org/docs/components/model-registry/overview/
Kubeflow Docs, Multi-tenancy design — https://www.kubeflow.org/docs/concepts/multi-tenancy/design/
GitHub, kubeflow/manifests (dependencias y recursos) — https://github.com/kubeflow/manifests
GitHub, kubeflow/manifests #2451: community feedback on complexity — https://github.com/kubeflow/manifests/issues/2451
InfoQ, Kubeflow expands AI capabilities as CNCF graduation nears — https://www.infoq.com/news/2026/08/kubeflow/

From public cloud to private: what you really lose and which projects fill the gap

Mon, 31 Aug 2026 09:15:00 +0200

With this one I open a run about the biggest gap a sovereign platform leaves you: how to give your teams the self-service experience of a public cloud, without the public cloud. Here you have the map of the problem; the next two go down into the detail of the tools, Crossplane as the infrastructure control plane and KubeVela with Score as application-centric self-service.

TL;DR

Leaving the public cloud has a solid business case, and I am not making it up. 37signals declared savings of close to two million dollars in 2024 after repatriating, the classic a16z analysis puts cloud spend at around half of the cost of revenue of a software company, and the Barclays CIO survey of 2024 recorded the highest repatriation intent in its history. If your workload is a predictable, sustained base (which is exactly the continuous LLM inference this blog is about) the numbers work out, and in Europe the sovereignty reasons pile on top: the Data Act, the tension with the US CLOUD Act, and that dependence on three hyperscalers that control around 70 % of the European cloud market.

But the problem is not the compute. Your on-premise GPU rack serves tokens just as well as a rented one. What the cloud was selling you and you now have to rebuild is something else: the elasticity of asking for a hundred machines at ten and handing them back at eleven, the catalogue of hundreds of managed services with an on-call team behind them, the unified provisioning API with fine-grained permissions, the multi-region network, the bill that arrives itemised, and that model where operating all of it was somebody else’s problem. All of that you rebuild piece by piece with open source projects, and the rebuild has a cost that the savings figures do not usually tell you about. What follows is the map of what you lose, which project fills each gap, and where a gap remains that nothing will close completely.

The analogy: leaving the hotel to set up your own house

Think of it like this: living in the public cloud is living in a hotel. You pay a lot per night, but you do not change light bulbs, reception is open at three in the morning, if you need another room you ask for it and it appears, and when you leave you take nothing with you to maintain. Setting up your own house works out much cheaper per square metre if you are going to stay for years, and on top of that you decide who comes in and where your things are kept. The problem arrives the day a pipe bursts: then you do not call reception, because reception is you.

Repatriation is that move. The savings are real and so is data ownership, but what in the hotel came included in the price of the night is now your to-do list: the plumbing is your network, the boiler is your storage, the concierge who got you anything is the catalogue of managed services you no longer have, and the itemised bill they slid under your door is a cost model you have to build, because electricity and depreciation do not arrive broken down per tenant. The house pays off, but only if you count what it costs to furnish it and fix the pipes, not just the price per square metre against the hotel.

The rest of the article walks through that to-do list and tells you, for each item, which open source tool does the job the hotel used to do for you.

Why this conversation belongs to 2026, not to always

Repatriating is nothing new, but in 2026 three things come together that make it urgent if you run a European AI platform.

The first is economic, and Andreessen Horowitz put it on the table in 2021 with The Cost of Cloud, a Trillion Dollar Paradox. Their thesis, which you should read knowing that a16z has interests in the ecosystem it analyses, is that cloud spend averages around 50 % of the cost of revenue in a software company, and that recovering half of that spend would free up margins the market penalises. The cleanest case, and one you can verify, is 37signals: DHH published that their cloud bill dropped from 3.2 to 1.3 million dollars a year after repatriating, with an initial investment of about 700,000 dollars in Dell servers. Now the honest caveat, which the specialist press itself underlines: those savings figures did not include the extra operations staff, nor the energy, nor the cooling. I come back to it in the hidden costs section.

The second is declared intent. The Barclays CIO survey from the first half of 2024 recorded that 83 % planned to repatriate some workload to private cloud or on-premise in the following twelve months, the highest reading in the series. But read that 83 % precisely: it is the proportion of managers who plan to move some workload, not the percentage of workloads or of spend that moves. Flexera, in its 2025 report, gives you the more grounded figure: large companies have repatriated around 21 % of the workloads they had in the public cloud, and cost saving is priority number one for almost 60 %.

The third is regulatory and European, and it is the one that turns repatriation into something more than a FinOps decision. The European Union Data Act came into force in January 2024 and has applied since September 2025; it mandates portability between providers and, from January 2027, bans the exit fees that today chain you to your provider. And underneath it beats a deeper legal tension: the US CLOUD Act allows United States authorities to demand data from an American provider even if it is stored in Frankfurt, which is exactly the argument for why a European region of a hyperscaler does not give you sovereignty. With AWS, Microsoft and Google controlling around 70 % of the European cloud market, the conversation about sovereign cloud stopped being theoretical. I developed this already in sovereign on-premise versus hyperscalers.

What you lose when you leave the cloud, specifically

Let us get to the core, ordered by what is going to hurt you most. And one thing before starting: none of these gaps is about compute. They are all about what the cloud put around the compute.

Hardware elasticity disappears. On-premise, the GPU you did not buy does not exist. The cloud bills you by the hour precisely because it takes on the risk of idle capacity: it lets you ask for a peak and hand it back. With your own iron, you buy the peak up front and pay for it whether it is in use or not, so your profitability depends on high sustained utilisation. Below a certain usage threshold, the cloud beats you again. This is the irreducible gap, the one no tool fills, and if you repatriate a workload with extreme peaks you are making the wrong decision.

Managed services become your operations. Every box you took for granted (the database, the queue, the object storage, the load balancer, DNS, the secrets manager) stops being a checkbox you tick and becomes a system you install, update and watch yourself. The 37signals case teaches it well: getting out of S3 object storage was the last and hardest part, with petabytes of data and a contract they dragged along for years. Your managed Postgres turns into operating CloudNativePG; your S3 turns into operating MinIO; and so on with every service.

You have to build the provisioning control plane yourself. AWS gives you a unified API where you ask for a resource with fine-grained permissions and it appears. On-premise that does not come for free: you have to build the control plane that translates “I want a database” into real resources. It is exactly the gap that Crossplane fills, and that is why it has its own article.

You replace the managed network piece by piece. The VPC, the security groups, NAT, the load balancer: every network abstraction in the cloud has its on-premise equivalent (Cilium for network policies on eBPF, MetalLB for the load balancer without ELB, Gateway API for L7 ingress), but they are pieces you assemble, not a service you switch on. A good part of this we already saw in the networking vertical with Cilium and eBPF.

Billing stops existing as such. In the cloud the cost reaches you broken down by service and by tag; you have a Cost Explorer that answers “how much does this team cost me”. On-premise there is no natural bill: there is hardware depreciation, an electricity bill and some payroll, and splitting that per workload is a model you have to build. OpenCost helps you with the split per workload inside the cluster, but the total physical cost (depreciation plus energy plus staff) you model separately, as we saw in OpenCost and cost allocation.

You design multi-zone availability yourself. The three availability zones the cloud gives you for free turn into several sites, several racks and several utility feeds that you have to plan. 37signals solved it with two datacenters and replication between them. It is neither free nor automatic.

And above all, the on-call is yours. The cloud was not selling you only compute: it was selling you operations included. When something falls over at three in the morning, in the cloud there is an on-call team at the provider; with open source, that team is you. This is the cost that savings spreadsheets almost never include, and it is the one that decides whether your repatriation was a good idea or a hidden bill.

The map of projects that fill each gap

With the gaps ordered, the catalogue of substitutes reads itself. I group it by layer, from the lowest to the closest to the developer.

In the infrastructure substrate, what replicates EC2 virtual machines, virtual networking and disks, you have OpenStack as the classic standard for a complete private cloud, SUSE’s Harvester as bare-metal hyperconvergence on KubeVirt managed from Kubernetes, and emerging projects such as Spinifex that rebuild an AWS-compatible API on your own iron. It is the heaviest layer and the one I touch least in this blog, because I assume your cluster already exists.

In the provisioning control plane, what replicates the AWS IAM and resource API, there is Crossplane, graduated in the CNCF in November 2025, which brings the Kubernetes declarative API model to infrastructure. Alongside it, Cluster API for declarative cluster provisioning. And so you can see how far this reaches in our own field, the Crossplane team itself published Modelplane in mid-2026, an inference control plane built on compositions that serves models with vLLM. This layer is developed by the Crossplane article.

In the developer self-service experience, what replicates the AWS console and CLI, you have three different approaches you should not confuse. Backstage is the web portal, the catalogue and the templates, and it already has its own article. KubeVela brings self-service to the application model with a friendly CLI. Score defines a workload specification that is portable between environments. KubeVela and Score share the third article because they cover the same layer from angles that complement each other.

In on-premise managed services, CloudNativePG operates Postgres for you with high availability and backups to object storage, MinIO gives you S3-compatible storage, and OpenCost covers that cost allocation you lost with Cost Explorer.

The way to read this map is that none of these pieces is surplus if you want the cloud experience: they are the boxes the cloud ticked for you and that you now tick one by one.

The gap you will not close

I will tell you plainly, because it is the part the “leave the cloud” brochures skip. No combination of open source projects reproduces 100 % of the public cloud for you, and not for lack of maturity, but by design.

What you will not replicate is, first, instant hardware elasticity: you still buy the peaks up front. Second, the depth and breadth of the managed services catalogue, hundreds of services with a service level agreement and an on-call team behind them, which no internal team matches. Third, the global multi-region network the cloud gives you almost for free. And fourth, that model of transferred operational responsibility: with open source, operations come back home.

The correct reading is not that repatriating is a bad idea, but that you swap a predictable operating expense and a delegated responsibility for a cheaper capital investment and a responsibility of your own. For sustained base load it works out. For everything else, think about it.

When NOT to repatriate

This section has to be here, because repatriation badly applied is expensive. There are three clear signs that the cloud is still your right answer.

The first is a workload with extreme peaks or very variable traffic. Most inference deployments operate between 40 and 65 % GPU utilisation, and below roughly 70 % sustained usage the cloud beats you on total cost. If your workload has long troughs, you are buying iron so it can sleep.

The second is uncertainty or the start of a project. The break-even point in favour of buying only applies to you when utilisation is consistently high and, above all, measured and not projected. Buying GPUs against an optimistic forecast is the fastest way to end up with a warehouse of idle silicon. Sizing it properly is worth a whole article: sizing and justifying the GPU investment.

The third is the lack of an operations team. The total cost of ownership on-premise includes between half and one full-time engineer per GPU server. Without that team, your saving is illusory, because the on-call the cloud included is now covered by nobody. a16z itself frames its thesis as a hybrid approach, not as “leave the cloud” at all.

The numbers, to ground it

So as not to leave you with the discussion in the abstract, here are the orders of magnitude, triangulating two sources with opposite biases, a cloud provider and a neutral FinOps tool. An H100 in the cloud runs you between 2.90 and almost 7 dollars per GPU-hour depending on provider; bought, an H100 card is around 31,000 dollars and an eight-way system between 250,000 and 320,000. At full utilisation, a rented H100 passes its purchase price in less than a year, and an eight-way system with second-hand hardware can pay for itself in about seven months. The three-year total cost of an eight-H100 system, counting staff, colocation, energy and cooling, is estimated at between 700,000 and 950,000 dollars, and the threshold where on-premise beats the hyperscalers is around 80 % sustained utilisation.

The two sources disagree on the exact number, as you would expect, but they agree on the deciding factor: sustained utilisation, measured and not projected. The full working of this calculation is in the TCO of an on-premise GPU cluster. Here the conclusion is enough for you: for continuous base load the numbers work out, and that is exactly the profile of an inference factory.

For your sovereign inference factory

Everything above arranges itself into a practical decision. If you operate an LLM inference platform on your own iron, you are not deciding whether to repatriate (you already took that decision when you built the cluster), but how to rebuild the cloud experience your teams miss.

The sensible sequence starts with the substrate, which you probably already have in the form of RKE2 and storage. It goes on to the provisioning control plane, where Crossplane turns “open a ticket and wait” into a declarative API. It continues with the developer experience, where you choose between the Backstage portal, the KubeVela application model or the portable Score specification, depending on whether your people prefer a web or a CLI. And do not forget the on-premise managed services or the cost allocation, because without a bill your platform loses the discipline the cloud imposed by default.

The next two articles go down into the two layers least covered in this blog: Crossplane for the infrastructure control plane, and KubeVela with Score for CLI self-service centred on the application. With them, the map of this layer is covered, and your own house starts to look like the hotel you left, with the difference that the data stays inside.

Sources

Andreessen Horowitz (S. Wang, M. Casado), The Cost of Cloud, a Trillion Dollar Paradox — https://a16z.com/the-cost-of-cloud-a-trillion-dollar-paradox/
Data Center Dynamics, 37signals claims it saved almost 2M USD last year from cloud repatriation — https://www.datacenterdynamics.com/en/news/37signals-claims-it-saved-almost-2m-last-year-from-cloud-repatriation/
The Register, Developer pockets 2M USD in savings from going cloud-free — https://www.theregister.com/2024/10/21/37signals_aws_savings/
Barclays, Technology: 1H24 CIO Survey (PDF) — https://8198920.fs1.hubspotusercontent-na1.net/hubfs/8198920/Barclays_Cio_Survey_2024-1.pdf
Channelnomics, Breaking Down the 83% Public Cloud Repatriation Number — https://channelnomics.com/breaking-down-the-83-public-cloud-repatriation-number/
The New Stack, Updated Stats on Cloud Sustainability, Repatriation and Cost Optimization (Flexera 2025) — https://thenewstack.io/updated-stats-on-cloud-sustainability-repatriation-and-cost-optimization/
Comisión Europea, Data Act | Shaping Europe’s digital future — https://digital-strategy.ec.europa.eu/en/policies/data-act
EU Data Act, Article 29 — Gradual withdrawal of switching charges — https://www.eu-data-act.com/Data_Act_Article_29.html
Kiteworks, How the EU Data Act and GDPR Conflict with U.S. CLOUD Act — https://www.kiteworks.com/gdpr-compliance/eu-data-act-gdpr-cloud-conflict/
Computerworld, EU takes first steps to reduce reliance on US hyperscalers — https://www.computerworld.com/article/4181816/eu-takes-first-steps-to-reduce-reliance-on-us-hyperscalers.html
CNCF, Announces Graduation of Crossplane (6-nov-2025) — https://www.cncf.io/announcements/2025/11/06/cloud-native-computing-foundation-announces-graduation-of-crossplane/
CNCF, Score accepted as a CNCF Sandbox Project — https://www.cncf.io/blog/2024/08/08/score-accepted-as-a-cncf-sandbox-project/
Harvester HCI (SUSE) — https://harvesterhci.io/
CloudNativePG — https://cloudnative-pg.io/
Crossplane Blog, Building Modelplane on Crossplane — https://blog.crossplane.io/building-modelplane/ · https://modelplane.ai
Spheron, LLM Inference On-Premise vs GPU Cloud: 2026 Cost and Break-Even Analysis — https://www.spheron.network/blog/llm-inference-on-premise-vs-cloud/
CloudZero, H100 GPU Cost in 2026: Buy, Rent, and Cloud Pricing Compared — https://www.cloudzero.com/blog/h100-gpu-cost/

Backstage as a self-service portal: giving your teams the LLM platform without giving them the cluster

Mon, 31 Aug 2026 09:00:00 +0200

This article opens the platform and self-service layer: the one that decides how you hand everything underneath to the teams that are going to use it. The three articles in this batch are Backstage (the portal), Kubeflow (the ML toolbox) and Knative (switching the GPU off when nobody is using it).

TL;DR

Backstage solves a real problem of a platform that has grown: nobody knows what is there, who maintains it, or how to ask for a new one without opening a ticket. Its answer is three pieces, a software catalogue that models every component and its relations, templates that create new resources through a form instead of through copy and paste, and TechDocs to keep documentation next to the code. On top of those three sits the concept that holds up everything else, the golden path: the supported route made also the easiest one, so that people take it because it is paved, not because they are forced onto it.

The price is high and it should be said plainly. Backstage is not a product you install, it is a Node and React application that you compile and maintain as your own code, at a declared cost of two to three full-time engineers, and with upgrades as its biggest pain point. As of August 2026 it is still a CNCF incubating project, not graduated, and it has no native entity type for models, datasets or inference endpoints: that gets modelled by hand, or you wait for a proposal that is still open. The decision is not about features, it is about scale: below a certain organisation size, a portal costs more than it saves.

The analogy: the industrial estate directory

A single unit needs no sign. You know what is inside because you built it. An estate with two hundred units, on the other hand, needs a directory at the entrance: which company sits on each plot, what it does, who is responsible, and an office where you can request a new plot with the utility connections already in place instead of negotiating each one separately.

An LLM platform starts out as a single unit. One cluster, one model, the team that built it. Six months later it is an industrial estate: fifteen models served, three RAG pipelines, LoRA adapters everywhere, and a question that comes back in every meeting, who maintains this? Backstage is the directory for that estate, with two additions an ordinary directory does not have: the office that hands over turnkey plots, and the archive of blueprints for each unit. The catalogue is the directory, the templates are the office, TechDocs is the archive.

The analogy also warns about the cost. The directory does not maintain itself. Somebody has to update it when a company moves, and a directory with half the plots mislabelled is worse than no directory at all, because people stop trusting it. In Backstage that failure has a name, catalog rot, and it is the leading cause of a portal being abandoned.

What Backstage is and what state it reaches 2026 in

Backstage was born inside Spotify and donated to the CNCF, which accepted it in September 2020 and moved it to incubating level in March 2022. That is where it still is: as of August 2026 it is incubating, not graduated, a detail that matters when someone sells it as a de facto standard. It is by far the most widely adopted developer portal (the official 2025 review declares more than 3,400 organisations using it, 31,000 GitHub stars and more than 250 plugins), but adoption is not governance maturity, and a large share of those plugins is maintained by the community, not by the core.

The architecture has kept changing pieces under the feet of whoever operates it. The New Backend System stabilised in 2024 and is now the standard. The New Frontend System became the default mode for new applications in version 1.49, of March 2026. The cadence is one minor release a month, and that rhythm is precisely the origin of the biggest operational problem, which we will see further down.

The three pieces

The software catalogue

The heart of Backstage is an entity catalogue. Every piece of the platform is described in a catalog-info.yaml file that lives next to its code, and Backstage discovers it by crawling the repository. The entity types are fixed: Component (something that runs or compiles), API (a contract that a component exposes), Resource (infrastructure that a component needs), System (a set of components that form a unit), Domain (a set of systems), plus Group and User for ownership.

The power is in the relations rather than in the entities, which Backstage computes and navigates: ownedBy connects each thing to its team, providesApi and consumesApi draw who depends on whom, partOf groups components into systems, dependsOn traces the dependency graph. With that, the meeting question (who maintains this, and what breaks if we touch it?) stops being answered from memory and starts being answered by query.

There is a design decision that conditions everything that follows: Backstage advises against creating new entity types. Plugins are compiled expecting the usual types, and an invented type leaves them blind. The supported route is to reuse an existing type and distinguish it with the spec.type field (for example a Resource with type: llm-model), adding whatever custom fields you need to the spec.

The templates

The second piece is the Scaffolder, the template engine. A template is an entity of type Template with three parts: the parameters requested through a form, the steps that get executed, and the actions that do the real work (create a repository, write the catalog-info.yaml, register the component, trigger the GitOps pipeline). The developer goes to /create, picks a template, fills in the form, and in a single operation has the resource created, catalogued and deploying.

This is what turns a directory into an office. Without templates, Backstage only lists what already exists. With templates, it is the door through which new things are created, and that door is where the golden path materialises.

TechDocs

The third piece is TechDocs, documentation as code. Documents are written in Markdown next to the code, generated with MkDocs in the CI pipeline, and served inside the portal itself, linked from each component’s page. The advantage is in the incentives: documentation that lives far from the code rots; documentation that lives next to it and gets reviewed in the same pull request has some chance of staying true.

The golden path, which is what all of this is about

Spotify coined the term golden path for the recommended and supported way to build something. It has two halves: the blessed tooling, visible in the portal, and a step-by-step tutorial in the documentation. The idea, in the formulation that platform reports keep repeating, is that nobody forces you to use the paved road; people take it because it is paved.

The data supports the golden path being the deciding variable. The 2026 State of Platform Engineering report, covering 518 professionals, puts voluntary adoption above 80 % where there is a well-designed golden path and below 20 % where there is not, and estimates that around 70 % of platform initiatives fail to reach significant adoption without one. It is a figure that reorders priorities: a portal without golden paths is a pretty catalogue nobody uses.

In Backstage, a golden path is implemented as a Scaffolder template. For an LLM platform, a typical golden path would be “serve a new model”: the form asks for the model, the GPU size, the owning team and the SLO; the template creates the repository with the InferenceService, writes the catalog-info.yaml, registers it under the right system, and opens the pull request against the GitOps repository you already have with Flux. What used to be half a day of copying a colleague’s YAML and guessing the values becomes a two-minute form with the correct defaults.

Authentication and permissions

For a sovereign platform the identity piece matters as much as the catalogue. Backstage ships connectors out of the box for GitHub, GitLab, Google, Microsoft, Okta, generic OIDC and SAML, among others. For on-premise the relevant one is Keycloak, which fits in like any other OIDC provider, with one nuance worth keeping in mind: the Keycloak connector is maintained by the community, not by the core, so its maintenance and its tightness fall on a third-party plugin. Once you have set up Keycloak as we saw in the article on authenticating MCP, the portal hooks into the same identity issuer.

The permissions framework appears today as stable in the documentation and supports RBAC, ABAC and custom logic. The small print: by default the endpoints are not protected, and the permission policy is written by the integrator. The most widely used RBAC solution with a graphical interface comes with the Red Hat distribution, not with the core. In other words, serious access control means work on top of what comes in the box.

The real cost, no frills

This is where an honest article parts ways with the brochure. Backstage is not an installable product, it is a framework that you compile and maintain as your own code. That has measurable consequences.

The most quoted analysis, from the managed Backstage vendor Roadie, puts the effort at around three full-time engineers in the first year and two from the second onwards, with a time to production of six to twelve months in self-managed mode. The figure comes from a vendor with a commercial interest in you buying its managed version, so it should be taken as an order of magnitude and not as a neutral data point, but the direction is confirmed by independent sources: ongoing maintenance takes up around a fifth of engineering time, and upgrades are the biggest pain point, cited by more than half of those who adopt it. With one minor release a month and architectural pieces shifting under your feet, keeping up to date is a permanent job.

The other cost is about adoption, not engineering. A widely repeated critical analysis points out that many organisations settle at around 10 % internal usage, against the 99 % voluntary usage Spotify achieved at home, and attributes it to three recurring failures: the catalog rot we already mentioned, the lack of a product owner treating the portal as a product, and the illusion that Backstage can be adopted over what already exists when in practice it only standardises new things well. The sentence that sums up the disenchantment is that Backstage is a free portal in the same way you would have a free car if somebody tipped all the parts of a Chevrolet onto your desk.

Applied to an LLM platform: what is there and what is not

Here an expectation needs correcting. In 2026 Backstage has no native entities for models, datasets or inference endpoints. There are three things and none of them is what you would hope for:

An open proposal (RFC 33060, from February 2026) to model model servers as entities of type API with type: ai-model-server. It is under discussion, not implemented.

An official module that introduces an AiResource type, which sounds like what we are after but is for something else: governing the skills and rules of coding assistants, not registering inference models.

And one real pattern, the MLOps templates that AWS documents on top of SageMaker, where Backstage acts as the Scaffolder that creates environments and pipelines, not as the model registry. It is a cloud case, not on-premise, and it uses Backstage for its template half, not for its model catalogue.

The practical conclusion for your own stack is twofold. First, what Backstage brings to an LLM platform today is above all the template half: the “create a new inference service” golden path wired into your GitOps. Second, if you want to catalogue models, today that is done by hand with custom spec.type values, treating the model as a Resource and its endpoint as an API, until the proposal matures. Registering the model’s bytes remains the job of the OCI registry, not of the catalogue.

The infrastructure plugins that are worth it in this stack do exist and are reasonably alive: the Kubernetes one (core), the Flux one for seeing GitOps state from the portal, the Grafana one for embedding dashboards in a component’s page, and the ArgoCD one for anyone using Argo instead of Flux.

Alternatives: when not to set up Backstage

The honest decision is not which portal, but whether a portal. The SaaS-type alternatives (Port, Cortex, OpsLevel, Humanitec) win on time to first value, measured in days instead of months, and on not having to maintain the framework, in exchange for a per-user fee that scales badly and for giving up pure on-premise. Roadie offers managed Backstage, which removes the framework maintenance while keeping its extensibility, in exchange for a per-developer fee and for taking your catalogue outside.

For a sovereign platform, where on-premise and data ownership are requirements, pure SaaS is usually ruled out by design, which leaves the real choice between self-managed Backstage and no portal at all. And there the criterion is size. Below a handful of teams, a well-maintained README and a tidy GitOps repository do the same job as the catalogue, without the cost of two engineers. The portal starts paying off when the cognitive load of knowing what is there and how to ask for it exceeds the cost of maintaining the tool that answers it.

Operational traps and honest scepticism

The catalogue rots if it is not automated. A catalog-info.yaml written by hand and never updated is lying within a few weeks. The only defence is generating and validating those files in the pipeline, not trusting people to maintain them.

Without a product owner, the portal dies. Backstage is not infrastructure you install and forget, it is an internal product with users you have to understand. Without somebody treating adoption as an objective, it ends up as a catalogue nobody looks at.

Upgrades are a recurring tax. One release a month and architectural pieces mid-migration mean the portal is never finished. That maintenance has to be budgeted from the start, not discovered in the second quarter.

Backstage deploys nothing. It is a portal, not an engine. Behind each template there has to be your GitOps, your operators and your cluster doing the real work. If that base is not solid, the portal just puts a pretty façade over chaos.

The data model for AI does not exist yet. Anyone setting up Backstage expecting a ready-to-use model catalogue is going to find themselves modelling by hand. It is doable, but it is work, and you should know that beforehand.

For an inference factory

Backstage fits at the high maturity level of a platform, not at the start. In the five maturity levels it is a tool of the level where the platform already serves several teams and the bottleneck has stopped being technical and become one of coordination. Three concrete decisions follow from all of the above.

The first is not setting it up too early. If the platform is used by two teams who know each other, the portal is cost without return. The trigger is cognitive load, not the calendar.

The second is starting with the templates, not with the catalogue. The value you notice from day one is the golden path that turns half a day of copying YAML into a two-minute form. The complete catalogue, with all its relations, is a bigger job that can come later.

The third is budgeting maintenance as a function, not as a project. Two part-time engineers indefinitely, an owner watching adoption, and automatic generation of the catalogue entries. Without that, the portal degrades until somebody asks why nobody uses it, and the answer is that it stopped being true.

The portal is the face your platform shows to whoever uses it. It is worth it when there is enough platform behind it that showing it tidily saves more than maintaining the shop window costs. Before that, it is the shop window of a shop that has no stock yet. The other two articles in this batch go into that stock: the ML toolbox with Kubeflow and the savings of switching the GPU off with Knative.

Sources

CNCF, Backstage (project page) — https://www.cncf.io/projects/backstage/
Backstage, Backstage Wrapped 2025 — https://backstage.io/blog/2025/12/30/backstage-wrapped-2025/
Backstage Docs, Software Catalog: descriptor format — https://backstage.io/docs/features/software-catalog/descriptor-format/
Backstage Docs, Well-known relations — https://backstage.io/docs/features/software-catalog/well-known-relations/
Backstage Docs, Extending the model — https://backstage.io/docs/features/software-catalog/extending-the-model/
Backstage Docs, Software Templates — https://backstage.io/docs/features/software-templates/
Backstage Docs, TechDocs — https://backstage.io/docs/features/techdocs/
Backstage Docs, Authentication — https://backstage.io/docs/auth/
Backstage Docs, Permissions overview — https://backstage.io/docs/permissions/overview/
Backstage GitHub, RFC #33060: modeling AI model servers in the catalog — https://github.com/backstage/backstage/issues/33060
Roadie, Backstage: how much does it really cost? — https://roadie.io/blog/backstage-how-much-does-it-really-cost/
Earthly, Backstage is at the peak of its hype — https://earthly.dev/blog/backstage-is-at-peak-hype/
AWS Prescriptive Guidance, Accelerate MLOps with Backstage and SageMaker templates — https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/accelerate-mlops-with-backstage-and-sagemaker-templates.html
Spotify Engineering, How we use golden paths to solve fragmentation — https://engineering.atspotify.com/2020/08/how-we-use-golden-paths-to-solve-fragmentation-in-our-software-ecosystem
The New Stack, DORA 2024: AI and platform engineering fall short — https://thenewstack.io/dora-2024-ai-and-platform-engineering-fall-short/
bex.co, State of Platform Engineering Report Vol. 4: golden path — https://bex.co/blog/2026/08/06/platform-engineering-report-vol-4-golden-path

The blog map: from the utility feed to the token served

Sat, 29 Aug 2026 10:00:00 +0200

This blog started in March 2026 with a handful of articles about Kubernetes and networking, and by now it has piled up more than 150 pieces that, read one at a time, look like unconnected topics: PagedAttention, diesel generators, OpenCost, SPIFFE, immersion cooling, RAGAS. They are not. All of them describe the same object from different altitudes: a sovereign inference factory, that is, the complete machine that turns megawatts from the grid into tokens served to a user, with ownership and control of the data kept in house.

This article is the index of that machine. It orders what has been published into ten layers, bottom up, and links what is written in each one. It is good for three things: finding a specific topic, seeing which layers are covered in depth and which are half done, and choosing a reading route according to the problem in front of you.

The map

The map reads bottom up, and the criterion for ordering it is physical dependency: there is no inference engine without a powered and cooled GPU, no RAG without an engine serving embeddings, no demonstrable regulatory compliance without observability recording what happened. Layer 10 is the exception, and that is why it appears cutting across everything: cost, performance and energy are not a floor of the building, they are the measure of the whole building.

Layer 0 · Physical infrastructure

The most recent vertical, and the one that closes the circle: the path of the watt from the utility feed to the chip, and the path of the heat from the chip to the atmosphere.

The power chain: redundancy, Tier, kW against kVA, PUE, AI density and the limit of the grid.
Generators: NFPA 110, transients, sizing, diesel against gas, fuel and paralleling.
Switching: ATS against STS, the ITIC window, open and closed transition, and the A and B paths.
The UPS and ride-through: IEC 62040-3 topologies, eco-mode, flywheel, lithium against lead and the pulsing loads of AI.
The thermal challenge: why AI density broke air, and the spectrum of solutions.
Air and its limits: CRAC and CRAH, the geometry of the flow, the cube law, containment, ASHRAE and rear doors.
Direct liquid to the chip: cold plate, CDU, TCS and FWS loops, coolant chemistry and leaks.
Immersion and the megawatt horizon: single-phase and two-phase, the PFAS wall and microfluidics in the silicon.

Layer 1 · The node

Inside the machine, before any process exists: the buses, the memory and the topology that decide how much performance actually reaches the model.

PCIe topology, GPUDirect and ACS: the corridors the tensors travel through.
NUMA, hugepages and CPU isolation: why the neighbour on your socket costs you latency.
NVLink, NVSwitch and NCCL: the cable a large model is split across.
Network NUMA with Cilium and DraNet: the kitchen door nobody watches.
CPU, Memory and Topology Manager in RKE2: how to ask for a GPU with affinity and no surprises.
SM, CUDA streams and CUDA graphs: where time is lost inside the accelerator.
Mixed NVIDIA and Intel environments: from the homogeneous cluster to the real estate.
Measuring the power of a GPU: sensor metrology, which belongs to this layer and to layer 10 at the same time.

Layer 2 · Storage

State of the art, performance, security and availability: the set of four on the storage that holds weights, corpus and checkpoints.
From disk to HBM: why starting a model takes as long as it takes.
Speeding up the cold start: from minutes to seconds with serialisation and cache.

Layer 3 · The cluster

vLLM on Kubernetes: the inference piece that does scale.
LLM inference operators: OME, vLLM Production Stack and company.
The GPU cluster as a multi-tenant platform: from shared machine to service.
Sharing a GPU: time-slicing, MPS and MIG: the three ways to split an accelerator.
Volcano and Kueue: gang scheduling, queues and quotas.
Autoscaling with HPA and KEDA: scaling on metrics that mean something.
Serving several models on one GPU: co-residency, swap and sleep.
On-premise capacity planning: from the SLO to the number of nodes.
GitOps for the stack with Flux: operating the platform as code.
Cilium BGP on RKE2 and eBPF from zero to Cilium: the network underneath everything above.

Layer 4 · The inference engine

The densest block of the blog: vLLM cut open, piece by piece.

KV cache and PagedAttention: the working memory and its block manager.
Continuous batching and the scheduler step: how each round of work is assembled.
Optimising prefill and optimising decode: the two phases and their controls.
The attention backend and FlashAttention: the core of the computation.
Prefix cache: hit rate engineering and batch sizing with grid search: the two settings that give away the most performance.
Disaggregated serving and one big replica or many small ones: the topology decisions.
Long context and KV offloading: the KV memory hierarchy when the window does not fit in HBM.
Structured output and function calling: holding the model’s output in place.
Serving reasoning models: the invisible draft you pay for all the same.
Multimodal on-premise with vLLM and serving embeddings and rerankers with TEI: the other models in the stack.
The inverted roofline: why a small model is not optimised the same way as a large one.

Layer 5 · The model

Everything done to the model before serving it, or while it is being served.

Quantization for inference, FP8 end-to-end, aggressive sub-4-bit quantisation and quantising on the fly.
Pruning and distillation: reducing without amputating, and teaching a small model.
MoE in inference and device-native architectures.
Multi-LoRA serving, QLoRA and multi-LoRA at the limit and the complete QLoRA runbook.
Speculative decoding and self-speculative with early exit.
Modern alignment: DPO, KTO, ORPO and SimPO.
Continuous fine-tuning and retrain: closing the loop.

Layer 6 · Data and RAG

Document ingestion from the PDF to the indexed chunk and corpus curation.
Embeddings in 2026 and reranker with hybrid retrieval.
PostgreSQL and Qdrant in ingestion, Debezium and CDC and RAG over Kafka.
Semantic cache: the receptionist with a memory.
RAG on CPU and aggressive RAG on small models.
Evaluating a RAG without fooling yourself: RAGAS and the golden dataset.
Ontologies and knowledge graphs and data versioning with DVC and lakeFS.

Layer 7 · Control plane and delivery

Choosing the OSS gateway and the L7 inference router: the switchboard in front of the engine.
KServe and the Open Inference Protocol: the standard control plane.
Model registry and distribution with OCI and ORAS: where the bytes live.
Canary, blue-green and shadow: deploying a model without breaking the service.
Prompt versioning: the contract that avoids the scare.
MCP from the inside and adding authentication with Keycloak.
Operating the control plane (September 2026 run): the LiteLLM and Langfuse pair, day 2 of the proxy and virtual keys, budgets and limits and humans and agents on the same gateway, the prefix routing LiteLLM does not do and the MCP gateway as a second front door, sizing for agents and the seams with identity and traces: the gateway no longer as a piece you choose, but as a piece you operate.
Identity: Keycloak in an AI platform, with its position in the architecture and the MCP standards it does not implement, and how to complete it for MCP by building the protected resource side.
Operating Langfuse (a run of eight, September 2026): what goes into a trace, the first article about day 2 of version 4.
Building an end-to-end sovereign assistant, with all of the above pieces working together, is the exercise that closes this layer and is pending publication.

Delivery to the teams that use it is its own family of pieces:

Backstage as a self-service portal: the catalogue, the templates and the golden path, with the real cost of operating it.
Kubeflow in depth: which pieces of the ML toolbox are worth it and which ones you already have.
Knative and scale-to-zero: when switching off the GPU saves money and when the cold start violates your SLO.

Rebuilding the cloud experience on your own iron is a run of three:

From public cloud to private: what you lose when you leave the cloud and which projects fill each gap, with the business case and the sovereignty drivers.
Crossplane as a control plane: the declarative provisioning API that on-premise does not give you, and the cost of operating your own control plane.
KubeVela and Score: application-centric self-service by CLI, the friendly face Crossplane deliberately does not offer.

Layer 8 · Observability and evaluation

Tracing with OpenTelemetry GenAI and instrumenting vLLM with OTel.
Langfuse from the inside: the sorting centre for traces.
GPU observability with DCGM and the anatomy of DCGM and vLLM metrics.
Evals: the layer after tracing and LLM-as-judge.
Incident response runbooks: from the alert to the action.
Hubble, AgentSight and drift detection with eBPF: observing without instrumenting.

Layer 9 · Security, trust and compliance

Guardrails and safety and LLM Guard: the lines of defence around the model.
Hardening and secrets of the sovereign stack.
Isolating AI agents from the workspace and the runbook with bubblewrap and Tetragon, with Tetragon as the base.
Signing, provenance and AIBOM and identity and isolation with SPIFFE and Confidential Containers.
ISO/IEC 42001, the EU AI Act article by article and the cross-mapping of ENS, 42001 and the AI Act.
Compliance in healthcare and in defence: the same infrastructure facing two sector frameworks.

Layer 10 · Cost, performance and energy

The data track, written in a different format from the rest of the blog: tables, figures and a verifiable source, no analogies. These are the articles that hold up an investment decision.

FinOps: cost model and state of the art, OpenCost in depth, Kubecost against OpenCost, from the GPU-hour to the cost per token, chargeback and showback, the cost of the idle GPU, cloud GPU and neoclouds, full on-premise TCO and multi-tenancy FinOps with LiteLLM.

Benchmarking: frameworks and metrics, the tool catalogue, GuideLLM and the SLO under load, GenAI-Perf in depth, how to read MLPerf Inference, measurement bias and reproducibility, quality benchmarks and the Pareto frontier between engines.

Energy: measuring GPU power, energy per token in Spain, energy benchmarking frameworks, measuring in production with Kepler and DCGM, MLPerf Power, Wh per token leaderboards, from the watt to carbon, efficiency levers and energy in the TCO and in regulation.

Synthesis: the three axes and the identity that binds them, sovereign on-premise against hyperscalers, with data, from the SLO to the number of GPUs and the reproducible harness.

To that list you can add two cost pieces that are not about GPUs: the cost of agents with durable execution and the gateway’s cost per request.

The master maps

Five earlier articles work as the general plan and are still the best way in for anyone arriving new:

The six-stage LLMOps pipeline: the master map of the life cycle.
The seven layers of the on-premise stack: the architecture version of this same map.
Anatomy of an LLM request in production: a real request crossing every layer.
Five maturity levels of the platform and seven greenfield deployment phases: where you are and what comes next.
The OSS catalogue entry by entry and the parallel catalogue against the hyperscalers: which tool for each stage.
MLOps for LLMs in 2026: the starting landscape.

Four routes

I am about to build the first platform. Landscape and the six-stage pipeline, the seven layers of the stack, the seven deployment phases, capacity planning, vLLM on Kubernetes, gateway, observability with OTel and Langfuse, and guardrails before opening the door.

I have the cluster and the performance is not there. Anatomy of a request, KV cache and PagedAttention, prefill and decode, prefix cache, batch sizing, GuideLLM to find the saturation point, measurement bias so you do not fool yourself with the numbers, and the Pareto frontier between engines.

I have to justify the investment to management. The three axes, on-premise TCO, cost per token, idle GPU, on-premise against hyperscalers, sizing and justifying, and the reproducible harness so the figures survive an audit.

My job is the room, not the software. The power chain, the generators, the thermal challenge, air and its limits, direct liquid to the chip, immersion, and from there to storage and node topology.

What is missing

The map also serves to show the gaps, and three are identified, plus a line that is half open:

A comparison of vector databases: Milvus, Qdrant, pgvector and OpenSearch with data, beyond the scattered mentions.
Lakehouse for RAG: Iceberg and Trino as the substrate of the corpus.
Out-of-band management: OpenBMC, Redfish and DCIM, which is exactly the bridge between layer 0 and layer 3.
Sovereign substrate: the CLI self-service part is already covered by the layer 7 run (Crossplane, KubeVela and Score), and what remains is the IaaS-type substrate underneath, the sovereign alternatives to the hyperscaler (Spinifex, Harvester, OpenStack) on which everything else runs.

MLPerf Inference: how to read it, what LoadGen enforces and what comparability it offers

Sat, 29 Aug 2026 09:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter).

TL;DR

MLPerf Inference is maintained by MLCommons on a six-monthly cadence. It imposes three things no home-made benchmark has: a mandatory load generator (LoadGen, compiled from a tagged revision and unmodified), model-equivalence rules that bound what can be optimised, and an audit process that reviews up to two submissions per round. The four scenarios (SingleStream, MultiStream, Server, Offline) have a minimum duration of 600 s and different metrics: Server reports the sustained Poisson rate and is the only one with a latency constraint. For the LLM benchmarks that constraint is not a request latency but two simultaneous SLOs at the 99th percentile: TTFT and TPOT, with per-model figures ranging from 2000 ms / 200 ms on Llama 2 70B to 450 ms / 40 ms on its Interactive variant. Quality is required relative to the reference model (99 % or 99.9 %) and the generation length is audited as well (Llama 2 70B: more than 90 % of 294.45 tokens per sample). Three reading rules sum up the real comparability: results are only compared within the same version, division, category and scenario; per-accelerator is not an official MLCommons metric, it is derived; and neither cost nor energy enters the primary metric, to the point that round v5.1 had 2 power submissions against 27 participating organisations.

What it is and how it is organised

MLPerf Inference is maintained by the MLCommons Association through the MLPerf Inference Working Group, whose participation is limited to MLCommons members and affiliates (MLPerf Inference Working Group). The declared cadence for publishing results is roughly six months.

Round	Results published
v4.0	27 March 2024
v4.1	28 August 2024
v5.0	2 April 2025
v5.1	9 September 2025
v6.0	1 April 2026
v6.1	in preparation as of this data sheet

The suite splits into two system categories: Datacenter and Edge, the latter defined in the rules as everything that is not datacenter (inference_rules.adoc). A Datacenter system has two requirements that do not apply in Edge:

Mandatory ECC on DRAM and HBM, active during all performance and accuracy runs. There is no requirement on SRAM.
Mandatory networking since round v3.0, with a minimum bandwidth computed from the achieved throughput. For Llama 3.1 405B the minimum ingress is throughput × 20000 × dtype_size bytes/s; for Llama 2 70B the minimum egress is throughput × 1024 × dtype_size.

Edge additionally allows inferred results: a MultiStream derived from SingleStream counts as 8 times the p99 latency; an Offline derived from MultiStream, 8000 divided by the mean latency in milliseconds.

The four scenarios

This is the normative table, transcribed from the rules:

Scenario	Query generation	Duration	Samples/query	Latency constraint	Tail latency	Reported metric
SingleStream	The next query is issued when the SUT completes the previous one	600 s	1	None	90 %	p90 latency with early stopping
Server / Interactive	LoadGen issues queries following a Poisson distribution	600 s	1	Benchmark-specific	99 %	Maximum Poisson parameter supported
Offline	All samples are delivered at the start in a single query	1 query, 600 s	At least 24,576	None	N/A	Measured throughput
MultiStream	The next query is issued when the SUT completes the previous one	600 s	8	None	99 %	p99 query latency with early stopping

Four operational clarifications that change how each number is interpreted:

In Server, LoadGen does not measure a latency: it runs a binary search over the target rate. For a given value it generates queries at that QPS with Poisson arrivals; if the run fails the latency gate, it lowers the value and repeats. The final published metric is, per the official FAQ, scheduled samples per second, not completed samples nor the target entered by the user.
In MultiStream, the latency of a query is the maximum of the latencies of its samples, and the 8 samples of each query are contiguous in the order in which they were loaded. The value multi_stream_samples_per_query = 8 is fixed in the official configuration.
In Offline there is no latency constraint at all. The rules themselves forbid techniques that only work in fixed-length experiments except in the Offline scenario, which is a normative acknowledgement that an Offline result says nothing about behaviour under an SLO.
The percentiles in the table are the lower theoretical bound for runs with very many queries. With early stopping, the percentile actually computed is somewhat above the nominal one, as a penalty for processing few queries.

The mandatory scenarios depend on the benchmark. In Datacenter, the summarisation, question answering, text generation and VLM tasks require (Server | Interactive), Offline, that is, the submitter chooses between Server and Interactive. Reasoning and recommendation require Server, Offline. Medical segmentation, node classification, speech-to-text and RAG require only Offline.

LoadGen: what it measures and what it does not

LoadGen is the module that generates the load and computes the metrics. Its use is mandatory for all submissions, and it must be compiled from a tagged and approved revision of the repository, unaltered; the README is explicit that local modifications of the C++ library are not accepted.

Its four declared responsibilities are generating the queries according to the scenario, tracking the latency of each query, validating the accuracy of the results and computing the final metrics. Latency is defined as the time from when LoadGen was scheduled to pass a query to the system under test until it receives the response.

Equally important is what it declares out of scope: LoadGen does not know the model, does not know the data formats, cannot score accuracy and does not know the scenario constraints of the MLPerf rules. The consequence is written in its own README: being model-agnostic, it cannot enforce the MLPerf requirements, for example the target percentiles and latencies. Those limits are injected from mlperf.conf and user.conf. The justification for making it mandatory appears in the founding ISCA 2020 paper: it establishes a clear boundary between the components that belong to the submitter and those that belong to MLPerf, and it measures the performance of the complete system instead of that of a single piece.

Duration, query count and early stopping

The minimum duration is 600,000 ms in all four scenarios. The minimum number of queries comes from a confidence interval table that the rules publish:

Tail percentile	Confidence	Margin of error	Inferences	Rounded
90 %	99 %	0.50 %	23,886	24,576
95 %	99 %	0.25 %	50,425	57,344
97 %	99 %	0.15 %	85,811	90,112
99 %	99 %	0.05 %	262,742	270,336

The submission checker applies Server: 270336, SingleStream: 1024 and MultiStream: 662. For Offline, the minimum is set per benchmark: 24,576 on Llama 2 70B, 15,000 on Mixtral 8x7B, 13,368 on Llama 3.1 8B, 8,313 on Llama 3.1 405B, 4,388 on DeepSeek-R1, 1,633 on Whisper and 43 on 3D UNet.

The early stopping mechanism allows runs to be shortened while keeping the statistical guarantee. With tolerance d = 0 and confidence c = 0.99, the algorithm resolves by binary search the smallest number of queries below the latency threshold that satisfies the criterion, given the number of queries observed above it.

LoadGen’s modes are SubmissionRun (accuracy followed by performance), AccuracyOnly, PerformanceOnly and FindPeakPerformance. The last, applicable only to Server, takes target_qps as a lower bound if it passes, estimates the upper bound at double and doubles it until failure, and then does a binary search. In performance mode LoadGen selects queries uniformly at random with replacement from a set of size QSL; in accuracy mode it uses a copy of the validation dataset, each sample exactly once. One accuracy run must be executed for every performance result submitted, and the same code in both modes.

The seeds are announced four weeks before the deadline and the mandatory generator is Mersenne Twister 19937. They change every round: in v5.1, qsl_rng_seed = 1780908523862526354; in v5.0 it was 6023615788873153749.

The compliance tests

They are activated by placing an audit.config file in the working directory, whose parameters override those of mlperf.conf and user.conf. Their logs are mandatory in the submission package. The declared purpose is to detect anomalies, not to diagnose their cause.

For the LLM benchmarks the required test is TEST06, specifically against the end-of-sequence token exploit. It runs 100 samples and demands three conditions: that the first token reported separately matches the first token of the model’s output, that the output ends with zero or a single EOS token, and that the number of tokens reported matches the real length of the generated sequence.

Divisions, categories and what can be touched

The three divisions

Closed: requires preprocessing, postprocessing and model equivalent to the reference implementation. It allows calibration for quantisation and does not allow retraining. It is the only one that can use the MLPerf name unqualified.
Open: allows arbitrary preprocessing, postprocessing and model, including retraining. The accuracy, latency and scenario constraints do not apply: instead, the accuracy obtained and the latency constraints under which the performance was obtained must be reported. The model may have any origin, be quantised in any way and be sparsified in any way. It must use the same validation dataset as the corresponding Closed and use all of it.
Network: inherits all the Closed requirements, applies only to Datacenter, and separates the LoadGen node from the system under test through a fabric. The submitter’s dispatch library cannot preprocess, postprocess, batch, pad or cache. Bus interconnects are forbidden by name (PCIe, CXL, CCIX, HyperTransport, NVLink, QPI, UPI, ICI) and only Ethernet, IEEE 802.11, InfiniBand and 3GPP are admitted, with a requirement to work chassis to chassis at more than ten metres.

The availability categories

Category	Hardware	Software
Available in cloud	Available for cloud rental	Available
Available on premise	Available for purchase	Available
Preview	Must be available for the next submission, or the one after 140 days, whichever is longer	Available except what is needed for substantially new hardware
RDI	Does not meet the above	Does not meet the above

Available demands four cumulative conditions: price available, having been rented or shipped to at least one third party, public evidence of availability and reasonable availability to additional third parties on the submission date. A Preview result obliges a resubmission as Available with equal or better performance, tolerating up to 2 % degradation from noise; if there is no resubmission, the Preview result is marked invalid. RDI components cannot be submitted as Available until the cycle after the next one, or 221 days, whichever is longer.

What Closed allows and forbids

MLPerf delivers the weights in fp16 or fp32. The central rule is that the submitter may perform purely mathematical and reproducible quantisation, using only the calibration data and the tensors of the delivered model, to any numerical format that reaches the required quality, and that the method must be described publicly at a level that allows it to be reproduced. The test against back doors is elegant: the description of the quantisation method must be much smaller than the non-zero weights it produces.

Allowed, among other things: any framework or runtime, arbitrary data layout, varying the matrix multiplication algorithm, mathematically equivalent transformations, polynomial approximations of transcendentals, processing queries out of order within what the scenario admits, replacing dense operations with mathematically equivalent sparse operations, hand-picking different precisions per operation, fusing and unfusing, dynamic batching, and mixtures of experts combining weights with different quantisation.

Forbidden: replacing or supplementing weights wholesale, discarding non-zero weights, including pruning, caching queries or responses, coalescing identical queries, modifying weights during the timed portion, quantisation algorithms of size comparable to the weights they produce, hardcoding the total number of queries, using knowledge of the LoadGen implementation to predict peaks or troughs in the Server scenario, changing the number of search beams, and incorporating statistics from the performance or accuracy sets.

Two clarifications that matter to anyone serving LLMs in production: the KV cache is allowed just as in the reference model provided it is not applied across queries; PagedAttention is admitted if the block is reused only within the batch; continuous or dynamic batching is allowed; and KV cache entries are treated as activations for quantisation purposes, and cannot be pruned. Speculative decoding is only admitted in the benchmark and scenario combinations explicitly listed, with the reference head at the same precision, and implementations that artificially manipulate the acceptance rate are forbidden.

The SLOs almost nobody cites: TTFT and TPOT

This is the point where an MLPerf table is most often misread. In the LLM benchmarks of the Server scenario, the constraint is not a request latency: the rules set target_latency = 0 and activate use_token_latencies = 1, so that the gate is formed by two simultaneous metrics, both evaluated at the 99th percentile:

TTFT (time to first token): latency of the first token.
TPOT (time per output token): mean interval between all generated tokens.

The figures are in mlperf.conf and change per round and per benchmark:

Benchmark (Server scenario)	TTFT	TPOT	Round
`llama2-70b`	2000 ms	200 ms	v5.0, v5.1, v6.x
`llama2-70b-interactive`	450 ms	40 ms	v5.1, v6.x
`mixtral-8x7b`	2000 ms	200 ms	v5.0, v5.1, v6.x
`llama3_1-405b`	6000 ms	175 ms	v5.0, v5.1, v6.x
`llama3_1-405b-interactive`	4500 ms	80 ms	v6.x
`llama3_1-8b`	2000 ms	100 ms	v5.1, v6.x
`llama3_1-8b-interactive`	500 ms	30 ms	v6.x
`deepseek-r1`	2000 ms	80 ms	v5.1, v6.x
`deepseek-r1-interactive`	1500 ms	15 ms	v6.x
`gpt-oss-120b`	3000 ms	80 ms	v6.x

A TPOT of 40 ms is equivalent to 25 tokens per second per user, which is the order of magnitude of a conversational interface; one of 200 ms is equivalent to 5 tokens per second, which is a batch use case dressed up as a service. Two results for the same model under the Server and Interactive labels are not the same measurement. Two documented discrepancies between primary sources remain: the llama2-70b-interactive variant does not appear in the mlperf.conf tagged as v5.0 despite having been announced in that round, and in the v6.x cycle the TPOT of gpt-oss-120b-interactive appears as 15 ms in mlperf.conf and as 20 ms in the text of the rules.

For contrast, the constraints of the non-LLM benchmarks in the Server scenario are conventional request latencies: ResNet-50 15 ms, RetinaNet 100 ms, BERT 130 ms, DLRMv2 60 ms, DLRMv3 80 ms, RNN-T 1000 ms, GPT-J and Stable Diffusion XL 20,000 ms.

Quality: the 99 % and the token count

The quality targets are expressed relative to the reference model, never in absolute terms, and each benchmark demands one variant, the other or both. In the current Datacenter suite, 3D UNet, Llama 3.1 8B and Whisper ask for 99 % and 99.9 % of FP32, which generates two distinct results; Llama 2 70B asks only for 99.9 %; Llama 3.1 405B, Mixtral, DeepSeek-R1, GPT-OSS-120B, Qwen3-VL, RGAT and WAN-2.2 ask only for 99 %; and E2E-RAG is the exception at 97 %. In the checker this materialises as separate benchmarks (bert-99 and bert-99.9), each with its computed numerical threshold.

On top of that metric there is a second LLM-specific constraint that is cited very rarely: the generation length is audited as well.

Benchmark	Length constraint
Llama 2 70B	tokens per sample > 90 % of 294.45
Llama 3.1 405B	tokens per sample between 90 % and 110 % of 684.68
Mixtral 8x7B	tokens per sample between 90 % and 110 % of 144.84
Llama 3.1 8B	total generated length > 90 % of 8,167,644

The FAQ settles the obvious shortcut: reducing the maximum output length below the reference is not allowed, and truncating tokens to improve performance or reach the accuracy target is not permitted. The inference parameters are fixed in Closed (max_new_tokens = 1024 on Llama 2 70B, 20000 on Llama 3.1 405B and DeepSeek-R1), as is the decoding algorithm: greedy search in the Llama family, Mixtral and DeepSeek-R1; sampling with temperature = 1.0 and top_p = 1.0 in GPT-OSS-120B and Qwen3-VL. Accuracy is reported to five significant figures with round-half-to-even.

The current Datacenter suite

Area	Task	Model	Dataset	QSL	Required quality
Vision	Medical segmentation	3D UNet	KiTS 2019	42	99 % and 99.9 % of FP32 (DICE 0.86330)
Language	Summarisation	Llama 3.1 8B	CNN DailyMail v3.0.0	13,368	99 % and 99.9 % of FP32
Language	Question answering	Llama 2 70B	OpenOrca	24,576	99.9 % of FP32
Language	Text generation	Llama 3.1 405B	LongBench, Ruler, GovReport	8,313	99 % of FP16
Language	QA, maths and code	Mixtral 8x7B	OpenOrca, GSM8K, MBXP	15,000	99 % of FP16
Language	Reasoning	DeepSeek-R1	`mlperf_deepseek_r1`	4,388	99 % of FP16 (exact match 81.9132 %)
Language	QA, maths and code	GPT-OSS-120B	AIME25, GPQA Diamond, LiveCodeBench v6	6,396	99 % of 83.13 %
Vision	Vision-language model	Qwen3-VL-235B-A22B	Shopify product catalogue	48,289	99 % of BF16 (hierarchical F1 0.7824)
Commerce	Recommendation	DLRMv3	Synthetic Streaming 100B	34,996	99.9 % of FP32
Generative	Text to video	WAN-2.2-T2V-A14B	VBench prompts	248	99 % of BF16 (VBench 69.7752)
Graphs	Node classification	RGAT	IGBH	788,379	99 % of FP32 (72.86 %)
Audio	Speech to text	Whisper	LibriSpeech	1,633	99 % and 99.9 % of FP32 (WER 2.0671 %)
Language	End-to-end RAG	E2E-RAG	FRAMES	824	97 % of FP32

The E2E-RAG benchmark deserves attention from anyone designing a RAG platform, because it fixes the whole pipeline: a FAISS HNSW index with mandatory parameters M = 32, efConstruction = 200 and efSearch = 100; e5-base-v2 embeddings; ColBERTv2 reranking; generation and query rewriting with GPT-OSS-120B; evaluation with Llama 3.1 8B as judge; and a maximum of 5 retrieval iterations, mandatory for all submitters. Model loading is not timed; building the vector database is.

The definition of sample changes per model and is a frequent source of confusion: in the LLMs it is a sequence, in DLRMv3 it is a request with a user history and 2048 candidates, in WAN-2.2 it is a pair of positive and negative prompts, and in PointPainting it is five images and a lidar point cloud.

How to read a results row

The columns of the official table in the Closed division are Submitter, Software, System, Benchmark Results, Processor/Count, Details, Accelerator/Count and Code. The Open division adds Model Used and Notes. Rows with power measurement add System Power in Server and Offline, or Energy Per Stream in the stream scenarios. Any citation must use the identifier in the format major-version.minor-version.entry.benchmark, of the style 5.1-0053, and carry a footnote with suite, version, division, benchmark, scenario, date and source.

Per-accelerator is not an official metric

There is no per-accelerator column. It is a derived metric, and the MLCommons messaging guidelines are blunt: any comparison based on a different or derived metric, such as power, cost, model size or accuracy, must make the basis of comparison clear in the text and in a footnote, and secondary and derived metrics cannot be presented as official or verified MLPerf metrics. NVIDIA itself states this with the standard formula in its round blogs: per-GPU performance is not a primary MLPerf Inference metric and is computed by dividing the reported throughput by the number of reported accelerators. The same applies to combining results from several benchmarks: MLCommons allows it but does not endorse it, and the composite cannot be presented as an official result.

The comparison rules

MLPerf results can only be compared with compatible MLPerf results: same benchmark, same scenario and compatible versions per the normative compatibility table.
MLPerf results cannot be compared with non-MLPerf results.
When comparing, any difference in version, division, category, verification status, scenario or number of chips must be clearly identified. When comparing Open with Closed, it must be identified in what sense the Open result would not qualify as Closed.
Submitters cannot publish results of a version before its official date; non-submitters must wait two weeks from that date.
The sanctions regime goes as far as barring an infringer from submitting results in the future and marking their results as non-compliant permanently in the database, with a deadline of three working days to withdraw the infringing content.

How much audit sits behind a row

In each round up to two submissions are audited: one chosen at random from all of them and zero or one chosen by the review committee. And there is a constraint that conditions the reading of the whole table: only Available submissions in the Closed division are auditable. Preview, RDI and Open fall outside the process. There is an exemption from the random audit if the system is equivalent to another already audited and neither the aggregate performance nor the per-accelerator performance differ by more than 10 % from those of the previous audit. The deadlines are 28 days to select an auditor, 30 for the report after signing the confidentiality agreements and around 90 days for the complete process, with two days of hardware access for the auditor. The rules close with three sentences that define the standard of proof: results that cannot be replicated are not valid results, benchmark detection is not allowed and input-based optimisation is not allowed either.

The recent rounds, in figures

Round	Organisations	Performance results	Power results	New benchmarks
v4.0	23	more than 8,500	900	2
v4.1	22	964	31	1
v5.0	23	17,457	n/a	4
v5.1	27	n/a	2	3
v6.0	24	n/a	n/a	5 of 11 Datacenter tests

These counts do not form a time series: the counting criteria differ between press releases and none documents the criterion, to the point that v4.0 and v4.1 differ by a factor close to nine.

The published performance jumps are comparable within each announcement. Between v4.0 and v5.0, twelve months apart, the number of Llama 2 70B submissions multiplied by 2.5, the median score doubled and the best score was 3.3 times faster. Between v5.0 and v5.1, in six months, the best systems improved by up to 50 % in some scenarios. Round v6.0 changed axis and its press release talks about scale rather than speed: 30 % more multi-node submissions, 10 % of the systems with more than ten nodes against 2 % in the previous round, and a system larger than 72 nodes and 288 accelerators, quadrupling the previous maximum node count.

Hardware that debuted in each round: MI300X, TPU v6e, Xeon Granite Rapids and B200 in v4.1; MI325X, Xeon 6980P, GB200 and Jetson AGX Thor in v5.0; MI355X, Intel Arc Pro B60, GB300 and RTX Pro 6000 Blackwell Server Edition in v5.1.

What MLPerf Inference does not measure

Cost. There is no cost or TCO metric, and cost is explicitly classified as a derived metric that cannot be presented as official.
Energy, except as an optional extension. MLPerf Power is regulated in a separate document, requires measuring at the wall with the power group’s workflow and PTDaemon, and forbids any other method. Its real adoption is marginal: 2 power submissions in v5.1 against 27 participating organisations. In addition, MLCommons bounds what that figure means: the measured power is only valid for the benchmark it accompanies, and any other reference to power, such as a TDP setting or a power supply rating, is neither measured nor validated by MLCommons. The messaging rule forbids submitters from publishing per-watt normalised comparisons using any proxy other than the measured power.
Configurations not optimised by the vendor. The bias is acknowledged in the audit rule itself, which contemplates submissions whose performance is not consistent with the known characteristics of the hardware, or where the committee lacks visibility into how it was achieved, or where hardware and software are not reasonably available to the public.
Barrier to entry. Only MLCommons members and test partners can submit results for review. Anyone measuring with the same code without submitting must label every figure as unverified with the corresponding legend.

Usage flow for an on-premise platform

MLPerf Inference is useful in a purchasing decision if it is used for what it is, a test bench with written rules, and not as a ranking.

Filter by division and category before looking at any number: keep Closed and Available. A Preview or RDI result describes hardware that cannot yet be bought, and neither of the two is auditable.
Choose the scenario that matches the load pattern: Server or Interactive if there are users waiting, Offline if the work is batch. An Offline result does not authorise promising a latency.
Read the SLO before the throughput: in any LLM benchmark, the number of queries per second only means something alongside its TTFT/TPOT pair. The same machine publishes very different figures under Server and under Interactive.
Check the quality variant: 99 % and 99.9 % are different results for the same model, and the second usually costs performance.
Translate to your own configuration carefully: dividing by the number of accelerators is a derived metric, not an official one, and it has to be declared as such.
Close the gap MLPerf leaves open with your own measurement: cost, energy and behaviour under real traffic. That is exactly the function of the reproducible harness and of the track B bench tools, and the reason no platform decision stands on a public table alone.

Cross-links from the benchmarking track

B1/B2 — LLM inference benchmarking: frameworks, metrics and state of the art: the TTFT, TPOT and goodput metrics outside the MLPerf corset, and which tool measures each one.
B2 — LLM benchmark tool catalogue: the tools with which you reproduce at home what MLPerf standardises.
B3 — GuideLLM and SLO validation under load: the practical equivalent of LoadGen’s binary search in the Server scenario.
B6 — Measurement bias and reproducibility: why LoadGen exists, told from the side of what happens when there is none.
B8 — Serving engines compared on the Pareto frontier: the comparison MLPerf does not make, with engines instead of systems.
C4 — MLPerf Power: the energy extension of this same machinery, with its certified measurement at the wall.

Sources

MLCommons, MLPerf Inference Working Group — https://mlcommons.org/working-groups/benchmarks/inference/
MLCommons, MLPerf Inference Rules (inference_rules.adoc) — https://github.com/mlcommons/inference_policies/blob/master/inference_rules.adoc
MLCommons, MLPerf Inference Power Measurement (power_measurement.adoc) — https://github.com/mlcommons/inference_policies/blob/master/power_measurement.adoc
MLCommons, MLPerf Submission Rules (submission_rules.adoc) — https://github.com/mlcommons/policies/blob/master/submission_rules.adoc
MLCommons, MLPerf Results Messaging Guidelines — https://github.com/mlcommons/policies/blob/master/MLPerf_Results_Messaging_Guidelines.adoc
MLCommons, LoadGen README — https://github.com/mlcommons/inference/blob/master/loadgen/README.md
MLCommons, loadgen/test_settings.h — https://github.com/mlcommons/inference/blob/master/loadgen/test_settings.h
MLCommons, loadgen/mlperf.conf (master) — https://github.com/mlcommons/inference/blob/master/loadgen/mlperf.conf
MLCommons, loadgen/mlperf.conf (tag v5.1) — https://github.com/mlcommons/inference/blob/v5.1/loadgen/mlperf.conf
MLCommons, loadgen/mlperf.conf (tag v5.0) — https://github.com/mlcommons/inference/blob/v5.0/loadgen/mlperf.conf
MLCommons, tools/submission/submission_checker.py (tag v5.0) — https://github.com/mlcommons/inference/blob/v5.0/tools/submission/submission_checker.py
MLCommons, compliance/nvidia TEST06 README (tag v5.1) — https://github.com/mlcommons/inference/blob/v5.1/compliance/nvidia/TEST06/README.md
MLCommons, Benchmark results: Inference Datacenter — https://mlcommons.org/benchmarks/inference-datacenter/
Reddi et al., MLPerf Inference Benchmark, ISCA 2020 — https://arxiv.org/abs/1911.02549
MLCommons, MLPerf Inference v5.0 Benchmark Results (abril 2025) — https://www.businesswire.com/news/home/20250402313932/en/MLCommons-Releases-New-MLPerf-Inference-v5.0-Benchmark-Results
MLCommons, MLPerf Inference v5.1 Benchmark Results (septiembre 2025) — https://www.globenewswire.com/news-release/2025/09/09/3147136/0/en/MLCommons-Releases-New-MLPerf-Inference-v5-1-Benchmark-Results.html
MLCommons, MLPerf Inference v6.0 Benchmark Results (abril 2026) — https://www.globenewswire.com/news-release/2026/04/01/3266801/0/en/MLCommons-Releases-New-MLPerf-Inference-v6-0-Benchmark-Results.html
MLCommons, MLPerf Inference v4.1 Benchmark Results (agosto 2024) — https://www.businesswire.com/news/home/20240828886616/en/New-MLPerf-Inference-v4.1-Benchmark-Results-Highlight-Rapid-Hardware-and-Software-Innovations-in-Generative-AI-Systems
NVIDIA Developer Blog, NVIDIA Blackwell delivers massive performance leaps in MLPerf Inference v5.0 — https://developer.nvidia.com/blog/nvidia-blackwell-delivers-massive-performance-leaps-in-mlperf-inference-v5-0/
NVIDIA Developer Blog, NVIDIA Blackwell Ultra sets new inference records in MLPerf debut — https://developer.nvidia.com/blog/nvidia-blackwell-ultra-sets-new-inference-records-in-mlperf-debut/
MLCommons, MLPerf Automotive (octubre 2025) — https://arxiv.org/html/2510.27065v1
Tschand et al., MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts — https://arxiv.org/abs/2410.12032

Measuring the power of a GPU: NVML, DCGM and the sampling errors that invalidate your watts

Sat, 29 Aug 2026 08:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar symbol is not used (on this site it is a formula delimiter).

TL;DR

A GPU power sensor does not return what almost everyone believes. The NVML documentation is explicit: on Ampere except GA100, and on later architectures including H100, nvmlDeviceGetPowerUsage returns power averaged over a one-second interval, not instantaneous power. The University of Oxford study presented at SC24, covering more than 70 GPUs across 25 models checked against an external 1 mΩ meter, adds the fact that breaks most published measurements: on the A100 and H100 the sensor only samples 25 % of the time (a 25 ms window within a 101 ms period), so during the remaining 75 % the GPU may be consuming something radically different. Integrating power naively over nine real benchmarks, the mean error was 39.27 %; applying good practice brought it down to 4.89 %. The operational consequence is short: if the hardware is Volta or later, use the accumulated energy counter (nvmlDeviceGetTotalEnergyConsumption, in millijoules) and subtract, rather than integrating samples. And the GPU counter is not the bill: on measured 8× H100 nodes, the node reaches 8.4 kW against 5.6 kW of summed GPU TDP, node idle is 1.8 kW, and with an average PUE of 1.54 the NVML watts are of the order of 43 % of the watts the utility bills.

What the hardware exposes

NVML: power

nvmlDeviceGetPowerUsage(device, unsigned int* power) returns milliwatts for the GPU and its associated circuitry, for example the memory. It has been supported since Fermi. The note in the official documentation is the part almost nobody quotes (NVML Device Queries):

On Fermi and Kepler the reading is accurate to ±5 % of current consumption. On Ampere (except GA100) or later, the API returns power averaged over a 1 s interval. On GA100 and earlier architectures instantaneous power is returned.

In other words, on an H100 or an L40S the usual call already delivers a one-second moving average. Anyone sampling it at 1 Hz and integrating is applying a second filter on top of an already filtered signal, without declaring it. To disambiguate, NVML exposes two separate fields accessible via nvmlDeviceGetFieldValues: NVML_FI_DEV_POWER_AVERAGE and NVML_FI_DEV_POWER_INSTANT, with command-line equivalents (nvidia-smi --query-gpu=power.draw.average and power.draw.instant).

NVML: the energy counter

nvmlDeviceGetTotalEnergyConsumption returns accumulated energy in millijoules since the last driver reload, supported on Volta or later. It is a 64-bit integer, so overflow is not a practical problem: an H100 at 700 W consumes of the order of 6.1·10¹⁰ J per year against a counter range of 1.8·10¹⁶ J. The only event that resets it is a driver reload or a system restart, and that does need watching: a modprobe -r nvidia halfway through a campaign produces a negative delta.

NVIDIA does not document the internal time resolution of the counter, its accumulation mechanism or its accuracy, and I have found no peer-reviewed work validating it against an external meter. The recommendation to use it, which the rest of this sheet rests on, is a structural argument (it removes quadrature error and aliasing error), not a published empirical validation.

NVML: power limits

Function	Unit	Semantics
`nvmlDeviceGetPowerManagementLimit`	mW	Currently configured limit
`nvmlDeviceGetPowerManagementLimitConstraints`	mW	Legal minimum and maximum range
`nvmlDeviceGetPowerManagementDefaultLimit`	mW	Limit the card boots with
`nvmlDeviceGetEnforcedPowerLimit`	mW	Effective limit after considering all limiters, including the out-of-band interface

For an auditable measurement you have to record EnforcedPowerLimit, not PowerManagementLimit: the BMC may be imposing an out-of-band ceiling that the latter does not reflect, and two apparently identical nodes may be operating at different ceilings.

DCGM: the equivalent fields

Field	ID	Unit	Type in the exporter
`DCGM_FI_DEV_POWER_USAGE`	155	watts in floating point	gauge
`DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION`	156	accumulated millijoules	counter
`DCGM_FI_DEV_POWER_MGMT_LIMIT`	160	mW	gauge
`DCGM_FI_DEV_ENFORCED_POWER_LIMIT`	164	mW	gauge

Both energy fields are active in the default counter file of dcgm-exporter. The audit conclusion is direct: a standard Prometheus panel integrating DCGM_FI_DEV_POWER_USAGE is integrating a gauge that is already a one-second average of the sensor, sampled at the exporter’s collect-interval. For energy per token the correct metric is counter 156 with increase().

The real accuracy of the sensor

The reference is the work of Yang, Adámek and Armour (University of Oxford), published at SC24 and available as arXiv:2312.02741: more than 70 GPUs across 25 models, including 10 H100s and 10 A100s, checked against an external meter with a 1 mΩ shunt, a 12-bit ADC and internal sampling at 34 kHz.

Update period and averaging window

GPU or architecture	Update period	Averaging window
Kepler and Maxwell	10-50 Hz	logarithmic growth, ~200 ms
Volta (V100)	20 ms	10 ms
Turing	100 ms	100 ms
A100 (GA100)	101 ms	25 ms
Non-GA100 Ampere and Ada, driver earlier than March 2023	100 ms	1 s
Non-GA100 Ampere and Ada, driver 530	100 ms	100 ms
H100	100 ms	25 ms with `.instant`; 1 s with `.average` and with the default field
GH200	100 ms	20 ms on the GPU, 10 ms on the Grace CPU

Two readings of this table. The first: the driver is an experimental variable, not an infrastructure detail; the averaging window changed between drivers earlier than 530, driver 530, and the later ones. Any measurement meant to be compared over time has to record the driver version.

The second is the central finding of the paper. On the A100 and H100 the 25 ms window sits inside a 101 ms period, which gives a sensor duty cycle close to 25 %. The sensor does not average the full interval: it averages a quarter of it and the system presents the result as if it described the whole interval. In the words of the work itself, during the other 75 % of the time the GPU may be consuming radically different power and nvidia-smi never notices.

The steady-state error

The paper also quantifies the claim in the nvidia-smi documentation, which spoke of an accuracy of ±5 watts: the real error is proportional, ±5 %, not absolute. On a GPU capable of consuming 700 W that is ±35 W of over- or underestimation. The errors measured against the external meter, after correcting the time offset, were −4.70 % and −4.53 % on the RTX 3090 and −5.43 % on the A100, and the residue is attributed to the tolerance of the card’s own physical shunt, which no software can correct.

The response to transients

Four distinct behaviours were observed in the 10 % to 90 % rise time: an almost instantaneous real rise with the reading following on the next tick and a delay of 0 to 100 ms; a real rise of several hundred milliseconds with the reading still updating on the next tick; linear growth over a second, which corresponds to power.draw.average; and logarithmic growth over 200 ms, only on Kepler and Maxwell. Out of this comes the most uncomfortable warning in the work: when running a short program, the measured power probably corresponds to the activity before the program.

The two ways of obtaining energy

The first is to integrate samples, which is what any poller does:

$$E \approx \sum_i P(t_i) \cdot \Delta t$$

It accumulates five sources of error: the ±5 % of the sensor, quadrature error from the finite step, aliasing if the load has spectral content above half the sampling frequency, the fact that each sample is already a moving average, and the jitter of 0 to 100 ms between the host clock and the sensor tick.

The second is to subtract the counter:

$$E = \text{counter}(t_1) - \text{counter}(t_0)$$

Two calls and a subtraction. It removes quadrature, aliasing and time misalignment; the ±5 % of the sensor remains.

Why 1 Hz destroys a 200 ms inference

With sampling at 1 Hz, the sampling theorem only allows components below 0.5 Hz to be reconstructed, while a 200 ms pulse has content in the region of 5 Hz. The expected number of samples inside the event is 0.2, so with probability 0.8 none falls inside it: the result is not noisy, it is undefined.

Even sampling at the maximum useful rate, which on the A100 and H100 is about 10 Hz given the 101 ms period, a 200 ms inference produces one or two distinct sensor updates, and each has observed only 25 ms of real activity: effective coverage is around 25 % of the event. And because of the fourth transient behaviour in the previous section, the reading obtained during those 200 ms may describe what was happening before the kernel was launched.

Aliasing with periodic loads

The paper observes the phenomenon directly on the A100: with a square wave of period slightly different from 100 ms, the reading fluctuates between high and low values with a clear beat. An inference server with periodic arrivals, health checks, fixed-size batches or a scheduler tick is exactly that pathological case.

When only instantaneous power is available, the measured good practice from the Oxford work consists of running 32 consecutive iterations or a minimum of 5 s, inserting 8 evenly spaced controlled delays if the window is smaller than the period, repeating over 4 separate trials with a random delay between them, and shifting the series in post-processing to synchronise it with the real activity. Over nine real benchmarks that took the mean error from 39.27 % to 4.89 %.

What the GPU counter does not measure

The scope declared by NVML is the GPU and its associated circuitry: the SXM module or the PCIe card, with its HBM and its regulators. Left out are the CPU, host DRAM, NVSwitch, network cards, storage, fans and power supply losses.

The measured node figures come from the empirical calibration work on 8× H100 SXM5 nodes (arXiv:2506.14551):

Quantity	Value
Nominal node TDP declared by the manufacturer	10.2 kW
Node power at idle	1.8 kW
Maximum measured under saturating load	8.4 kW
Peak on real production loads	never above 76 % of TDP

Two conversion factors come out of that, both my own estimates from those figures: 8 × 700 W is 5.6 kW of GPU against 8.4 kW of node, that is, the GPU is of the order of 67 % of node power and the GPU-to-node factor is around 1.50×. Chaining that with an industry average PUE of 1.54 (Uptime Institute, 2025), the total factor from the NVML counter to the utility feed is around 2.3×; with a PUE of 1.1 it drops to 1.65×. Put in the form that matters to whoever signs the bill: the watts NVML reports are roughly 43 % of the watts the utility bills in an average datacenter.

As a complementary reference, an 80 PLUS Titanium supply at 230 V is 96 % efficient at 50 % load and 91 % at 100 %, so supply losses add between 4 % and 9 % that is already included if you measure at the socket.

The tax of keeping a context open

A GPU serving a loaded model with no traffic does not consume its nominal idle. Measurements over 335,267 production samples from 14 H100s across 18 days, plus controlled experiments, give this table:

GPU	Idle without a CUDA context	Idle with a CUDA context	Extra cost	Percentage of TDP
H100	71.8 W	121.7 W	+49.9 W	7.1 %
A100	53.7 W	80.0 W	+26.3 W	8.8 %
L40S	35.6 W	102.1 W	+66.4 W	19.0 %

The result that changes the mental model: more than 98 % of that extra cost is produced by the open CUDA context, independently of the memory occupied. Varying allocated VRAM between 0 and 72 GB moves power by less than 1 W. The size of the loaded model does not cost watts; keeping the context open does.

That cost is not marginal on a real platform. Over 11,791 long-running jobs, the measured split was 24 % of the time and 7 % of the energy in deep idle, 15 % of the time and 10 % of the energy idle with an active context, and 61 % of the time with 83 % of the energy in execution. On serving loads, GPUs can spend 48 % of their energy in low-activity periods.

CPU and DRAM: RAPL

To close the node you have to measure what is not GPU, and there the interface is RAPL through the kernel powercap framework, at /sys/devices/virtual/powercap/intel-rapl/. The hierarchy exposes intel-rapl:N as the socket and intel-rapl:N:M as subzones (core, uncore, dram), plus the psys domain for the full SoC since Skylake.

File	Content
`energy_uj`	Energy counter in microjoules; writing “0” resets it if the counter supports it
`max_energy_range_uj`	Counter range, that is, the overflow point
`power_uw`	Current power in microwatts
`name`	Zone name

The figures that condition its use come from the characterisation published in ACM TOMPECS: an update roughly every 1 ms, with jitter; an energy quantum of 61 µJ on Haswell and Skylake, 15.3 µJ on Sandy Bridge; and counter overflow in 52 minutes on a Haswell at 84 W, which forces polling at a far shorter period and detecting the wrap with max_energy_range_uj. There is also a thermal drift that is rarely declared: package power for the same load grows between 10 % and 12 % between 37 °C and 74 °C on Haswell. Warming up beforehand is not optional.

Access, on the other hand, is no longer free. The PLATYPUS attack (IEEE S&P 2021) demonstrated a purely software power side channel through RAPL, with CVE-2020-8694 and CVE-2020-8695 attached. The Linux mitigation arrived in commit 949dd0104c49, included in 5.10.0-rc4, which changed the default permissions so that only root can read energy_uj. The microcode mitigation associated with SGX goes further and introduces random noise into the reported energy and changes the reporting frequency: on machines with that mitigation active, RAPL readings are deliberately degraded. Measuring CPU and DRAM from a container therefore requires root, a udev rule relaxing the permissions, or a privileged daemon.

The software stack

Tool	Source of the GPU figure	Counter or power	Default interval	Overhead
Zeus	NVML	Counter if the architecture is Volta or later; polling otherwise	Delimited by windows	Less than 10 ms per call
dcgm-exporter	NVML via DCGM	Both fields available	`collect-interval` = 30,000 ms	5-10 W extra on the server’s IPMI reading
CodeCarbon	NVML	Counter	`measure_power_secs` = 15 s	5.38 % to 46.75 % of time at 1 kHz
Scaphandre	No GPU support; RAPL	`energy_uj` counter	Not documented	3.81 % to 28.38 % at 1 kHz
Kepler	RAPL and IPMI; GPU only as an experimental option	Hardware counters	—	—

Four details that change the choice:

Zeus decides in code with nvmlDeviceGetArchitecture and uses the accumulated counter on Volta or later, with a nestable window API (begin_window / end_window). It is the reference implementation of what this sheet recommends. Its NSDI'23 paper also documents energy reductions of 23.8 % to 75.7 % from choosing batch size and power limit well, and of 3.0 % to 31.5 % from touching only the limit, with five seconds of profiling per point enough for stable results.
Kepler has removed eBPF. The CNCF announcement of June 2026 describes the redesign towards reading /proc and /sys, motivated by the CAP_BPF and CAP_SYSADMIN permissions that blocked deployments, and by the loss of short-lived processes that underestimated the footprint. The new node metric closely follows the IPMI pattern and removes the spurious multi-kilowatt spikes of the previous version. For GPUs it remains an experimental option: today it is not the tool for measuring Wh/token on an accelerator.
CodeCarbon uses the NVML counter, which is correct, but its RAM estimate is a heuristic of 5 W per module derived from the total gigabytes, and the code itself acknowledges it.
DCGM delivers profiling metrics at 1 Hz by default, and warns that collecting at higher frequencies returns zeros because it groups metrics internally.

On the cost of measuring, the empirical overhead study of RAPL-based tools, all forced to 1 kHz, leaves a clear ordering:

Tool	Time overhead
CodeCarbon	5.38 % to 46.75 %
Scaphandre	3.81 % to 28.38 %
Turbostat	2.73 % to 14.37 %
PowerJoular	1.67 % to 8.88 %
perf	−1.00 % to 4.26 %
RAPL read in user space	−0.70 % to 2.93 %
RAPL read in kernel space	−0.17 % to 0.99 %

The underlying difference is that a system call costs of the order of 1.36·10⁻³ ms and an rdmsr instruction between 2.3·10⁻⁴ and 5.6·10⁻⁴ ms, an order of magnitude less. The authors recommend matching the granularity to the phenomenon being measured instead of raising the default frequency, and limiting polling to the necessary domains.

Procedure for making a Wh/token auditable

Fix and record the electrical state. Note EnforcedPowerLimit, the default limit and the driver version; lock clocks or the power ceiling if the experiment requires it.
Warm up to a stable temperature. RAPL thermal drift is 10 % to 12 % between 37 °C and 74 °C, and the power.draw.average transient takes up to a second to settle.
Measure the idle baseline twice: without a CUDA context and with the model loaded. They are two different values and both are needed.
Use windows of at least 5 s or 32 consecutive iterations.
Repeat over 4 separate trials with a random delay, and add 8 evenly spaced delays if you are integrating instantaneous power with a window smaller than the period.
Prefer the accumulated counter whenever the architecture is Volta or later, watching for driver reloads.
Report dispersion. The Oxford paper itself publishes the mean and standard deviation of the error; without an interval, a Wh/token figure is not auditable.
Declare the measurement boundary: GPU, node or utility feed. Without that label, two figures from the same facility differ by a factor of 2.3 with nobody knowing why.

Subtracting idle, or not

There is no consensus, and the honest approach is to treat it as a declared decision. In production the GPU is parked with the model loaded, and that consumption shows up on the bill: subtracting it erases a real cost. To compare the marginal efficiency of two models or two kernels, by contrast, subtracting it isolates the variable. The workable way out is to report both figures with labels: gross Wh/token, which includes idle and corresponds to the bill, and marginal Wh/token, comparable between models. What does not work is mixing them in the same table without distinguishing them.

One estimation trap remains alive in many spreadsheets: multiplying TDP by time overestimates real energy by up to 4.1 times, according to the ML.ENERGY Benchmark.

Power capping: what it costs and what it saves

Configuration	Energy saving	Performance loss
V100 at 200 W against 300 W (67 % of TDP), BERT training	~15 %	barely any degradation
V100 at 200 W, set of workloads	10-20 %	less than 5 %
V100 at 100 W (33 % of TDP)	40-60 %	30-40 %
A100 at 175 W against 400 W (44 % of TDP), LLaMA 65B inference	22-24 %	5-8 %

The asymmetry between phases explains the result. The Microsoft measurement at ASPLOS'24 documents that the prompt phase is short and reaches or exceeds TDP, while the token generation phase is longer and consumes less: the power ceiling penalises TTFT far more than TPOT. In their production cluster, inference uses 79 % of peak power against 97 % for training, and their measurements show that up to 20 % of power can be recovered with less than a 7 % performance loss.

The published sweep from 200 W to 700 W on the H100 and H200 adds the nuance of where the knee is: between 500 W and 700 W each 100 W step only contributes around 10 % of performance on compute-bound loads, while the H100 holds peak memory bandwidth even at 200 W. The most stable operating zones identified are 400 W on the H100 and 500 W on the H200. That the memory subsystem is protected against the trim is the physical reason why the decode phase of an LLM, which is memory-bound, loses so little performance under a power ceiling.

Two warnings about those figures: the H100/H200 sweep was measured with nvidia-smi polling every 10 s, exactly the problem described above, although on long steady loads the bias averages out; and there is no publication isolating the effect of memory clock locking on Wh/token in LLM inference, so any concrete figure on that point has to be your own measurement.

Reference figures

GPU	TDP
H100 SXM	up to 700 W, configurable
H100 NVL	350-400 W, configurable
A100 80GB SXM	400 W
A100 80GB PCIe	300 W
RTX 5090	575 W, with a recommended 1000 W supply
HGX node of 8× H100 SXM	10.2 kW nominal

Average power measured in inference: on an HPC cluster with vLLM on A100, the aggregate power of the GPU set ranged between 999 W and 2,983 W depending on model and parallelism, with Nemotron 70B on 8× A100 at 2,983 W, about 373 W per GPU, around 93 % of TDP.

Energy per token, with the warning that published figures measure different boundaries and cannot be mixed without normalising by GPU count and output tokens:

Source	Figure
TokenPowerBench, H100 94 GB	40 J per token on a standard load, more than 60 J per token in high-throughput configurations
TokenPowerBench, batch effect	−25 % of energy per token between batch 32 and 256 on a 70B model
TokenPowerBench, context effect	from 2K to 10K tokens of context on Llama 3 70B: energy ×3
TokenPowerBench, quantisation effect	Llama 3 405B in FP8 against FP16: −30 % of energy per token
TokenPowerBench, engine effect	TensorRT-LLM and vLLM against the Transformers engine: −25 % to −40 %
Per-prompt measurement in batch	0.0074-0.0289 Wh on 7B to 14B models; 0.0835-0.6912 Wh on 70B to 405B; single unbatched requests, between 10 and 100 times more expensive

Cross-links from the energy track

C2 — Energy per token: methodology and the electricity market: what to do with the joule once it is measured properly, and how the electricity price multiplies it.
C3 — Measuring energy in production: Kepler, DCGM and the practical stack: the continuous deployment of the stack whose metrology this sheet documents.
C3 — Energy benchmarking: frameworks, metrics and the state of the art: the full catalogue of tools and their metrics.
C4 — MLPerf Power: the certified alternative, measuring at the wall with an external analyser instead of the card’s sensor.
C6 — From watt to carbon: PUE, grid and grams of CO₂: the chain that turns the measured watt into emissions and into euros.
C7 — Energy efficiency levers: the quantified catalogue where the power ceiling of this sheet is just one more lever.

Sources

NVIDIA, NVML API Reference, Device Queries — https://docs.nvidia.com/deploy/nvml-api/group__nvmlDeviceQueries.html
NVIDIA, DCGM Library API, Field Identifiers — https://docs.nvidia.com/datacenter/dcgm/3.1/dcgm-api/dcgm-api-field-ids.html
NVIDIA, DCGM Feature Overview — https://docs.nvidia.com/datacenter/dcgm/latest/user-guide/feature-overview.html
NVIDIA, dcgm-exporter — https://github.com/NVIDIA/dcgm-exporter
NVIDIA, H100 Tensor Core GPU, especificaciones — https://www.nvidia.com/en-us/data-center/h100/
NVIDIA, A100 Tensor Core GPU, especificaciones — https://www.nvidia.com/en-us/data-center/a100/
NVIDIA, GeForce RTX 5090, especificaciones — https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5090/
Yang, Adámek y Armour (Oxford), Part-time Power Measurements: nvidia-smi’s Lack of Attention / SC24 — https://arxiv.org/html/2312.02741v3
Yang, GPU_Power_Benchmark, microbenchmark del trabajo anterior — https://github.com/JimZeyuYang/GPU_Power_Benchmark
ML.ENERGY, Measuring GPU Energy: Best Practices — https://ml.energy/blog/energy/measurement/measuring-gpu-energy-best-practices/
You et al., Zeus: Understanding and Optimizing GPU Energy Consumption of DNN Training, NSDI'23 — https://www.usenix.org/system/files/nsdi23-you.pdf
Zeus, Measuring Energy — https://ml.energy/zeus/measure/
ML.ENERGY, The ML.ENERGY Benchmark — https://arxiv.org/html/2505.06371v1
CNCF, Kepler, re-architected: improved power accuracy (junio 2026) — https://www.cncf.io/blog/2026/06/30/kepler-re-architected-improved-power-accuracy-and-a-community-call-to-action/
CodeCarbon, How Power Estimation Works — https://docs.codecarbon.io/latest/explanation/power-estimation/
Hubblo, Scaphandre: cálculo del consumo por proceso — https://github.com/hubblo-org/scaphandre/blob/main/docs_src/explanations/how-scaph-computes-per-process-power-consumption.md
Linux Kernel, Power Capping Framework (powercap sysfs) — https://docs.kernel.org/power/powercap/powercap.html
Intel, Running Average Power Limit Energy Reporting, INTEL-SA-00389 — https://www.intel.com/content/www/us/en/developer/articles/technical/software-security-guidance/advisory-guidance/running-average-power-limit-energy-reporting.html
Lipp et al., PLATYPUS: With Great Power comes Great Leakage, IEEE S&P 2021 — https://platypusattack.com/
Khan et al., RAPL in Action: Experiences in Using RAPL for Power Measurements, ACM TOMPECS — https://www.devsustainability.com/p/paper-notes-rapl-in-action
What Is the Cost of Energy Monitoring? An Empirical Study on the Overhead of RAPL-Based Tools — https://arxiv.org/html/2604.26815v1
Patel et al. (Microsoft), Characterizing Power Management Opportunities for LLMs in the Cloud, ASPLOS'24 — https://www.microsoft.com/en-us/research/wp-content/uploads/2024/03/GPU_Power_ASPLOS_24.pdf
MIT Lincoln Laboratory, Sustainable Supercomputing for AI: GPU Power Capping at HPC Scale — https://arxiv.org/html/2402.18593
Architectural Trade-offs in the Energy-Efficient Era: power-capping NVIDIA H100 and H200 — https://arxiv.org/html/2604.11391v1
Empirically-Calibrated H100 Node Power Models for Datacenter Energy Analysis — https://arxiv.org/pdf/2506.14551
The Model Parking Tax: Quantifying the Hidden Energy Cost of Always-On GPU Model Deployment — https://arxiv.org/html/2605.23918
The Energy Cost of Execution-Idle in GPU Clusters — https://arxiv.org/html/2604.04745
TokenPowerBench: Benchmarking the Power Consumption of LLM Inference — https://arxiv.org/pdf/2512.03024
From Prompts to Power: Measuring the Energy Footprint of LLM Inference — https://arxiv.org/html/2511.05597
Serving LLMs in HPC Clusters — https://arxiv.org/abs/2507.00418
Uptime Institute, Global Data Center Survey 2025 — https://datacenter.uptimeinstitute.com/rs/711-RIA-145/images/2025.Annual.Survey.Report.pdf

Model chain of trust (1/4): KServe and the Open Inference Protocol, the control plane your serving is missing

Sun, 26 Jul 2026 20:00:00 +0200

This blog has devoted a good part of its 145 articles to what happens inside the inference process and another good part to what happens around it. What is missing is a thread that cuts across all of that and answers a question that takes thirty seconds to ask in an audit and does not get answered in three weeks in production: when the endpoint returns a token, where did that model come from, who signed it, and who proves that the binary running on the GPU is the one that was approved?

This series walks that chain backwards from the only visible point, the endpoint: the control plane and the API contract (this article), the registry and the distribution of the bytes (2/4), the signing and provenance that let you trust them (3/4), and the identity and isolation of whoever runs them (4/4). All of it with projects under the governance of CNCF, LF AI & Data or the Linux Foundation: a chain of trust that depends on a proprietary product is no better than a support contract.

TL;DR

A vLLM Deployment serves tokens perfectly well. What it does not give you is a contract: no model versioning, no rollout by percentage, no scale-to-zero, and no common surface for the thirty non-LLM artifacts of an inference factory.
KServe provides the vocabulary: InferenceService, ServingRuntime/ClusterServingRuntime, storage initializer and storageUri. It has been CNCF incubating since 29 September 2025, after seven years as KFServing and a spell in LF AI & Data. The cadence has accelerated: v0.18.0 on 29 April 2026 and v0.19.0 on 14 June 2026.
The LLM branch is recent and different: the LLMInferenceService CRD (serving.kserve.io/v1alpha1) arrived in v0.16 in November 2025, on top of llm-d, with spec.prefill, spec.worker, spec.parallelism and spec.router.
The Open Inference Protocol (V2) is the contract: /v2/health/*, /v2/models/<m>, /v2/models/<m>/infer, in REST and gRPC, implemented by Triton, OpenVINO Model Server, Seldon MLServer and AMD Inference Server. It is tensor-centric, and the LLM world standardised de facto on the OpenAI API: KServe serves both, with the OpenAI ones under the openai/ prefix.
The honest position: the portability of an LLM today comes from the OpenAI API, not from V2. Accepting the dual track is correct; pretending there is a single standard is not.
KServe is overkill if you serve one model on one node. It starts to pay off from the fifth heterogeneous artifact onwards, and it competes on a different axis from the LLM operators: contract and lifecycle versus engine and topology.

The analogy: the station master and the track gauge

An inference factory is like a railway station with several lines.

The station master does not drive any train. His job is the service timetable: which formation leaves from which platform, how many units are coupled each hour, when an engine is pulled out for maintenance and, when new rolling stock arrives, how to divert ten per cent of the passengers to the test train before committing the whole service. That is the control plane. KServe does not generate a single token, but it decides what runs, with what configuration, with how many replicas and with what traffic split. The trains are the engines (vLLM, SGLang, TensorRT-LLM, Triton, TEI), and a competent station master does not marry a manufacturer.

And then there is the track gauge: as long as everybody builds to the same one, any locomotive rolls on any line and changing supplier is a commercial decision, not a civil engineering project. That gauge is the Open Inference Protocol, the contract that makes a client written against Triton work against OpenVINO Model Server without touching a line.

The analogy has a twist that needs no explanation in Spain: the agreed gauge is not the gauge the new rolling stock runs on. The conventional network was built to Iberian gauge and the high-speed one to standard gauge, and the practical solution was not to convert the network but to install gauge changers at the boundary points. The same thing has happened in inference: V2 is the agreed gauge and the OpenAI API is the gauge everything generative runs on. What sustains interoperability is the changer, not a single standard. We will come back to it.

A `Deployment` serves tokens, but offers no contract

In vLLM on Kubernetes we built a service with Deployment, Service, HPA and an initContainer that pulls the weights down. It works, and for one model on one node it is the right answer. What it does not give you, and cannot give you without reinventing it every time:

A stable API contract. The endpoint is “whatever the image you put there exposes”: moving from vLLM to TensorRT-LLM changes routes, error format and metric names, and the clients find out in production.

Versioning with semantics. The model is a path inside a volume; knowing which version a pod is serving requires kubectl exec. There is no declarative field that says “this service serves artifact X” and can be audited from outside.

Progressive model rollout. A RollingUpdate replaces pods; it does not split traffic between two versions by percentage. For that you need an L7 gateway in front, like the one in canary, blue-green and shadow, or a control plane that emits it.

Scale-to-zero. The HPA does not go below one replica: for a model queried twelve times a day, that is an 80 GB GPU tied up twenty-four hours.

And, above all, a common surface for what is not an LLM. A real factory does not serve three LLMs: it serves two or three LLMs, four embedding models, a couple of rerankers, ticket classifiers, a PII detector, an OCR, industrial vision and three scikit-learn artifacts that nobody remembers training. Fifteen or thirty artifacts, each with its own runtime and its own way of saying “I am ready”. The cost is not in serving the first one: it is in making sure the thirtieth does not cost thirty times what the first one did.

KServe: from KFServing to a CNCF incubating project

KServe was born in 2019 as KFServing, inside Kubeflow, in a collaboration between Google, IBM, Bloomberg, NVIDIA and Seldon. In February 2022 it was donated to the LF AI & Data Foundation; in September 2022 it split from Kubeflow and was renamed. The relevant jump: on 29 September 2025 the CNCF accepted it directly at incubating level, without going through sandbox, with a public announcement on 11 November. The figures it declared then, more than 300 contributors, 19 maintainers, more than 30 adopting organisations (Bloomberg, Red Hat, Cloudera, Nutanix, SAP, NVIDIA), come from an interested party, but the list is verifiable and the 625 contributing organisations on the CNCF dashboard support the thesis of governance not captured by a vendor.

On cadence you have to be precise, because the cliché of “KServe is slow” no longer describes reality:

Version	Date	What it brought
v0.15.0	31 March 2025	`LocalModelCache`, multi-node inference, OpenAI-compatible embeddings API, Gateway API in raw mode, KEDA integration
v0.16.0	November 2025	`LLMInferenceService`; rename of `RawDeployment` to `Standard` and of `Serverless` to `KNative`; progressive rollout in raw mode
v0.18.0	29 April 2026	WVA + KEDA/HPA autoscaling for LLMISvc, `/v1/responses` route (OpenAI Responses API), `restricted` Pod Security Standards by default on LLMISvc, namespace-scoped `ModelCache`
v0.19.0	14 June 2026	Routing by model name, static LoRA adapters, balancing across heterogeneous GPUs, dual REST/gRPC routing in Standard mode, automatic migration to `InferencePool` v1

Two major releases in under two months, with the bulk of the changelog in the LLM branch. The project is not slow: it has pivoted from classic predictive serving to generative serving. The consequences run in both directions.

The negative one is called ModelMesh. It was the high-density mode (hundreds of small models sharing processes, with dynamic load and unload) and it was, on paper, the natural answer to the problem of thirty heterogeneous artifacts. Its last release was v0.12.0 on 9 July 2023, and the kserve/modelmesh-serving repository was archived read-only on 14 April 2026. Red Hat published specific guidance for migrating to Standard mode. If somebody proposes ModelMesh in a design today, it is a sign that the documentation they read is three years old.

Anatomy of the control plane

`InferenceService`: predictor, transformer, explainer

The core CRD is InferenceService (serving.kserve.io/v1beta1), and its contribution, the one that became the common vocabulary of the field, is to break an inference service down into three components each with its own contract:

predictor: mandatory. It declares modelFormat, optionally runtime, the storageUri the weights come from and the protocolVersion it exposes.
transformer: optional. Pre- and post-processing. That it is a separate component, rather than code inside the server, is what allows it to be scaled separately (it is usually CPU-bound while the predictor is GPU-bound) and versioned separately.
explainer: optional, and the one that has aged worst: the V2 protocol does not support the explain endpoint, which only exists in V1. Anyone who needs explainability with a standard contract has a real gap here.

All three share ComponentExtensionSpec: minReplicas, maxReplicas, scaleTarget, scaleMetric, containerConcurrency, timeout and canaryTrafficPercent.

`ServingRuntime` and `ClusterServingRuntime`: the catalogue of trains

A ServingRuntime is a pod template that knows how to serve one or several formats. It declares supportedModelFormats (with name, version, autoSelect and priority), protocolVersions (v1, v2) and containers, with args parameterised by templates in the {{.Name}} style that are substituted with metadata from the InferenceService.

Selection is what makes this scale. If the InferenceService names a runtime, the controller looks for it first in the namespace and then at cluster level; if it does not name one, it picks among those with autoSelect: true for that modelFormat, breaking ties by priority. That is what lets a platform publish a curated catalogue (“here, huggingface is served with this vLLM and these flags”) without the product teams having to know what --gpu-memory-utilization is. In multi-tenant setups, the ClusterServingRuntimes belong to the platform and the InferenceServices to the teams: the cleanest separation-of-duties line KServe offers.

The storage initializer and `storageUri`

The storageUri field triggers the storage initializer: an init container that downloads the artifact to /mnt/models before the server starts. Schemes supported as of July 2026: s3://, gs://, https://, pvc://, hf://, oci:// and git repositories; credentials are attached via a ServiceAccount with annotated secrets. That oci:// connects with the second article of the series: the weights can live in the same registry as the images, with the same access control and the same signing capability.

The deployment modes

This is where nearly everything operational is decided. It is selected with the serving.kserve.io/deploymentMode annotation.

	KNative (formerly Serverless)	Standard (formerly RawDeployment)	ModelMesh
Resources emitted	Knative `Service` + `Revision`, `VirtualService` with Istio	`Deployment`, `Service`, `HPA`, Gateway API `HTTPRoute`	Its own high-density runtime
Dependencies	Knative Serving + Istio (or an alternative gateway)	Kubernetes 1.32+, cert-manager 1.15.0+, Gateway API v1.2.1 and a controller	Its own components
Autoscaling	KPA (concurrency / RPS)	HPA or KEDA	—
Scale-to-zero	Yes, with an activator that holds the request	No for HTTP requests	—
`canaryTrafficPercent`	Yes, by revision weights in Istio	Silently ignored (issue #5335, open since 2 April 2026)	—
Sidecars in the pod	`queue-proxy` (+ `istio-proxy`)	None	—
Status as of July 2026	Current	Current and the default mode of the LLM branch	Archived on 14 April 2026

That table has an uncomfortable reading: the two most quoted KServe capabilities, scale-to-zero and canary by percentage, live in the mode that drags Knative and Istio along, while the mode the project has pivoted towards is the other one.

The LLM branch: `LLMInferenceService`

Since v0.16 KServe has a separate CRD for LLMs: LLMInferenceService, in serving.kserve.io/v1alpha1. It is not an extension of the classic InferenceService and does not share its API version, and confusing the two is the first mistake made by anyone arriving from old documentation.

It is built on llm-d, the project that Red Hat, Google, IBM, CoreWeave and NVIDIA donated to the CNCF. The division of roles the project itself declares is clean: KServe is the control plane (lifecycle, scaling and operational governance) and llm-d contributes distributed scheduling, with GPU utilisation, queue depth, cache residency and SLA. In the analogy: llm-d decides which line each train takes; KServe decides which trains exist and when they are withdrawn.

The fields that matter: spec.model (URI, invocation name, scheduling criticality and LoRA adapters, with static adapter reconciliation from v0.19, which connects with multi-LoRA serving); spec.template (single node, or the decode pool); spec.worker (multi-node, which triggers LeaderWorkerSet); spec.prefill (the disaggregation we analysed in disaggregated serving turned into a declarative field); spec.parallelism (tensor, data and expert parallelism); and spec.router (gateway, route, scheduler), where KServe leans on the Gateway API Inference Extension, v1.5.0 of 19 April 2026, already GA, with automatic migration to InferencePool v1 from v0.19.

The endpoint it exposes is OpenAI-compatible: /v1/chat/completions with streaming, and since v0.18 also /v1/responses. That detail underpins the next section: the LLM branch of KServe does not speak V2, it speaks OpenAI.

For multi-node there is the kserve-huggingfaceserver-multinode runtime, based on vLLM over Ray, with workerSpec.tensorParallelSize and workerSpec.pipelineParallelSize. Its restrictions are hard: it requires a ReadWriteMany PVC, it only works in Standard mode, it does not support autoscaling and it requires exactly one head pod. A reference 4×H100 SXM node with NVLink needs none of this, because a 70B fits with tensor parallel 4 on a single node; you need it from there upwards.

The Open Inference Protocol: the contract itself

The routes and the payload

The Open Inference Protocol (historically the “V2 Inference Protocol”) lives in kserve/open-inference-protocol, with REST and gRPC specifications and versioning under SemVer 2.0. Its HTTP surface is deliberately small:

Resource	Verb	Route
Server metadata	GET	`/v2`
Liveness	GET	`/v2/health/live`
Server readiness	GET	`/v2/health/ready`
Model metadata	GET	`/v2/models/<model>[/versions/<v>]`
Model readiness	GET	`/v2/models/<model>[/versions/<v>]/ready`
Inference	POST	`/v2/models/<model>[/versions/<v>]/infer`

The payload is tensorial: id, a list of inputs (each with name, shape, datatype and data) and optionally the desired outputs. The types are BOOL, signed and unsigned integers from 8 to 64 bits, 16-, 32- and 64-bit floating point, and BYTES. In gRPC the same contract is GRPCInferenceService, with six RPCs: ServerLive, ServerReady, ModelReady, ServerMetadata, ModelMetadata and ModelInfer. For high-volume embeddings, the serialisation difference is not cosmetic.

Who actually implements it

The specification repository declares KServe, NVIDIA Triton, Seldon MLServer and Core v2, OpenVINO Model Server, AMD Inference Server and TorchServe as adopters. Not all entries are worth the same as of July 2026:

Triton implements it in a documented way (“Triton exposes HTTP/REST and gRPC endpoints based on the standard inference protocols proposed by the KServe project”), plus its own extensions. It is the de facto reference implementation.
OpenVINO Model Server exposes the KServe API over gRPC and REST, the TensorFlow Serving one and OpenAI-compatible endpoints: the clearest example of a server that speaks both track gauges.
TorchServe forces you to be honest. Its official documentation carries the warning: “This project is no longer actively maintained. While existing releases remain available, there are no planned updates, bug fixes, new features, or security patches.” An adopter on zero maintenance is not an adopter: it is a row nobody has updated.

On the health of the standard, the most eloquent data point is not in any press release: the specification repository does not have a single published release, has accumulated something like thirty commits and seventy-five stars, and its governance consists of monthly meetings and a Slack channel. There is also a “vendor neutral” fork at open-inference/open-inference-protocol with more commits than the original and no traction whatsoever. V2 is a stable contract, widely implemented and barely evolving. For a serialisation format that can be a virtue; for a standard aspiring to cover the generative world, it is a death sentence.

The `generate` extension and why it did not win

The standard did try to cover generation: the specification contains a generate_rest.yaml with two routes, /v2/models/<m>/versions/<v>/generate and .../generate_stream, and a minimalist schema: a request with text_input and parameters (temperature, top_p, max_tokens defaulting to 20, stop, details); a response with text_output, model_name, model_version and, if requested, finish_reason and logprobs; streaming over server-sent events. It is clean, it is enough for a simple completion and nobody uses it: the KServe generative inference documentation, as of July 2026, lists four endpoints, and none of them is generate:

Task	Endpoint in KServe
Text generation	`openai/v1/completions`
Chat	`openai/v1/chat/completions`
Embeddings	`openai/v1/embeddings`
Reranking	`openai/v1/rerank`

The openai/ prefix exists, according to the documentation itself, “to avoid confusion with the V1 API protocol”, and it is changed with KSERVE_OPENAI_ROUTE_PREFIX. It is plumbing that says a lot: the in-house standard was relegated to a defensive prefix while the outside one takes all the traffic.

The tension, unvarnished

V2 solved the problem of 2021, and it solved it well. A tensorial, typed contract with metadata and health checks, in REST and gRPC, is what you need to serve a classifier, a vision model, a ranker or an embedding, and it is still the right answer for that half of the catalogue: swapping an embedding server from MLServer to OpenVINO Model Server without touching clients is the track gauge doing its job.

The LLM world standardised somewhere else, not by committee, but because the entire client ecosystem (SDKs, agent frameworks, gateways, evaluation) was written against the OpenAI API, and any engine that wanted to be used had to speak it. What that means for an architecture:

There is not one contract: there are two, and both are legitimate. Designing as if there were one does violence to half the catalogue.
LLM portability is superficial. It covers chat and completion well; as soon as you use detailed logprobs, guided decoding, priority headers, prefix caching parameters or tool calling with specific schemas, you leave the common subset and get tied back to the engine. It is the warning from structured output: what is portable is the envelope, not always the contents.
The gauge changer is the gateway. What makes the set switchable is the translation and routing layer of choosing an OSS LLM inference gateway and the router as an L7 gateway, not a single standard.
Crossing over loses the versioning. V2 has /versions/<v> in the route; the OpenAI API only has the model field. The version has to be encoded in the name (asistente-70b-2026-07) or managed in the control plane. That is the difference between being able to audit which version answered and not being able to.

Four manifests you can actually apply

Platform runtime for the blog’s reference node, 4×H100 SXM 80 GB with NVLink, with tensor parallel 4:

apiVersion: serving.kserve.io/v1alpha1
kind: ClusterServingRuntime
metadata:
 name: vllm-h100-tp4
spec:
 annotations:
 prometheus.kserve.io/port: "8080"
 prometheus.kserve.io/path: "/metrics"
 supportedModelFormats:
 - name: huggingface
 version: "1"
 autoSelect: true
 priority: 10
 protocolVersions:
 - v2
 containers:
 - name: kserve-container
 # ALWAYS pin by digest in production; the tag is only for readability
 image: vllm/vllm-openai:v0.11.0
 command: ["python3", "-m", "vllm.entrypoints.openai.api_server"]
 args:
 - --port=8080
 - --model=/mnt/models
 - --served-model-name={{.Name}}
 - --tensor-parallel-size=4
 - --max-model-len=32768
 - --gpu-memory-utilization=0.92
 - --enable-prefix-caching
 - --kv-cache-dtype=fp8
 resources:
 requests: { cpu: "16", memory: 128Gi, nvidia.com/gpu: "4" }
 limits: { cpu: "16", memory: 128Gi, nvidia.com/gpu: "4" }
 # the probe that stops the pod dying while the model loads
 startupProbe:
 httpGet: { path: /health, port: 8080 }
 periodSeconds: 15
 failureThreshold: 40 # up to 10 minutes of loading
 readinessProbe:
 httpGet: { path: /health, port: 8080 }
 periodSeconds: 10
 livenessProbe:
 httpGet: { path: /health, port: 8080 }
 periodSeconds: 30

The InferenceService that consumes it, with KEDA on vLLM metrics:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
 name: asistente-70b
 namespace: inferencia
 annotations:
 serving.kserve.io/deploymentMode: "Standard"
 serving.kserve.io/autoscalerClass: "keda"
spec:
 predictor:
 minReplicas: 1
 maxReplicas: 4
 model:
 runtime: vllm-h100-tp4
 modelFormat: { name: huggingface }
 # the OCI registry from article 2/4 of this series
 storageUri: "oci://registry.example.internal/modelos/asistente-70b:2026-07"
 autoScaling:
 metrics:
 - type: External
 external:
 metric:
 backend: "prometheus"
 serverAddress: "http://prometheus.monitoring.svc.cluster.local:9090"
 query: 'sum(vllm:num_requests_waiting{model_name="asistente-70b"})'
 target:
 type: Value
 value: "8"

The canary, with its small print:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
 name: asistente-70b
 namespace: inferencia
 annotations:
 # WATCH OUT: canaryTrafficPercent only takes effect in KNative mode.
 # In Standard mode it is silently ignored (issue kserve/kserve#5335, open).
 serving.kserve.io/deploymentMode: "KNative"
spec:
 predictor:
 canaryTrafficPercent: 10
 minReplicas: 1
 maxReplicas: 4
 model:
 runtime: vllm-h100-tp4
 modelFormat: { name: huggingface }
 storageUri: "oci://registry.example.internal/modelos/asistente-70b:2026-08"

KServe keeps three references in the status (LatestReadyRevision, LatestRolledoutRevision and PreviousRolledoutRevision): promoting means setting the percentage to 100, reverting means pinning traffic on the previous one, and a revision that does not become ready automatically receives zero traffic.

And the two calls. Against the V2 contract, for a classifier served with MLServer or Triton:

# metadata: which model it is, what inputs it expects, what types
curl -s http://clasificador.inferencia.example/v2/models/clasificador-tickets | jq .

# readiness of the specific model, not of the server
curl -s -o /dev/null -w '%{http_code}\n' \
 http://clasificador.inferencia.example/v2/models/clasificador-tickets/ready

curl -s -X POST \
 http://clasificador.inferencia.example/v2/models/clasificador-tickets/infer \
 -H 'Content-Type: application/json' \
 -d '{ "id": "req-4711",
 "inputs": [ { "name": "input-0", "shape": [1, 4], "datatype": "FP32",
 "data": [5.1, 3.5, 1.4, 0.2] } ],
 "outputs": [ { "name": "output-0" } ] }'

Against the same control plane, but on the OpenAI-compatible surface of the LLM:

curl -s -N -X POST \
 http://asistente-70b.inferencia.example/openai/v1/chat/completions \
 -H 'Content-Type: application/json' \
 -d '{ "model": "asistente-70b",
 "messages": [ {"role": "user",
 "content": "Summarise last night'\''s incident report."} ],
 "max_tokens": 256,
 "stream": true }'

Two contracts and two payload shapes, with the same InferenceService as the unit of governance. That is the value proposition of a control plane: the contract is chosen by the model and the management is single.

Autoscaling and the problem of zero

Three autoscalers coexist, and choosing the wrong one is the most common cause of disappointment. KPA, in KNative mode, scales by concurrency or RPS and is the only one that goes down to zero in a usable way, because Knative provides an activator that holds the request while the pod starts; its metric does not distinguish a 30-token request from an 8,000-token one. HPA, in Standard, scales by CPU or memory: for LLMs it is useless, because the CPU of a saturated vLLM pod and that of an idle one look too much alike. KEDA, in Standard, is the right option, the one we develop in autoscaling with KEDA: it is enabled with serving.kserve.io/autoscalerClass: "keda" and autoScaling.metrics accepts external metrics against Prometheus (vllm:num_requests_running or vllm:num_requests_waiting) or, with the OpenTelemetry add-on, pushed metrics. Since v0.18 there is also Workload Variant Autoscaling, which uses HPA or KEDA as a backend.

That leaves the serious problem, which no autoscaler solves: zero. A 70B model in fp8 is on the order of 70 GB to move into HBM, and that path, from disk to HBM, is measured in minutes: the first request after idling takes that long, or it fails.

Worse: the installation documentation is explicit as of July 2026: “Scale from Zero is currently not supported in Standard mode for HTTP requests”. The mode the project has pivoted towards, and the only one that supports KEDA, does not know how to come up from zero on an HTTP request: KEDA can go down to zero (default cooldown of 300 seconds), but the way back requires an activator that Standard does not have. Anyone who wants real zero with HTTP activation still needs Knative.

The levers, in order of effectiveness: do not scale the hot model to zero (minReplicas: 1 for anything with an SLA; zero is for the long tail); attack the load time, the only solution that is not a patch, with the techniques from speeding up cold start; use LocalModelCache, available since v0.15 and extended to LLMInferenceService in v0.19, to preload artifacts into a node PVC; and set correct probes, which is what comes next.

Operational traps

The probe that kills the pod while the model loads. The most frequent and the most expensive to diagnose, because the symptom is a CrashLoopBackOff with no error in the server logs. Until early 2026 the KServe vLLM templates used initialDelaySeconds of 120 to 300 seconds on liveness, with the result described in issue #5062 (12 February 2026, now resolved): the fast models waited too long and the large ones died before finishing loading. The fix applied: a startupProbe of up to ten minutes and probes with no initial delay. A legacy runtime has to be reviewed by hand. In KNative mode there is a worse variant (issue #3795, closed as not planned): what dies is not the model container but the queue-proxy, which does not respond OK within the deadline and takes the whole pod down with it.

canaryTrafficPercent silently ignored. It deserves its own trap because there is no error and no condition in the status to give it away: in Standard mode the field does nothing. The team believes it is serving 10 % with the new version and it is serving 100 %. While issue #5335 remains open, in Standard the canary is done with HTTPRoute weights by hand, or in the gateway.

Canary and scarce GPUs do not get along. A 10 % canary of a 70B with tensor parallel 4 costs a whole 4-GPU node, not 10 % of the resources, because the minimum granularity is a complete replica. During the window there are two versions occupying twice the hardware, or capacity has to be taken from the stable one. That is the difference between a microservice canary and a model canary, and the reason a shadow over duplicated traffic often pays off better.

storageUri and credentials. Storage initializer failures are always the same ones: a ServiceAccount without the annotated secret, a KMS-encrypted bucket with a role that has no decryption permission, an S3-compatible but not exactly S3 endpoint, or intermittent 403s, with a specific fix in v0.18. It is diagnosed in the init container logs, not in the server’s. And one standing recommendation: no downloading over https:// from the internet in production; that is precisely the problem of article 2/4.

The model size against the emptyDir. By default the artifact lands in an ephemeral volume on the node’s disk. Seventy gigabytes per replica fill a system disk with surprising ease, and then the kubelet evicts pods for disk pressure, including ones that have nothing to do with it. You have to set sizeLimit or use a dedicated pvc://. For multi-node the restriction is harder, with a mandatory ReadWriteMany PVC, which rules out a good part of local block storage and pushes you towards shared file systems, with the consequences covered in storage for AI.

The dependency on Knative and Istio. The real cost of KNative mode: two more control planes to version and correlate with Kubernetes, and two sidecars, queue-proxy and istio-proxy, in every pod, with their CPU and memory consumption and their network hop per request. On a platform that already has a mesh it is a marginal cost; on a cluster dedicated to inference it has to be justified by the capabilities you are actually going to use.

Observability: the name of the metric. The ServingRuntime declares prometheus.kserve.io/port and /path annotations, but the metrics that come out are the engine’s, not KServe’s: vllm:num_requests_waiting, vllm:time_to_first_token_seconds, vllm:gpu_cache_usage_perc. The control plane does not normalise names across runtimes, so a dashboard written against vLLM is no use for Triton, and correlating with DCGM is still your own work. v0.19 raises the HPA/KEDA scaling status into the service conditions and emits events on readiness transitions: exactly what was needed to alert on the control plane and not only on the engine.

Decision map: when KServe, and when it is overkill

The useful comparison is not “KServe versus OME, vLLM Production Stack, Dynamo or llm-d”. Those four are covered in LLM inference operators and compete on a different axis:

Axis	What gets decided	Who competes
Contract and control plane	Which CRD describes a service, which API clients see, how the model is versioned, how the rollout is done, how LLM and non-LLM coexist	KServe, Seldon Core v2
Engine and topology	Which engine runs, how prefill/decode is disaggregated, how routing is done with cache awareness, how multi-node tensor parallel is done	vLLM Production Stack, OME, NVIDIA Dynamo, llm-d

The axes are orthogonal, and LLMInferenceService proves it: KServe did not reimplement distributed scheduling, it leaned on llm-d. Choosing KServe is not choosing instead of an LLM operator: it is choosing the contract layer that operator lives on.

Choose KServe when the catalogue is heterogeneous, LLMs plus embeddings, rerankers, classifiers and vision, which is its decisive advantage and the one none of the four operators has: a service like the one in TEI in production fits into the same vocabulary as the LLM. Also when you need a contract decoupled from the engine because you have clients you do not control; when you need declarative, auditable governance (which artifact, from which origin and with which runtime, in an object versionable in git and subject to admission policies), the piece that makes the rest of this series possible; and when you want a catalogue of platform runtimes separated from the teams’ services.

KServe is overkill when you serve one model, on one node, with up to three replicas: the Deployment from vLLM on Kubernetes is the right answer. When your workload is LLM only and the priority is performance per GPU, because there the value is in the engine. When you are going to use neither scale-to-zero nor canary by revision, because what remains is a Deployment with more YAML. And when you already have an LLM operator and only serve LLMs: putting KServe on top for completeness is adding CRDs without adding decisions.

For an inference factory

First: separate the runtime catalogue from the service catalogue, and do it on day one. The ClusterServingRuntimes belong to the platform; the InferenceServices, to the teams. That boundary turns “serving the thirtieth model” into a five-line change instead of a negotiation about vLLM flags, and it fits with GitOps with Flux: the runtime is reviewed as platform code, the service as product configuration.

Second: accept the two contracts and put the gauge changer where it belongs. V2 for the tensorial world, the OpenAI API for the generative one, and the translation in the gateway, not in every client. And since the OpenAI API does not version in the route, encode the version in the model name and make that name the same identifier that appears in the registry, in the signature and in the traces. That is what makes a request traceable back to the artifact.

Third: measure start-up before promising elasticity. The time from the pod being scheduled to the readiness probe going green, with the real model and from the real origin, decides whether scale-to-zero is a FinOps lever or an incident machine. That number, and not the price of the GPU-hour, is the one to take into the cost conversation.

And here this piece falls short on purpose. KServe gives you the contract, the lifecycle and the rollout, but its whole trust model hangs off a text string: storageUri. The storage initializer downloads whatever is at that destination and puts it in front of the GPU without asking anything else; nothing in the InferenceService guarantees that the artifact is the one that was approved, that it has not changed since the last audit, or that the origin is legitimate. The next article goes there: where the model bytes come from, and why an OCI registry with ORAS, Harbor, MLflow or the Kubeflow Model Registry is the difference between distributing models and scattering them.

Sources

CNCF, KServe becomes a CNCF incubating project (11 de noviembre de 2025) — https://www.cncf.io/blog/2025/11/11/kserve-becomes-a-cncf-incubating-project/
CNCF, KServe project page (incubating, aceptado el 29 de septiembre de 2025) — https://www.cncf.io/projects/kserve/
CNCF TOC, Project Moving Levels Checklist: KServe joining CNCF at Incubation level — https://github.com/cncf/toc/issues/1905
KServe, Releases (v0.15.0, v0.16.0, v0.18.0, v0.19.0) — https://github.com/kserve/kserve/releases
KServe, Inference Protocol V2 (Open Inference Protocol) — https://kserve.github.io/website/docs/concepts/architecture/data-plane/v2-protocol
KServe, open-inference-protocol (especificación, adoptantes, gobernanza) — https://github.com/kserve/open-inference-protocol
KServe, open-inference-protocol — generate_rest.yaml — https://github.com/kserve/open-inference-protocol/blob/main/specification/protocol/generate_rest.yaml
open-inference, Vendor neutral fork of kserve/open-inference-protocol — https://github.com/open-inference/open-inference-protocol
KServe, Generative Inference — Runtime Overview (endpoints con prefijo openai/) — https://kserve.github.io/website/docs/model-serving/generative-inference/overview
KServe, Understanding LLMInferenceService — https://kserve.github.io/website/docs/model-serving/generative-inference/llmisvc/llmisvc-overview
KServe, Cloud-Native AI Inference at Scale using KServe and llm-d (5 de marzo de 2026) — https://kserve.github.io/website/blog/cloud-native-ai-inference-kserve-llm-d
KServe, Serving Runtime (ServingRuntime y ClusterServingRuntime) — https://kserve.github.io/website/docs/concepts/resources/servingruntime
KServe, Kubernetes Deployment Installation Guide (K8s 1.32+, cert-manager 1.15.0+, Gateway API v1.2.1, sin scale-from-zero en Standard) — https://kserve.github.io/website/docs/admin-guide/kubernetes-deployment
KServe, Canary Rollout Strategy — https://kserve.github.io/website/docs/model-serving/predictive-inference/rollout-strategies/canary
KServe, Autoscaling with KEDA — https://kserve.github.io/website/docs/model-serving/predictive-inference/autoscaling/keda-autoscaler
KServe, Multi-node/Multi-GPU Inference (PVC RWX, sin autoescalado) — https://kserve.github.io/website/docs/model-serving/generative-inference/multi-node
KServe, Storage Options for Model Artifacts — https://kserve.github.io/website/docs/model-serving/storage/overview
KServe, issue #5335 — Support canaryTrafficPercent for RawDeployment mode via Gateway API HTTPRoute weights (abierta, 2 de abril de 2026) — https://github.com/kserve/kserve/issues/5335
KServe, issue #5062 — Add startupProbe to vLLM main containers (12 de febrero de 2026) — https://github.com/kserve/kserve/issues/5062
KServe, modelmesh-serving (última release v0.12.0 de julio de 2023; repositorio archivado el 14 de abril de 2026) — https://github.com/kserve/modelmesh-serving
Red Hat, Converting ModelMesh and Serverless InferenceServices to RawDeployment (Standard) Mode — https://access.redhat.com/articles/7134025
NVIDIA, Triton Inference Server — Inference Protocols and APIs — https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/customization_guide/inference_protocols.html
OpenVINO, What is OpenVINO Model Server (API KServe, TFS y endpoints compatibles con OpenAI) — https://docs.openvino.ai/2026/model-server/ovms_what_is_openvino_model_server.html
PyTorch, TorchServe — Notice: Limited Maintenance — https://docs.pytorch.org/serve/
Kubernetes SIG Network, Gateway API Inference Extension (v1.5.0, 19 de abril de 2026) — https://github.com/kubernetes-sigs/gateway-api-inference-extension

Model chain of trust (2/4): where the bytes live — registry, OCI artefacts and distribution to the GPU node

Sun, 26 Jul 2026 19:00:00 +0200

In the first article in the series we described the control plane: who declares that an inference endpoint exists, who reconciles it and what contract the clients speak. But an InferenceService is no more than a promise until somebody puts 140 GB of weights on the filesystem the container sees. This second article is about exactly that: where those bytes come from and by what route they reach the GPU’s memory.

It is a subject the blog has touched around the edges, in the cold start series, in data versioning with DVC and lakeFS, in the catalogue of OSS LLMOps tools, but never head on. It is time to, because almost every on-premise inference platform that gets audited shares the same original defect, and it is conceptual before it is about performance: they conflate three things that live in different places and fail in different ways.

TL;DR

Three planes, not one. Model registry (metadata, versions, lineage, promotion) ≠ artefact repository (the bytes, addressed by digest) ≠ deployment storage (what the pod mounts at /mnt/models). Mixing them is what breaks reproducibility.
The anti-pattern: a Hugging Face token in the pod and a download from the internet on every start-up. No immutability, no cache, no audit trail, no air-gap and cold starts measured in minutes.
The OCI artefact is today’s pragmatic answer: OCI Image Spec 1.1 (March 2024) brought artifactType, subject and the Referrers API; ORAS v1.3.0 (October 2025) added backup/restore for air-gap; ModelPack (CNCF sandbox since June 2025) standardises the mediaTypes for weights.
Kubernetes already knows how to mount OCI artefacts: the image volume source (KEP-4639) reached GA in v1.36 (22 April 2026). It is the most operationally relevant change of the year for model distribution.
The real bottleneck is almost never the GPU. 140 GB over a saturated 10 Gb/s link is about 112 s; over Gen4 NVMe about 20 s; over PCIe Gen5 about 2-3 s. Optimising the loader without fixing the network is optimising the wrong link.
A model registry (MLflow, Kubeflow Hub) does not store the bytes: it stores the record card and a pointer. If you were expecting it to solve your distribution, you bought the wrong tool.

The analogy

The simile is a serious historical archive, with three things a distracted visitor constantly confuses.

The catalogue: record cards with a shelfmark, author, date, provenance, conservation state and consultation restrictions. The card weighs grams and describes something that weighs kilos. The repository: the compact shelving in the basement where the physical boxes are, with access control, temperature control and an inventory that balances to the gram; it knows nothing about why a document matters, but it guarantees that box 4711 still contains exactly what it contained. And the reading room table: where the researcher has the document open in front of them. A temporary and expensive place, from which the document goes back to the repository when the session ends.

The model registry is the catalogue. The artefact repository is the repository. The GPU node with the model in HBM is the reading room table. Anyone who says “we have set up a model registry” and what they have set up is MLflow has set up the catalogue and left the repository unbuilt: the boxes are still in somebody’s car boot. Putting the model inside the container image is the equivalent of putting the document inside the table: every time the table changes, the document moves. And anyone who downloads from the internet on every start-up has no repository: they order the document by courier from another city every time a researcher sits down.

We will come back to the archive later, with the weeding that deletes boxes that are still referenced, the trolley that carries things between the basement and the reading room, and the certified reproduction sent to the isolated site.

Three planes that are not the same

Before discussing tools you have to fix the vocabulary, because a good part of the architecture discussions at this layer are really discussions about nomenclature.

Plane	What it stores	Unit	Question it answers	OSS examples
Model registry	Metadata, logical versions, lineage, promotion state, evaluation metrics	The model’s record card (KB)	Which version is approved for production and where did it come from?	MLflow, Kubeflow Hub, ClearML
Artefact repository	The bytes, addressed by digest, immutable	The blob (GB-TB)	What exactly are the bytes of that version?	Harbor, `distribution`, Zot, MinIO, lakeFS
Deployment storage	The copy the pod sees mounted	The volume (`/mnt/models`)	What is the inference process reading right now?	PVC, image volume, emptyDir + NVMe cache

The three layers have different life cycles. A record card lives forever (audit); a blob, for as long as some version references it (retention); a volume, for as long as the pod lives. Implemented as a single thing, it inherits the worst of the three: if the registry is also the deployment storage, deleting an old version takes down a pod in production; if the deployment storage is also the canonical reference, there is no way to answer what was served last Tuesday.

Hence the operational rule that gets written on the wall: the source of truth for the bytes is a digest, not a tag and not a path. Everything else (tags, aliases, PVC paths) is a mutable pointer that resolves to that digest at a given moment. Article 3/4 builds all of its provenance verification on that premise.

The anti-pattern we start from

This is, by a distance, the most widespread pattern in on-premise deployments that started as a pilot and stayed that way:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: vllm-anti-patron
spec:
 template:
 spec:
 containers:
 - name: vllm
 image: vllm/vllm-openai:v0.11.0
 args: ["--model", "meta-llama/Llama-3.3-70B-Instruct"]
 env:
 - name: HF_TOKEN
 valueFrom:
 secretKeyRef:
 name: hf-token
 key: token
 resources:
 limits:
 nvidia.com/gpu: "4"

It works and it serves tokens. And it is unacceptable on a sovereign platform for five independent reasons, any one of which would be enough:

An external dependency on the critical start-up path. A Deployment that cannot recover from a failure without outbound internet access carries a third-party dependency that nobody has declared in the SLA.
Zero immutability. meta-llama/Llama-3.3-70B-Instruct is a mutable reference, and a repository’s default branch changes (tokenizer fixes, chat templates). Two replicas of the same Deployment started a week apart can serve different bytes, and nothing in the cluster says so.
Zero coordinated cache and zero audit trail. Every pod downloads its own copy: eight replicas of a 140 GB model is 1.1 TB of outbound traffic per rolling update. And no internal record remains of what was downloaded, with which digest or who authorised it; faced with an ENS or ISO 42001 auditor, the answer is “we got it off the internet”.
Incompatible with air-gap. The whole design collapses when the platform has to be replicated in an isolated environment, which is the defence and healthcare case dealt with in regulatory compliance infrastructure.
Cold start measured in minutes. Downloading 140 GB over a 1 Gb/s outbound link (125 MB/s effective, being generous) is about 19 minutes of transfer alone, before touching the disk or the GPU; with 10 Gb/s dedicated and saturated, about 112 seconds. In a load-driven scaling event, that is service latency.

The blog’s cold start series explores how to trim the disk-to-HBM leg. What this article adds is that if the first leg is the internet, the rest of the optimisation is irrelevant.

The model as an OCI artefact

The pragmatic answer is boring, and that is why it works: the model is stored in the same registry where the containers are already stored. There is already access control, cross-site replication, scanning, quotas, audit and a team that knows how to operate it.

What OCI 1.1 made possible

For years, putting things that were not container images into an OCI registry was a bodge. The OCI Image Spec 1.1 and Distribution Spec 1.1 specifications, published on 13 March 2024, turned it into a first-class use case with three pieces:

artifactType in the manifest: it explicitly declares “this is not a runnable image, it is an artefact of type X”. Previously this was encoded by abusing the config mediaType, which is why even today there are registries that show a model as a broken image.
subject: a manifest can declare that it refers to another manifest. That is what lets you hang a signature, an SBOM or an attestation off a model without modifying it and without changing its digest.
Referrers API (GET /v2/<repo>/referrers/<digest>): the reverse query, “give me everything that points at this digest”, with a tag-based fallback for registries that do not implement it yet.

Article 3/4 builds signing, provenance and AIBOM on top of these three pieces. Without them, verifying a model forces you to invent a convention of your own.

ORAS: the generic client

ORAS (OCI Registry As Storage) is the Swiss army knife for getting arbitrary artefacts in and out of an OCI registry. Version v1.3.0 was published on 6 October 2025, conformant with distribution-spec v1.1.1, and added three things relevant to models: backup/restore to a directory or tarball (key for air-gap), management of multi-platform indexes and --format for structured output.

One data point to calibrate maturity: ORAS entered the CNCF sandbox on 13 July 2021 and as of this article is still in sandbox. Five years at the entry level does not invalidate the tool, which is stable and the de facto reference, but it says something about the project’s governance that is worth weighing before turning it into a critical dependency with no plan B.

Pushing a model with ORAS, with explicit content types:

oras push registro.interno/modelos/llama-3.3-70b:2026.07.1 \
 --artifact-type application/vnd.cncf.model.manifest.v1+json \
 --annotation "org.opencontainers.image.created=2026-07-20T09:00:00Z" \
 --annotation "es.ejemplo.modelo.precision=bf16" \
 --annotation "es.ejemplo.modelo.origen=hf:meta-llama/Llama-3.3-70B-Instruct" \
 config.json:application/vnd.cncf.model.weight.config.v1.raw \
 tokenizer.json:application/vnd.cncf.model.weight.config.v1.raw \
 model-00001-of-00030.safetensors:application/vnd.cncf.model.weight.v1.raw \
 model-00002-of-00030.safetensors:application/vnd.cncf.model.weight.v1.raw

# Retrieve it, with a local blob cache so nothing already present is re-downloaded
export ORAS_CACHE=/var/cache/oras
oras pull registro.interno/modelos/llama-3.3-70b:2026.07.1 --output /srv/modelos/llama-70b
oras manifest fetch registro.interno/modelos/llama-3.3-70b:2026.07.1 --descriptor

ModelPack: the missing convention

ORAS lets you invent the mediaTypes, with the predictable result that every organisation invents its own and nothing interoperates. ModelPack is the attempt to standardise them, accepted into the CNCF sandbox in June 2025. On top of OCI Image Manifest 1.1 it defines:

artifactType: application/vnd.cncf.model.manifest.v1+json
config: application/vnd.cncf.model.config.v1+json
weight layers: application/vnd.cncf.model.weight.v1.raw (and the .tar, .tar+gzip, .tar+zstd variants)
weight configuration, documentation, code and dataset layers following the same pattern

Its reference CLI is modctl, which works with a declarative Modelfile:

NAME llama-3.3-70b-instruct
ARCH transformer
FAMILY llama
FORMAT safetensors
PARAMSIZE 70
PRECISION bf16
CONFIG config.json
CONFIG tokenizer.json
MODEL *.safetensors
DOC *.md

modctl build -t registro.interno/modelos/llama-3.3-70b:2026.07.1 -f Modelfile .
modctl push registro.interno/modelos/llama-3.3-70b:2026.07.1
modctl pull registro.interno/modelos/llama-3.3-70b:2026.07.1 \
 --extract-dir /srv/modelos/llama-70b --extract-from-remote

The alternative with more traction among users is KitOps (ModelKit), with Docker-like ergonomics and native integration with Hugging Face. As of this article there is no convergence: ModelPack aims at the enterprise case and at integration with runtimes and registries; KitOps and Docker Model Runner, at developer ergonomics. Picking one today is a bet; using artifactType and explicit annotations instead of home-grown conventions is not.

How a 140 GB model is sliced up

A large model is 20-40 safetensors shards plus half a dozen small configuration files, and the layering strategy matters:

One layer per shard, uncompressed. Weights in bf16 or fp8 are incompressible in practice: gzip over safetensors burns CPU to save 2-3 %, and forces decompression on the node, adding a full pass through CPU and disk on the critical path. The right type is .raw.
The small files in their own layer. config.json, tokenizer.json and the templates change far more often than the weights. If they go in the same layer as a 5 GB shard, every tokenizer fix invalidates 5 GB of cache on every node.
Layer size between 2 and 8 GB. Large enough not to pay per-blob overhead, small enough to parallelise and so that a network failure does not force you to retry 40 GB.

In practice three things hurt. The upload uses the distribution-spec’s chunked upload, with historically uneven interoperability between implementations, so pushing 140 GB is an operation measured in tens of minutes that demands a decent link and retries. Deduplication is by layer digest, so two versions that differ in one shard share the rest only if those blobs are byte-for-byte identical, and any repackaging that alters metadata breaks it. And the default values of reverse proxies (timeouts, maximum body size, upload session lifetime) are calibrated for 1 GB images, not for 8 GB blobs.

Harbor as the on-premise repository

Harbor is the reference OSS registry for on-premise: a CNCF graduated project since June 2020, with a healthy release cycle (the 2.15 series came out on 20 March 2026 and patch 2.15.2 on 2 July 2026) and support for the last three minors for about nine months each. What matters for models:

Cross-site replication. Push or pull rules, filtered by repository, tag and resource type, with manual, scheduled or event-driven triggering. For models, scheduled and within a window, with a label marking which versions get replicated (see traps).

Proxy cache. Here you have to be precise, because a lot of misinformation circulates: Harbor’s proxy cache works against OCI registries (Harbor, Docker Hub, a distribution registry, ECR, ACR, GCR, Quay and GHCR) and not against Hugging Face, which does not expose an OCI distribution API. If the idea was “I put Harbor in front of Hugging Face and that is it”, it does not work that way. The real options are (a) an ingestion task that pulls from Hugging Face in a staging environment and pushes to Harbor as an OCI artefact, the recommended pattern because it introduces an explicit control point, or (b) a P2P accelerator with native support for model repositories, such as Dragonfly. What the proxy cache does solve is mirroring the runtime base images (vLLM, TGI, TEI), which is no small thing: it creates a 7-day retention by default and the project is read-only.

Per-project quotas. In bytes, via the UI or the API:

curl -u admin:REDACTED -X PUT "https://registro.interno/api/v2.0/quotas/7" \
 -H "Content-Type: application/json" \
 -d '{"hard": {"storage": 21990232555520}}'

Those 21990232555520 bytes are 20 TiB, tight sizing, not generous, for a dozen models with three live versions each. Two warnings: the quota is checked when the manifest arrives, that is after uploading the blobs, so a push that exceeds it has already consumed disk and network; and the accounting does not always reflect deduplication of blobs shared between projects.

Immutable tags. Per-project rules that prevent overwriting a tag matching a pattern. For models it is not optional: a rule making everything under modelos/** with tag pattern 2* (date-based versions) immutable eliminates the entire class of “the tag changed underneath us” incidents.

Retention and GC. Per-project retention rules (last N versions, or those from the last N days) that mark artefacts for deletion, and a separate garbage collector that frees the unreferenced blobs. They are two different things, and deleting in the UI does not free disk: the weeding runs separately. Running GC with a concurrent push in flight is the classic recipe for deleting blobs that were about to be referenced.

Scanning. Trivy is integrated, with one caveat: scanning a weights artefact detects nothing relevant about the model. It is useful for the runtime images, which do execute code. Detecting dangerous formats, a PyTorch pickle with arbitrary code instead of safetensors, requires specific tools, and that is the subject of article 3/4.

Model registries proper

Now the catalogue, with the uncomfortable claim up front: a model registry does not solve distribution.

MLflow

MLflow (LF AI & Data) is the de facto catalogue standard, with a mature 3.x series; 3.13.0 was published in June 2026, focusing on RBAC, an administration UI and trace retention. Its data model: registered model (logical name) → model version (incrementing integer) → aliases and tags.

The important conceptual change: stages (Staging, Production, Archived) have been deprecated since MLflow 2.9.0 and are replaced by aliases and tags, because a fixed enumeration is too inflexible to express real MLOps flows. An alias is a named mutable pointer:

from mlflow import MlflowClient

client = MlflowClient()
client.set_registered_model_alias("llama-3.3-70b-soberano", "champion", 7)
client.set_model_version_tag("llama-3.3-70b-soberano", "7",
 "estado_validacion", "aprobado_ens_medio")
client.set_model_version_tag("llama-3.3-70b-soberano", "7",
 "oci_digest", "sha256:9f2c...")

Consumption happens through the alias URI: models:/llama-3.3-70b-soberano@champion.

What MLflow gives you: lineage back to the run that produced the model, evaluation metrics attached to the version, a shared promotion language and an API to hang approvals off. What it does not give you: it is not a CDN and it does not manage distribution to nodes, and its artifact store (S3/MinIO/NFS) provides neither immutability by digest, nor Referrers, nor cross-site replication. The healthy pattern is the one in the snippet: the catalogue stores the OCI digest as a tag on the version and points at the repository, instead of trying to be the repository.

Kubeflow Hub (formerly Model Registry)

The Kubeflow Model Registry was renamed Kubeflow Hub and unified two functions: the registry itself and a federated Model Catalog that discovers external models (YAML, Hugging Face), with lineage across data, code and models, metadata and state-based promotion.

The honest assessment: as of this article the component is still in the 0.3.x series and marked Alpha under Kubeflow’s versioning policy, with limited support. Interesting if you already run the full Kubeflow and want a single console; it is not yet the piece on which to build model governance for a sovereign platform in production. The rename in the middle of 0.3.x is, moreover, a sign that the API is still moving.

Comparison table by function

Function	OCI registry (Harbor + ORAS/ModelPack)	MLflow Registry	Kubeflow Hub	lakeFS / DVC
Stores the bytes	Yes, by digest, immutable	No (pointer to an artifact store)	No (pointer)	Yes (data and versions)
Lineage to data/code	Only via annotations	Yes, to the run	Yes, explicit	Yes, for datasets
Rich metadata	Annotations and JSON config	Yes, typed	Yes, typed	Partial
Promotion / approval	By convention (tags, labels)	Aliases and tags	States	No
Signing and attestations	Yes, native (`subject` + Referrers)	No	No	No
Cross-site replication	Yes, native	No	No	Partial
Standard API	OCI Distribution 1.1	Its own REST	Its own REST	S3/Git-like
Air-gap	Yes (`oras backup`/`restore`, `skopeo`)	Manual	Manual	Manual
Maintainer	CNCF (Harbor graduated; ORAS and ModelPack sandbox)	LF AI & Data	Kubeflow (CNCF)	Commercial OSS
Real maturity	High (registry), medium (model convention)	High	Alpha	High

The thesis of the table: the columns are complementary, not alternatives. The OCI registry is the repository; MLflow, the catalogue; lakeFS or DVC version the dataset that everything originates from, as detailed in data versioning with DVC and lakeFS.

The hot path: from the registry to HBM

Image volumes: the change of the year

The image volume source (KEP-4639) mounts an OCI image or artefact directly as a read-only volume in a pod: no initContainer, no copying, no PVC. Its track record: alpha in v1.31 (August 2024), beta in v1.33 (April 2025) with subPath support, beta by default in v1.35 and GA in v1.36, released on 22 April 2026.

apiVersion: v1
kind: Pod
metadata:
 name: vllm-modelo-oci
spec:
 containers:
 - name: vllm
 image: registro.interno/runtime/vllm:v0.11.0
 args: ["--model", "/mnt/models", "--served-model-name", "llama-3.3-70b"]
 volumeMounts:
 - name: pesos
 mountPath: /mnt/models
 readOnly: true
 resources:
 limits:
 nvidia.com/gpu: "4"
 volumes:
 - name: pesos
 image:
 reference: registro.interno/modelos/llama-3.3-70b@sha256:9f2c...
 pullPolicy: IfNotPresent

Two details that change operations: the reference is by digest, so immutability stops depending on anyone’s discipline; and pullPolicy: IfNotPresent makes the node’s runtime cache the blobs in its local store, so the second replica on that node starts without touching the network. It is the coordinated cache the anti-pattern was missing.

KServe: `storageUri` and modelcars

In KServe there are two routes: the classic one, storageUri with a storage initializer that copies from S3, PVC, HTTP or GCS into an emptyDir; and the modelcar, with the oci:// scheme:

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
 name: llama-70b-soberano
spec:
 predictor:
 model:
 modelFormat:
 name: huggingface
 storageUri: oci://registro.interno/modelos/llama-3.3-70b:2026.07.1
 resources:
 limits:
 nvidia.com/gpu: "4"

You have to understand the modelcar’s internal mechanism before depending on it: KServe sets shareProcessNamespace: true on the pod, starts a sidecar container with the model image and creates a symbolic link from /mnt/models to that process’s root filesystem through /proc. The runtime reads the weights without copying them, which is the saving being sought. Two warnings: it is not enabled by default (you have to turn on enableModelcar in the inferenceservice-config ConfigMap) and the latest tag forces pullPolicy: Always, cancelling out the cache. With image volumes now GA, the modelcar remains the route for clusters that are not yet on 1.36.

Loading fast: where each technique attacks

You have to separate two problems that get confused all the time: getting the bytes to the node (network) and getting them into HBM (disk → CPU → PCIe → GPU).

For the first, Dragonfly is the mature piece: it graduated in CNCF on 14 January 2026, distributes weights at the scale of hundreds of terabytes to hundreds of nodes “in minutes”, cuts the bandwidth consumed at the origin by up to 90 %, and makes use of between 70 % and 80 % of each node’s bandwidth. Its v2.5.0 (30 June 2026) added direct download from model repositories (dfget hf://...), P2P acceleration of Git LFS and a dragonfly-injector that injects P2P capability via a webhook without rebuilding images. Its subproject Nydus supplies the lazy-loading image format. There is a very common misplaced expectation here: for model weights lazy loading helps little, because an inference engine reads all the weights almost immediately; the real benefit is for fat runtimes and for models where only part is read.

For the second, the numbers published by the NVIDIA Run:ai Model Streamer on Llama-3-8B (15 GB, safetensors, a node with one A10G) are the cleanest reference available:

Source	Safetensors loader	Run:ai Model Streamer	Tensorizer
SSD gp3	47.99 s	14.34 s (concurrency 16)	16.11 s (16 workers)
SSD io2	47.0 s	7.53 s (concurrency 8)	10.36 s (8 workers)
S3 object storage	not supported	4.88 s (concurrency 32)	37.36 s (16 workers)

And the total time until vLLM is ready to serve: 66.13 s with the standard loader over gp3 versus 35.08 s with the streamer. Two critical readings. First: these are the project’s own benchmarks, not an independent measurement, on a 15 GB model on a modest GPU; extrapolating linearly to 140 GB on 4×H100 is a leap nobody has published. Second: the gain comes from saturating the storage medium with concurrency. If the bottleneck is an NFS at 1 Gb/s, no loader fixes it. The detail of this leg is in speeding up cold start with tensorizer and in from disk to HBM.

As the diagram shows, on a reference node of 4×H100 SXM 80 GB with NVLink each leg is between 5 and 10 times faster than the previous one. The conclusion is boring: the money is in having the model already on the node’s NVMe before the pod starts.

LoRA adapters: when the artefact weighs megabytes

Everything above assumes artefacts of tens or hundreds of gigabytes. A LoRA adapter breaks that assumption: for a 70B base, a rank-16 adapter over the attention projections weighs tens or a few hundred megabytes, three orders of magnitude less, and that changes the whole strategy.

With artefacts like that, the local cache and P2P stop mattering: the download is instant at any reasonable bandwidth. What starts to matter is cardinality and rate of change. A multi-tenant platform like the one described in multi-LoRA serving can have hundreds of adapters, each with its owner and its access policy, over a handful of bases. The practical consequences:

The adapter is an OCI artefact in its own right, with its own repository and an immutable tag. But the registry then has thousands of small repositories instead of tens of large ones, which stresses metadata, listings and granular RBAC, not disk.
It must declare its base. An adapter with no immutable reference to the digest of the base model it was trained on is a time bomb: applied over another version, it degrades quality silently. It is the natural use case for subject: the adapter refers to the base, and the Referrers API answers “which adapters exist for this digest”.
Loading is dynamic, not at start-up. vLLM allows hot loading of adapters with VLLM_ALLOW_RUNTIME_LORA_UPDATING and the /v1/load_lora_adapter endpoint, or declaratively with the LoRA resolver plugins (VLLM_PLUGIN_LORA_RESOLVERS), which resolve an adapter by name from a directory. The clean pattern on Kubernetes is to mount an image volume with the adapter and let the filesystem resolver discover it.
Versioning becomes the dominant problem. With 300 live adapters, knowing which adapter serves tenant X, over which base and trained with which dataset, only has an answer if the catalogue exists. Here a model registry does beat convention over tags.

Air-gap and sovereignty: promotion without losing traceability

The strict sovereign case: a connected environment where you ingest and validate, and an isolated environment where you serve, with no network route between them. Promotion crosses a data diode or a physical medium, and traceability has to survive the journey. It is the archive’s certified reproduction: the document, its record card and the seal certifying that the copy is faithful all travel together.

# 1. In the connected environment: export the artefact with EVERYTHING that hangs off it
oras backup \
 --output /transfer/llama-3.3-70b-2026.07.1.tar \
 --include-referrers \
 registro-dmz.interno/modelos/llama-3.3-70b:2026.07.1

# 2. Verify the exact digest before crossing
oras manifest fetch --descriptor \
 registro-dmz.interno/modelos/llama-3.3-70b:2026.07.1 > /transfer/descriptor.json
sha256sum /transfer/llama-3.3-70b-2026.07.1.tar > /transfer/SHA256SUMS

# 3. Physical crossing or data diode

# 4. In the isolated environment: restore preserving digests and referrers
oras restore \
 --input /transfer/llama-3.3-70b-2026.07.1.tar \
 registro-aislado.interno/modelos/llama-3.3-70b

The --include-referrers flag is what makes this worth anything: without it the weights cross, but the signature, the AIBOM and the attestations stay behind, and the isolated environment receives a model indistinguishable from one downloaded by hand. oras backup is marked experimental in v1.3.0, which has to be accepted explicitly before turning it into a procedure; the consolidated alternative for pure copying is skopeo copy with the oci-archive format, at the cost of managing the referrers separately.

What must travel attached to the artefact so that article 3/4 can verify it: the digest of the artefact and of its blobs; the provenance of the origin (the repository and the specific revision, not the model’s name, plus who ingested it and when); the identity of the process that packaged it (pipeline, commit, run), the raw material for an SLSA attestation; the cross-references to the evaluation dataset and to the base model if it is a derivative; and the promotion state with its approver, which in an isolated environment cannot be queried against the connected environment’s MLflow. All of it fits in OCI annotations and in artefacts referred to via subject. The rule: the isolated environment must be able to verify without calling anyone; if verification needs an external query, it is not air-gap.

Decision map

An OCI registry and a convention are enough when you have fewer than twenty models or so, a single team decides what gets promoted, lineage back to training is not a regulatory requirement (typical if you consume third-party models) and you already operate Harbor. Here a model registry adds one more console to maintain and no answer that immutable tags and annotations were not already giving. It is the majority case in pure inference.

A model registry is needed if you train or fine-tune and need auditable lineage from model to dataset and to code, the continuous fine-tuning cycle; if several teams compete to promote versions and an explicit approval flow is needed; if you have to answer an ISO 42001 or EU AI Act auditor about which version was in production on a given date; or if you manage tens of adapters with different owners.

The registry becomes valueless bureaucracy when the model is registered after being deployed (the catalogue as an administrative act, not as a gate); when the promotion state is not connected to any technical control and an unapproved version can be deployed; or when a model registry and an OCI registry coexist with two truths and no digest joining them. The last one is the most common failure and the most expensive: two versioning systems that diverge in silence.

Operational traps

Mutable tags and latest in production. Beyond style, a mutable tag turns any audit into guesswork and lets two replicas of the same Deployment serve different bytes. On top of that, in KServe latest forces pullPolicy: Always and cancels the node cache, so you pay in cold start what you lose in traceability. Rule: immutable tags by project policy and deployment by digest.

GC that deletes referenced layers. The archive’s weeding. The garbage collector identifies blobs that no manifest references, and the window between “I have uploaded the blobs” and “I have uploaded the manifest” is exactly where a concurrent GC can take them away. In Harbor this has historically shown up in _uploads and in discrepancies between the space the UI claims to have freed and the real figure. Mitigation: GC in a window with pushes blocked, and do not trust the first pass.

The registry’s disk and the cost of versioning. Brutal arithmetic: 140 GB per version with three live versions is 420 GB; twelve models like that, 5 TB; with six-version retention and two replicated sites, 20 TB. And it is expensive-class storage if the registry lives on replicated block. Deduplication helps little, because the weights change completely between versions. The retention policy has to be designed before filling the registry, and models must be separated from runtime images by project, because their retentions have nothing to do with each other.

Replication that saturates the inter-site link. An event-driven rule on a model repository moves 140 GB every time someone pushes a version: on a shared 1 Gb/s link, almost twenty minutes flat out in the middle of the working day. Schedule it in a window, limit bandwidth, filter by label which versions get replicated and use P2P inside each site.

Putting the model inside the runtime image. Tempting because “that way there is only one artefact”. The result is a 145 GB image that has to be rebuilt and redistributed every time a runtime CVE is patched, with the security life cycle coupled to the model’s. Separated: a small, patchable runtime and a large, stable model.

For an inference factory

Three actionable things for anyone running an on-premise inference factory, not a training one.

First: cut the internet dependency at start-up, this week. You do not need a platform project: an ingestion task that pulls each model once, pushes it to Harbor as an OCI artefact with an immutable tag and provenance annotations, plus switching the deployments to reference by digest, is enough. The HF_TOKEN disappears from the pod and moves to the ingestion pipeline, which is where it makes sense and where it fits with secrets hardening.

Second: decide consciously whether you need a catalogue. If you only consume third-party models and do not train, probably not: the OCI registry with a convention gives 90 % of the value at 10 % of the operational cost. If you train, fine-tune or manage per-tenant adapters, set up MLflow and store the OCI digest as a tag on each version, so that catalogue and repository cannot diverge.

Third: move the bottleneck to the right place. Before investing in load accelerators, measure the three legs. If the model arrives over the network on every start-up, the work is a local NVMe cache, image volumes with pullPolicy: IfNotPresent and, with many nodes, P2P. Only when the model is already on the node’s disk does it make sense to wrestle with loader concurrency.

So much for where the bytes come from and by what route they arrive. What remains is what we have taken for granted all along: why trust those bytes. A digest guarantees integrity, that nobody has changed them, but not provenance: it does not say who produced them, with what data or whether somebody with authority certified that they could be served. The third article in the series builds that answer with Sigstore, SLSA, in-toto and AIBOM, hanging precisely off the subject and the Referrers API that we have left ready here.

Sources

Open Container Initiative, OCI Image and Distribution Specs v1.1 Releases (13 March 2024) — https://opencontainers.org/posts/blog/2024-03-13-image-and-distribution-1-1/
CNCF, Announcing ORAS v1.3.0: Elevating artifact and registry management workflows (6 October 2025) — https://www.cncf.io/blog/2025/10/06/announcing-oras-v1-3-0-elevating-artifact-and-registry-management-workflows/
CNCF, ORAS project page (sandbox since 13 July 2021) — https://www.cncf.io/projects/oras/
ORAS, oras backup (experimental) — https://oras.land/docs/commands/oras_backup
ModelPack, CNCF ModelPack Specification — model spec (artifactType and mediaTypes) — https://github.com/modelpack/model-spec
ModelPack, modctl — getting started (Modelfile and commands) — https://github.com/modelpack/modctl/blob/main/docs/getting-started.md
CNCF, Cloud Native Computing Foundation announces Harbor graduation (23 June 2020) — https://www.cncf.io/announcements/2020/06/23/cloud-native-computing-foundation-announces-harbor-graduation/
Harbor, Configure proxy cache (supported registries) — https://goharbor.io/docs/2.6.0/administration/configure-proxy-cache/
OneUptime, How to configure Harbor project quotas for storage limits (9 February 2026) — https://oneuptime.com/blog/post/2026-02-09-harbor-project-quotas-storage/view
endoflife.date, Harbor releases (2.15.2 on 2 July 2026; 2.15 on 20 March 2026) — https://endoflife.date/harbor
VMware Cloud Foundation Blog, Using Harbor as an AI Model Registry (3 March 2026) — https://blogs.vmware.com/cloud-foundation/2026/03/03/using-harbor-as-an-ai-model-registry/
MLflow, Model Registry workflows — stages deprecated since MLflow 2.9.0 — https://mlflow.org/docs/latest/ml/model-registry/workflow/
MLflow, Release 3.13.0 highlights — https://mlflow.org/releases/3.13.0/
Kubeflow, Hub (formerly Model Registry) — overview — https://www.kubeflow.org/docs/components/hub/overview/
kubernetes/enhancements, KEP-4639: OCI VolumeSource (alpha 1.31, beta 1.33/1.35, GA 1.36) — https://github.com/kubernetes/enhancements/blob/master/keps/sig-node/4639-oci-volume-source/README.md
PerfectScale, Kubernetes v1.36 release: OCI artifact volume support reaches GA (22 April 2026) — https://www.perfectscale.io/blog/kubernetes-v1-36-release
KServe, OCI storage (modelcar) — https://kserve.github.io/website/docs/model-serving/storage/providers/oci
CNCF, CNCF announces Dragonfly’s graduation (14 January 2026) — https://www.cncf.io/announcements/2026/01/14/cloud-native-computing-foundation-announces-dragonflys-graduation/
CNCF, Dragonfly v2.5.0 is released (30 June 2026) — https://www.cncf.io/blog/2026/06/30/dragonfly-v2-5-0-is-released/
NVIDIA Run:ai, Model Streamer benchmarks — https://github.com/run-ai/runai-model-streamer/blob/master/docs/src/benchmarks.md
vLLM, LoRA adapters (dynamic loading and resolver plugins) — https://docs.vllm.ai/en/stable/features/lora/

Model chain of trust (3/4): signature, provenance and AIBOM — why you trust those bytes

Sun, 26 Jul 2026 18:00:00 +0200

In the first article of this series we built the serving control plane; in the second, where the bytes come from. At the end of that route there is an initContainer that runs oras pull on a 140 GB artefact and leaves it on a volume vLLM is going to read. The uncomfortable question is the one that gives this third article its title: why you trust those bytes.

The usual answer (“because they come from our registry”) is the one the supply-chain discipline has spent a decade dismantling: a registry is a warehouse, not an authority; it keeps whatever is pushed to it, and whoever can push can poison. Here we cover the four pieces that turn that sentence into something a machine verifies before the pod starts: signing (Sigstore), model signing (OpenSSF Model Signing), provenance (SLSA and in-toto) and a bill of materials (AIBOM). And the part that almost never gets told: what each one proves.

TL;DR

Model artefact: third-party code with access to your data. pickle executes code in torch.load; safetensors removes that class of attack, but says nothing about whether the weights are poisoned.
Sigstore is the mature piece: cosign 3.1.1 (9 June 2026), Rekor v2 GA since 10 October 2025. In air-gap, your own PKI or a private Sigstore; the new bundle allows offline verification.
OpenSSF Model Signing (OMS) v1 (June 2025) signs a manifest of hashes for the whole directory, because a servable model is a directory and not a file; the reference library, model_signing 1.1.1, is in the OpenSSF sandbox.
SLSA v1.2 (24 November 2025) adds the Source track; Build L3 is the realistic target in fine-tuning. in-toto graduated in the CNCF on 23 April 2025.
AIBOM: SPDX 3.0.1 and CycloneDX 1.7 (ECMA-424 2nd ed.) have the fields; the tooling that fills them in automatically barely exists. It is emitted from the pipeline, not from a scanner.
Regulation: Art. 53 of the AI Act binds GPAI providers from 2 August 2025, and Regulation (EU) 2026/1744 pushed high risk back to December 2027 and August 2028 without touching GPAI. The CRA requires an SBOM, not an AIBOM, with full application from 11 December 2027.
Limit: none of this says the model is good. Ultralytics produced valid attestations of artefacts with a cryptominer inside.

The analogy: batch traceability

A batch of ham arriving at a supermarket’s loading bay brings four things that are easy to confuse. A batch number, which identifies it and no other: the sha256. A seal from the abattoir, which if broken gives away tampering in transit: the signature. A health register saying which abattoir it came from, on what date and under whose inspection: the provenance attestation. And an ingredients and allergens label: the AIBOM. At the bay, the supervisor does not accept the pallet if the seal is broken or the register is missing, and that is admission control.

The analogy holds down to the detail: the seal travels stuck on the outside and not inside the ham, because it is a detached signature; the register is issued by the abattoir and not by the carrier, so its value depends on the inspection being real. What lets you recall product when a problem shows up is the batch number, not the seal.

But it serves above all for what it does not promise: a ham with an intact seal and an impeccable label can be off. Traceability answers “where did it come from and did anybody touch it”; it does not answer “is it good”. This article is about the first question; the second is the job of evaluation and of guardrails, and confusing the two is what leads to treating as safe a model you only know the origin of.

The threat model, without generalities

`pickle`, and what safetensors solves

The problem with pickle is not a bug, it is its semantics: an object can define __reduce__ and on deserialisation Python executes whatever that method returns. A torch.save checkpoint is a program that on loading can open a reverse shell. This is not theoretical: in February 2024 JFrog documented around a hundred malicious models on Hugging Face with a real payload (PyTorch with an injected __reduce__, Keras abusing the Lambda layer), and in February 2025 ReversingLabs described nullifAI, two models with deliberately corrupted pickle files that evaded Picklescan by exploiting the fact that the payload runs before the file fails. The response came late: PyTorch 2.6 flipped the default value of weights_only to True, restricting deserialisation to an allowlist of types. It broke a lot of code (there are open issues in nnUNet, accelerate and half a dozen more projects) and it is not a formal guarantee: that allowlist has had documented escapes.

safetensors does solve the whole class: pure data, a JSON header with offsets plus a blob of tensors, with no execution possible. Trail of Bits’ audit for EleutherAI and Hugging Face, on 23 May 2023, found no critical flaw leading to arbitrary code execution, and as a bonus the direct memory mapping gives loads on the order of a hundred times faster on CPU, something we already exploited in the route from disk to HBM.

The nuance that is systematically forgotten: safetensors guarantees that loading the file does not execute code, and guarantees nothing about the numbers inside. A model with a trained backdoor, normal except in the face of a specific trigger, is distributed in safetensors and sails past the scanners and the signature. It is the sealed ham that is off.

LoRA adapters, mutable tags and Friday’s model

An adapter weighs megabytes, gets shared casually and in a multi-LoRA stack dozens are hot-loaded on the same base: it is the cheapest vector for the attacker and the one with the least governance. The 2026 literature is dense on detection: arXiv 2602.15195, revised in April 2026, reports a ROC-AUC of 1.00 classifying adapters by spectral statistics of the attention projections. But with a cool head these are adapters poisoned by the authors themselves with a known method: live research, not deployable defence. The realistic operational answer is still where the adapter came from and who signed it.

And two threats with no attacker. A tag is mutable: mi-registro/llama-70b:produccion can point to one digest today and another tomorrow without a line of your manifests changing, and one well-meaning oras push is enough; the mitigation is identical to the one for images in stack hardening, pinning by digest and never by tag, with the detail that the digest is what the signature covers. And the classic: an engineer converts a checkpoint on their laptop, runs oras push with their credentials and on Monday there is an InferenceService serving it, with no signature, provenance or AIBOM. It is not malicious; it is ungovernable. Here the signature helps because it makes the shortcut unworkable.

Ultralytics: valid signature, poisoned artefact

On 4 and 5 December 2024, versions 8.3.41 and 8.3.42 of ultralytics (YOLO) were published on PyPI with a cryptominer, through poisoning of the GitHub Actions cache in the publishing workflow; days later a second round arrived (8.3.45 and 8.3.46) with API tokens that had not been rotated when migrating to Trusted Publishing. All four malicious versions carried valid attestations, because the workflow itself generated them. The signature was correct and so was the provenance; what was compromised was the build environment. As the researcher who uncovered it summed up, an attestation guarantees the relationship between an artefact and a build job plus a commit, not between the developer’s intentions and the final artefact.

Sigstore: who signed, and how you check without calling anybody

Sigstore is three pieces and a client. Fulcio is a CA that, in exchange for a valid OIDC token, issues a very short-lived X.509 certificate bound to that identity. Rekor is an append-only transparency log where the signature is recorded with its timestamp. cosign orchestrates it. The result is keyless signing: instead of a long-lived private key to look after, there is an OIDC identity (the CI workflow) and an immutable record that this identity signed that digest.

Status as of July 2026: cosign 3.1.1, from 9 June 2026. The 3.x branch, since 8 October 2025, enabled the new bundle format by default, along with --trusted-root and --use-signing-config; the practical consequence is that the bundle carries the verification material inside and allows verification without calling Rekor or Fulcio. Rekor v2 has been GA since 10 October 2025: a reimplementation over tiles that makes running your own log far cheaper, because Trillian log server and log signer are switched off and reads are cached in a CDN, at the price of cutting down to two entry types (hashedrekord and dsse) and going without a search index. It requires cosign 3.0.1+ or 2.6.0+; v1 continues in parallel and its freeze will be announced a year in advance.

Signing and verifying a model artefact in OCI

On the artefact from the second article, the operation is identical to that for an image: for the registry, a model artefact is an OCI object with its digest.

# Resolve the tag to a digest ONCE, in the pipeline. Sign the digest, never the tag.
oras resolve registro.interno/modelos/llama-70b-fin:v7
# -> sha256:9f2a4c1e77b0d3a8...

# Keyless signature from CI (the OIDC token comes from the runner)
cosign sign --yes registro.interno/modelos/llama-70b-fin@sha256:9f2a4c1e...

# Verification: exact workflow identity, exact issuer
cosign verify \
 --certificate-identity-regexp '^https://git.interno/plataforma/modelos/.*@refs/heads/main' \
 --certificate-oidc-issuer 'https://git.interno' \
 registro.interno/modelos/llama-70b-fin@sha256:9f2a4c1e...

The most frequent mistake is omitting --certificate-identity and --certificate-oidc-issuer, or setting them so loosely that they accept any identity from the provider. Without those two flags narrowed down, verification checks that somebody signed, not that the right party signed: the difference between “it has a seal” and “it has our abattoir’s seal”.

The air-gap problem

Keyless needs a reachable OIDC when signing and, in the classic model, a reachable Rekor when verifying. In a disconnected datacenter, three strategies:

Strategy	What you gain	What it costs
Own key / PKI (`cosign sign --key`)	Works without a network; fits HSM and corporate PKI	Custody and rotation of long-lived keys comes back; without a transparency log there is no detection of unauthorised signing
Private Sigstore (internal Fulcio + Rekor v2 + OIDC)	Full keyless and real transparency inside the perimeter	Three more services to operate, with their TUF root and their rotation cycle
New bundle + offline verification	Verification without a network using `--offline` and a local `--trusted-root`	Transporting and keeping the trust root up to date by sneakernet

cosign verify --offline=true \
 --trusted-root /etc/sigstore/trusted_root.json \
 --certificate-identity-regexp '^https://git.interno/plataforma/modelos/.*' \
 --certificate-oidc-issuer 'https://git.interno' \
 registro.interno/modelos/llama-70b-fin@sha256:9f2a4c1e...

Two warnings. --insecure-ignore-tlog exists and gets used a lot in air-gap, but its name does not lie: it disables the inclusion check against the transparency log, which is exactly what distinguishes Sigstore from a conventional PKI; if you use it, you are doing certificate signing, not Sigstore. And there is documented friction: cosign issue 4550 describes 3.0.2 trying to reach the TUF CDN despite having a local key and access only to an internal Nexus. Air-gap works, but it is not the happy path.

Signing a model is not signing a file

A servable model is a directory: several safetensors shards, config.json, tokenizer, perhaps a chat_template.jinja. Changing config.json (the context length, the rope_scaling) alters behaviour without touching a single weight. Signing only the weights leaves the door open.

The answer is the OpenSSF Model Signing (OMS) specification, published in June 2025 with contributions from Google, HiddenLayer, NVIDIA, Red Hat, Intel, Meta, IBM and Microsoft. Its design is a detached signature, which neither modifies nor repackages the content, over a manifest listing every file by its hash (SHA-256 by default, BLAKE2b as an alternative), with one signature covering the whole manifest. It is deliberately PKI-agnostic: it accepts bare keys, self-signed certificates, corporate PKI or keyless Sigstore. The reference implementation is model_signing, from sigstore/model-transparency, version 1.1.1 of 10 October 2025; 1.1.0 added PKCS#11 (HSM), private Sigstore instances, BLAKE3 and OpenTelemetry traces.

# Sign the whole directory: weights, config and tokenizer
model_signing sign /modelos/llama-70b-fin --signature /modelos/llama-70b-fin/model.sig

# Verification with a narrowed OIDC identity
model_signing verify /modelos/llama-70b-fin \
 --signature /modelos/llama-70b-fin/model.sig \
 --identity 'https://git.interno/plataforma/modelos/.github/workflows/publicar.yml@refs/heads/main' \
 --identity-provider 'https://git.interno'

The payload is a DSSE envelope with an in-toto statement whose subjects are the path-digest pairs of each file and whose predicateType is https://model_signing/signature/v1.0; support for file shards allows hashing in chunks. Maturity without embellishment: the project is in the OpenSSF sandbox. The specification is solid and has the vendors that matter behind it, with NVIDIA signing models in NGC with it, but it is the first formal step and there are few independent verifiers beyond the reference CLI.

The arithmetic of hashing, which turns out not to be the problem

What remains is how long it takes to hash a model of hundreds of gigabytes before every start. Let $ S $ be the total size, $ p $ the threads running in parallel over different shards, $ r_{\text{cpu}} $ the hashing throughput per thread and $ r_{\text{io}} $ that of reading from storage:

$$t_{\text{hash}} = \frac{S}{\min\left(p \cdot r_{\text{cpu}},\ r_{\text{io}}\right)}$$

With a dense 70B model in bf16, some 140 GB across thirty-odd shards, SHA-256 accelerated by SHA-NI runs at around 1.5-2 GB/s per thread. With eight threads the CPU term is around 14 GB/s, well above what a Gen4 NVMe delivers, on the order of 6 GB/s. The minimum is set by the disk, not by the hash: about 23 seconds. And that is the same throughput the loader is going to consume anyway to move the weights to HBM: if verification happens in the same initContainer that already reads the artefact, the marginal cost is CPU, not I/O. What is expensive is having put verification in a separate step that reads the disk twice (the rest of the startup budget is in the cold start post). What does have a real cost is incremental verification: if only one shard changes, the manifest allows checking only that one, but only if the distribution layer does incremental pulls. With an OCI artefact and granular layers it works; with a monolithic tarball, it does not.

Provenance: SLSA and in-toto over a QLoRA pipeline

The signature says who published; provenance says how it was produced. The framework is SLSA, whose v1.2 was published on 24 November 2025, backwards compatible with v1.1. Its novelty is the Source track; the Build Environment track and the Dependency track are still in development, and the first is exactly the one that would have helped with Ultralytics.

Level	What SLSA requires	What it means in a QLoRA pipeline
Build L0	Nothing	The adapter somebody trained on their workstation
Build L1	Automatically generated provenance: who built it, with what process and with what top-level inputs. It may be unsigned	The job emits provenance with commit, dataset and hyperparameters. It catches mistakes, not attacks
Build L2	The above plus a build on a hosted platform that generates and signs the provenance, verifiable by the consumer	The CI runner signs with its identity. Now there is something an admission controller can verify
Build L3	The above plus isolation between runs and signing material inaccessible from user-defined steps	An ephemeral runner per job, with no signing secrets within reach of the training script. This is the realistic target

in-toto is the format in which all of this is expressed: it graduated in the CNCF on 23 April 2025, which in practice makes it the common substrate of SLSA, of cosign attest and of OMS itself. Over the flow from the QLoRA runbook and from continuous fine-tuning, the useful attestation is the one that allows reproduction and auditing:

{
 "_type": "https://in-toto.io/Statement/v1",
 "subject": [{ "name": "adapter-soporte-v7",
 "digest": { "sha256": "e3b0c44298fc1c149afbf4c8..." } }],
 "predicateType": "https://slsa.dev/provenance/v1",
 "predicate": {
 "buildDefinition": {
 "buildType": "https://interno/build-types/qlora-finetune@v2",
 "externalParameters": {
 "modeloBase": "registro.interno/modelos/llama-70b@sha256:9f2a...",
 "dataset": "lakefs://corpus-soporte@commit-4c1f9e",
 "hiperparametros": { "rank": 16, "alpha": 32, "lr": 0.0002,
 "epochs": 3, "quant": "nf4" }
 },
 "internalParameters": {
 "gpu": "4xH100-SXM-80GB", "cuda": "12.6", "torch": "2.8.0",
 "imagenEntrenamiento": "registro.interno/ci/qlora@sha256:71ca..."
 },
 "resolvedDependencies": [
 { "uri": "git+https://git.interno/plataforma/finetune@a91c3f",
 "digest": { "gitCommit": "a91c3f..." } }
 ]
 },
 "runDetails": {
 "builder": { "id": "https://git.interno/plataforma/runners/gpu-efimero" },
 "metadata": { "invocationId": "run-2026-07-19-0041",
 "startedOn": "2026-07-19T02:14:33Z" }
 }
 }
}

The five fields you need and that are almost never complete: hash of the base model, immutable identifier of the dataset (here a lakeFS commit, in line with data versioning), hyperparameters, hardware and stack versions, and the code commit. Without the first the chain back to the original model cannot be rebuilt; without the second, the retrain loop is left without reproducible evidence. And the obligatory warning: SLSA says nothing about the quality of the dataset; Build L3 guarantees that nobody tampered with the build, not that the corpus was clean.

AIBOM: SPDX 3.0.1 versus CycloneDX 1.7

If the manifest is the seal and provenance the health register, the AIBOM is the ingredients label. There are two formats, and the honest comparison is less flattering than the vendor literature suggests. SPDX 3.0 (April 2024), with patch 3.0.1 on 17 December 2024, reorganised the specification into profiles, two of them relevant here: AI and Dataset. The AI profile defines on AIPackage fields such as typeOfModel, hyperparameter, informationAboutTraining, metric, safetyRiskAssessment and limitation, plus four energy-consumption fields, unique to it, broken down into training, fine-tuning and inference. v3.1 has been in RC since January 2025 without GA, and 3.0 is going through ISO as ISO/IEC DIS 5962 (the standard in force is still 2.2.1). CycloneDX 1.7 was published on 21 October 2025 and ratified as ECMA-424, 2nd edition, in December 2025; its ML-BOM rests on the machine-learning-model type and the modelCard object.

Criterion	SPDX 3.0.1	CycloneDX 1.7
Formal standardisation	ISO/IEC DIS 5962 in progress; the standard in force is 2.2.1	ECMA-424 2nd ed. (December 2025)
AI model	`AI` profile over `AIPackage`	`machine-learning-model` + `modelCard`
Dataset	Dedicated `Dataset` profile	`data` and `modelParameters.datasets`
Energy	Four broken-down fields (unique)	Not natively covered
Intended use and ethics	`limitation`, `safetyRiskAssessment`	`considerations` (closer to Model Cards)
Ergonomics	Rich model, verbose, steep curve	Compact JSON, more adoption in tooling
Fit with VEX	Via the `Security` profile	Native, aligned with CSAF VEX 2.0

{
 "bomFormat": "CycloneDX", "specVersion": "1.7", "version": 1,
 "components": [{
 "type": "machine-learning-model",
 "bom-ref": "modelo/adapter-soporte-v7",
 "name": "adapter-soporte", "version": "7.0.0",
 "hashes": [{ "alg": "SHA-256", "content": "e3b0c44298fc1c14..." }],
 "licenses": [{ "license": { "id": "Apache-2.0" } }],
 "modelCard": {
 "modelParameters": {
 "task": "text-generation",
 "modelArchitecture": "llama-70b + LoRA r=16",
 "datasets": [{ "type": "dataset", "name": "corpus-soporte",
 "contents": { "url": "lakefs://corpus-soporte@commit-4c1f9e" } }]
 },
 "quantitativeAnalysis": {
 "performanceMetrics": [
 { "type": "exactitud-eval-interna", "value": "0.83", "slice": "soporte-es" }
 ]
 },
 "considerations": {
 "useCases": ["asistencia interna a agentes de soporte"],
 "technicalLimitations": ["no evaluado fuera de castellano"]
 }
 }
 }]
}

What gets filled in on its own is little. Automated tooling gives you name, version, hashes, licences and dependencies of the inference environment: that is a classic SBOM and Trivy or Syft do it unaided. From the pipeline, because no scanner infers it from the bytes, must come dataset, hyperparameters, metrics, base model and energy. And from a person: limitations, intended uses and ethical considerations. The conclusion is uncomfortable: an AIBOM generated after the fact by a tool that looks at the artefact is, for the most part, an empty document with a valid structure. The useful AIBOM is emitted by the training job, the only one that knows those fields, and is signed along with the artefact.

Regulation: what binds, and from exactly when

The dates call for precision, because they changed in 2026 and much of the documentation in circulation is out of date.

EU AI Act

The obligations on providers of general-purpose AI models (GPAI) apply from 2 August 2025. Article 53 requires technical documentation in line with Annex XI, information for downstream providers in line with Annex XII, a copyright policy and a public summary of the training content. Annex XI, section 1, explicitly asks for the specifications of the training process, information on the data used for training, testing and validation, including its type and provenance, the computational resources employed (for example, floating-point operations), the training time and the known or estimated energy consumption. Anyone who had placed a model on the market before that date has until 2 August 2027.

The 2026 change: Regulation (EU) 2026/1744, the AI “Digital Omnibus”, was published in the Official Journal on 24 July 2026 and came into force on 27 July 2026. Chapter III for Annex III high risk moves from 2 August 2026 to 2 December 2027; Annex I high risk (AI as a safety component of regulated products), to 2 August 2028; on 2 December 2026 new prohibitions under Article 5 and machine-readable marking for GPAI come into force. The GPAI obligations of Articles 51 to 56 are not delayed.

The reading for an architect: if you fine-tune a model and place it on the market, Annex XI is enforceable now, and its fields are almost literally those of a SLSA attestation plus an AIBOM. If your case is high risk you have a year and a half more headroom than you thought, but that headroom is for the harmonised standards, not for starting late (detail in the AI Act post).

Cyber Resilience Act

The CRA came into force on 10 December 2024; the notification obligations of Article 14 apply from 11 September 2026 and full application from 11 December 2027. What it requires, in Annex I, Part II, point 1, is a software bill of materials in a commonly used, machine-readable format “covering at the very least the top-level dependencies”.

Three nuances that get cited badly. “At least the top-level dependencies” is a very low floor: it does not require the transitive tree. The SBOM does not have to be public, and recital 77 says so expressly: it is internal documentation that the authorities may require. And the CRA names no format; the Commission reserves the right to specify one by implementing act, and the most precise guidance today is BSI TR-03183-2, which accepts CycloneDX 1.6+ or SPDX 3.0.1+. The CRA talks about SBOM and not AIBOM; that the model artefact falls under its umbrella is interpretation, not text.

Fit with ENS and ISO/IEC 42001

None of this is voluntary hygiene. The signature and its verification at admission implement op.exp.6 and op.ext.3 of the ENS (supply chain) and control A.10 of ISO/IEC 42001; the provenance attestation covers op.exp.2 and A.6 (AI system life cycle); the AIBOM with dataset and hyperparameters is op.exp.1 (inventory) and A.7 (data); and the transparency log, op.exp.8. The evidence an ISO/IEC 42001 auditor asks for on the life cycle is, to a large extent, the same JSON the pipeline emits; the control-by-control breakdown is in the technical controls post.

Verification in the cluster: the receiving bay

None of the above is worth anything if nobody checks the seal at the door. Kyverno graduated in the CNCF on 16 March 2026; its 1.17 (February 2026) promoted CEL policies to v1, including ImageValidatingPolicy, and marked ClusterPolicy as deprecated, and 1.18 (24 April 2026) polished image verification. The alternative, Sigstore’s policy-controller, is at v0.15.1 (26 March 2026), which moved from cosign v2 to v3 and migrated to go-tuf v2, relevant for anyone running a private Sigstore with delegated roles.

apiVersion: policies.kyverno.io/v1
kind: ImageValidatingPolicy
metadata:
 name: modelos-firmados-y-con-provenance
spec:
 validationActions: [Deny] # fail-closed: it does not admit, it does not audit
 failurePolicy: Fail  # if the webhook fails, it is denied
 webhookConfiguration:
 timeoutSeconds: 20 # pulling the signature can take a while
 matchConstraints:
 resourceRules:
 - apiGroups: [""]
 apiVersions: ["v1"]
 resources: ["pods"]
 operations: ["CREATE", "UPDATE"]
 matchImageReferences:
 - glob: "registro.interno/modelos/*"
 attestors:
 - name: ci-plataforma
 cosign:
 keyless:
 identities:
 - issuer: "https://git.interno"
 subject: "https://git.interno/plataforma/modelos/.github/workflows/publicar.yml@refs/heads/main"
 ctlog:
 url: "https://rekor.interno"
 attestations:
 - name: slsaProvenance
 intoto:
 type: "https://slsa.dev/provenance/v1"
 validations:
 - expression: >-
 images.containers.map(image,
 verifyImageSignatures(image, [attestors.ci_plataforma])).all(e, e > 0)
 message: "Artefacto de modelo sin firma valida de la CI de plataforma"
 - expression: >-
 images.containers.map(image,
 verifyAttestationSignatures(image, attestations.slsaProvenance,
 [attestors.ci_plataforma])).all(e, e > 0)
 message: "Falta atestacion SLSA de procedencia"

Verify the model, not just the image

That policy covers the serving container’s image. The model is downloaded afterwards, in the initContainer, where the admission controller no longer reaches. The pattern that works is to verify there, before writing to the shared volume:

initContainers:
 - name: traer-y-verificar-modelo
 image: registro.interno/plataforma/oras-cosign@sha256:4b7e...
 command: ["/bin/sh", "-ec"]
 args:
 - |
 oras pull registro.interno/modelos/llama-70b-fin@sha256:9f2a... -o /modelos
 cosign verify --offline=true \
 --trusted-root /etc/sigstore/trusted_root.json \
 --certificate-oidc-issuer https://git.interno \
 --certificate-identity-regexp '^https://git.interno/plataforma/modelos/.*' \
 registro.interno/modelos/llama-70b-fin@sha256:9f2a...
 model_signing verify /modelos --signature /modelos/model.sig \
 --identity-provider https://git.interno \
 --identity 'https://git.interno/plataforma/modelos/.github/workflows/publicar.yml@refs/heads/main'
 volumeMounts:
 - { name: modelos, mountPath: /modelos }
 - { name: sigstore-root, mountPath: /etc/sigstore, readOnly: true }

The double verification is not redundant: cosign verify covers the OCI artefact as it stands in the registry, and model_signing verify covers the unpacked content as the engine is going to read it. If somebody mounts a ConfigMap that overwrites config.json after the pull, the first passes and the second does not.

Fail-closed versus fail-open, and what it costs

failurePolicy: Fail with validationActions: [Deny] means that if the webhook does not answer nothing starts: the right call from a security standpoint, and exactly what takes a cluster down on a Sunday morning. The defensible posture is fail-closed in the production namespace, with the controller in high availability and its own namespace excluded from its policies so it can be recovered; Audit during adoption, the same “observe first, block later” rule we applied with Tetragon in hardening; and a written exception procedure with an expiry date, because on the day of the 3 a.m. hotfix somebody is going to bypass the policy and it is better that they do it down an audited path.

On latency, the cost is at startup and not per request: verification happens once per pod. A third-party benchmark from March 2026 on policy-controller v0.15 reports 92 ms at p50 and 184 ms at p99 for internally signed images, with peaks of up to 4 seconds in mass deployments. It is a third-party measurement with no published independent replication, but the order of magnitude is coherent: hundreds of milliseconds against the minutes a pod takes to load 140 GB into HBM.

As a second net, Harbor stores cosign signatures as referrer artefacts and can prevent the deployment of anything unsigned by project policy. And in GitOps with Flux, what closes the circle is always referencing by digest and verifying the OCI artefact’s signature at reconciliation: two independent checks at different moments.

Decision map: what to put in, and in what order

By decreasing return on effort:

Pin by digest throughout GitOps. An afternoon; without this, the rest is decorative.
Image signing with keyless cosign from CI and ImageValidatingPolicy in Audit. Days. It reveals how much of your cluster is unsigned, which is usually a surprise.
Signing the model artefact with model_signing in the job that publishes it. Days. It kills the model uploaded by hand on a Friday.
Moving to Deny in the inference namespace, with audited exceptions and the controller in HA.
SLSA attestation, targeting Build L2 and then L3. Weeks: it requires ephemeral runners and taking the signing material out of the training script’s reach.
An AIBOM emitted from the pipeline, with the five fields only it knows. Weeks, and above all process work.
Private Sigstore, only with a real air-gap requirement and with the previous six done.

Between Kyverno and policy-controller: Kyverno if there is or will be a broader policy programme (Pod Security, labels, quotas), for unification and for its graduated status; policy-controller if the organisation is Sigstore-centric and does not want more policy engine than strictly necessary. Between keyless and a key: keyless if CI can reach an OIDC, even an internal one; a key with an HSM in strict air-gap where you are not going to run your own Fulcio.

Operational traps

Verifying without narrowing the identity. cosign verify without --certificate-identity or --certificate-oidc-issuer, or with a regexp that accepts any repository from the provider, is security theatre. Trap number one and the quietest.
Fail-closed bugs in the verifier itself. Kyverno issue 16435, opened on 2 July 2026 against 1.18.0, describes two simultaneous regressions in ImageValidatingPolicy with key and certificate attestors: a null-pointer SIGSEGV and a tlog failure with the message not enough verified log entries from transparency log: 0 < 1. Both deny legitimate images or bring the controller down. Fixed for 1.19, with the underlying lesson intact: the verifier is a critical component and its regressions are unavailability.
The trust root that expires in air-gap. It was copied by hand nine months ago, nobody keeps the calendar, and one day verification fails without anyone having touched anything.
The AIBOM nobody regenerates. It is emitted at the initial deployment, the model is fine-tuned three times and it still describes version one. An out-of-date inventory is worse than none: it creates unjustified confidence.
Verifying the image and forgetting the model. The most common of all: the vLLM container impeccably signed, and the 140 GB it loads coming from an unverified bucket.

What this does NOT prove

A signature says who published, not that the model is good or safe. It is a claim of authorship, not of quality. Ultralytics documents it: four releases with a cryptominer and valid attestations. If the build environment is compromised, the signature faithfully certifies the compromised artefact.

A signed model can have a backdoor. Nothing in Sigstore, OMS or SLSA examines the weights. A backdoor trained by someone with legitimate access to the pipeline passes through the whole chain without triggering an alarm; the defence is adversarial evaluation and canary or shadow, not cryptography. And SLSA says nothing about dataset quality: a formally perfect pipeline over poisoned data produces a poisoned model with impeccable provenance.

The AIBOM is only as good as the process that generates it. The fields that matter are not inferred by any tool. And the operational cost is recurring: rotation of keys and TUF roots, verification incidents in the small hours, policy noise during adoption, and the continuous work of maintaining an inventory nobody reads until there is an audit.

The maturity split as of July 2026. Mature and deployable: Sigstore and cosign over OCI artefacts, Kyverno and policy-controller as admission control, in-toto as a CNCF-graduated format, pinning by digest. Usable with judgement: OMS and model_signing (a solid specification with strong industry backing, but a sandbox project), SLSA Build L2-L3 in fine-tuning, CycloneDX ML-BOM. Still work in progress: SLSA’s Build Environment and Dependency tracks (precisely the ones that would cover the Ultralytics scenario), the tooling that fills in an AIBOM automatically, and all backdoor detection in weights, which today is literature and not product.

For an inference factory

First: the initContainer is your receiving bay, and today it probably checks nothing. The admission controller looks at the image; the model comes in afterwards through a side door. Adding cosign verify --offline and model_signing verify there costs a couple of afternoons, fits inside the cold start budget (23 seconds of hashing over a disk throughput you are already paying for) and closes the largest hole in most deployments.

Second: pin by digest today, sign tomorrow. If only one thing fits this quarter, it is removing mutable tags from GitOps: it is cheap, it breaks nothing and it turns “I trust the registry” into “I trust these specific bytes”. The signature adds the issuer’s identity on top, but without a digest it has nothing to hold on to.

Third: require the attestation from whoever hands you the model, even if it is the team next door. An inference factory consumes what somebody else produces, and that contract should be explicit: a signature from a known CI identity, an attestation with the base model’s hash and the dataset’s commit, and an AIBOM emitted from the pipeline. Annex XI of the AI Act already requires that, and it is in force, and it is what lets you answer in twenty minutes, rather than in two weeks, when somebody asks what exactly is behind the endpoint.

The chain now has three links: you know how the model is served, where its bytes come from and why you trust them. The last question remains: what that pod is relative to the rest of the system when it starts with the verified model, and how far you can trust the machine it runs on. The fourth article goes into workload identity with SPIFFE/SPIRE and isolation with Confidential Containers: a verified model, served by an unidentified process on a host you cannot attest, leaves the chain open right at the last link.

Sources

Sigstore, Releases · sigstore/cosign (v3.1.1, 9-6-2026) — https://github.com/sigstore/cosign/releases
Sigstore Blog, Cosign v3 is now available (8-10-2025) — https://blog.sigstore.dev/cosign-3-0-available/
Sigstore Blog, Rekor v2 GA — Cheaper to run, simpler to maintain — https://blog.sigstore.dev/rekor-v2-ga/
Some Natalie, Verifying Cosign signatures offline — https://some-natalie.dev/blog/cosign-disconnected/
sigstore/cosign, Issue 4550: Cosign 3.0.2 keeps reaching out to the TUF CDN — https://github.com/sigstore/cosign/issues/4550
OpenSSF, An Introduction to the OpenSSF Model Signing (OMS) Specification — https://openssf.org/blog/2025/06/25/an-introduction-to-the-openssf-model-signing-oms-specification/
Sigstore, model-transparency — README y releases (model_signing 1.1.1) — https://github.com/sigstore/model-transparency/blob/main/README.md
SLSA, Announcing SLSA v1.2 y Build track basics — https://slsa.dev/blog/2025/11/announce-slsa-v1.2
CNCF, Graduation of the in-toto security framework — https://www.cncf.io/announcements/2025/04/23/cncf-announces-graduation-of-in-toto-security-framework-enhancing-software-supply-chain-integrity-across-industries/
SPDX, AI profile — SPDX Specification 3.0.1 — https://spdx.github.io/spdx-spec/v3.0.1/model/AI/AI/
CycloneDX, CycloneDX v1.7 released (y ratificación ECMA-424 2ª ed.) — https://cyclonedx.org/news/cyclonedx-v1.7-released/
EU Artificial Intelligence Act, Article 53 y Annex XI — https://artificialintelligenceact.eu/article/53/
NicFab, Digital Omnibus on AI: Regulation (EU) 2026/1744 published in the Official Journal — https://www.nicfab.eu/en/posts/digital-omnibus-ai-official-journal/
cyberresilienceact.eu, The Cyber Resilience Act Explained: Scope, Classes & Deadlines — https://www.cyberresilienceact.eu/explained.html
CRA Decoded, CRA SBOM Requirements: What’s Mandated, What’s Optional — https://medium.com/@cra-decoded/cra-sbom-requirements-whats-mandated-what-s-optional-and-what-s-still-unclear-6de9270666de
Kyverno, Announcing Kyverno Release 1.18 e Issue 16435 — https://kyverno.io/blog/2026/04/24/announcing-kyverno-release-1.18/
Safeguard, Sigstore Policy Controller v0.15 Admission Review 2026 — https://safeguard.sh/resources/blog/sigstore-policy-controller-v0-15-2026
JFrog, Malicious Hugging Face ML Models with Silent Backdoor — https://jfrog.com/blog/data-scientists-targeted-by-malicious-hugging-face-ml-models-with-silent-backdoor/
ReversingLabs, Novel ML Malware (nullifAI) on Hugging Face — https://www.reversinglabs.com/press-releases/reversinglabs-identifies-novel-ml-malware-hosted-on-leading-hugging-face-ai-model-platform
EleutherAI / Hugging Face, Safetensors audited as really safe (Trail of Bits) — https://blog.eleuther.ai/safetensors-security-audit/
Socket, Ultralytics PyPI Package Compromised Through GitHub Actions Cache Poisoning — https://socket.dev/blog/ultralytics-pypi-package-compromised-through-github-actions-cache-poisoning
arXiv, Weight Space Detection of Backdoors in LoRA Adapters — https://arxiv.org/html/2602.15195v3

Model chain of trust (4/4): who serves the model and which machine you trust

Sun, 26 Jul 2026 17:00:00 +0200

The three previous articles closed questions about the artefact: what contract the served model exposes, where its bytes come from and why you trust them. What remains is the one almost nobody asks until an auditor asks it: nothing proves that the process currently returning tokens on port 8000 is the inference engine and not something that slipped into the cluster and inherited its token. And a second one, worse: whether the machine it runs on is trustworthy, and against whom.

These are different problems with different projects. Workload identity is solved by SPIFFE/SPIRE, a CNCF graduated project since 2022. Environment isolation is covered by Kata Containers and Confidential Containers, the latter promoted to CNCF incubating on 22 July 2026, four days before this article was published. The first is cheap and almost always worth it. The second is expensive, more than the vendor’s press release suggests, and is only justified against a specific threat model.

TL;DR

User identity ≠ workload identity. OIDC says which person is behind a request; it does not say which process serves it. ServiceAccount tokens, static keys and gateway API keys are bearer credentials: whoever holds them, is them.
SPIFFE defines a URI identifier (spiffe://domain/path) and a verifiable document (SVID, X.509 or JWT) delivered by a Workload API that does not require the workload to hold any prior secret. SPIRE implements it with node and workload attestation; defaults: default_x509_svid_ttl 1 h, default_jwt_svid_ttl 5 min, ca_ttl 24 h.
Identity is only worth something if something enforces it: Istio/Envoy via SDS, Cilium with mutual authentication (with serious reservations) or an ext_authz with OPA in front of the gateway.
Three rungs of isolation: container with a shared kernel → Kata Containers (its own kernel and VM; v3.32.0 from June 2026) → Confidential Containers (hardware TEE, hypervisor outside the trust base, remote attestation with Trustee per RFC 9334).
The overhead of confidential mode runs from ~0 % to ~28 % depending on what you measure: GPU-only in CC with large batches sits around 4-8 %, while the full stack (a CVM with Intel TDX plus an H100 in CC), measured independently in 2026, gives +21.8 % to +27.8 % on TTFT and −17.7 % to −21.1 % on throughput. Honest heuristic: reserve 15-25 % extra capacity.
MIG and vGPU are forbidden in CC mode according to NVIDIA’s reference architecture, and every GPU on a node goes in the same mode.
Closing criterion: the order is contract → digest → signature verified at admission → workload identity → TEE.

The analogy: the biosafety laboratory

A laboratory handling pathogens solves two problems much like ours, separately.

The first is who gets in. The company badge is not enough: at the airlock there is a specific credential, short-lived and automatically renewed (if lost, it expires on its own within an hour), issued only after independently checking that the applicant is where they say they are and is who they say they are. That credential is the SVID; the prior check is attestation.

The second is which room you work in, and here there are containment levels. On the open bench the air is shared and anything aerosolised affects everyone: the normal container, with a kernel shared by all the neighbours. One rung up, the biological safety cabinet, with a physical barrier and its own airflow: Kata Containers. At the very top, the maximum containment laboratory, where maintenance staff change filters without ever seeing the sample: the TEE, where the operator maintains the machine but cannot read its memory. And before opening the airlock somebody verifies that the integrity of the suit and of the room is as expected: remote attestation with conditional secret release.

The uncomfortable lesson: the maximum containment level is a cost decision before it is one more degree of security. Nobody builds a BSL-4 to grow yeast. The useful question is who it isolates from, not how much.

Part A — Workload identity

Six services talking to each other and none of them knows who the other is

In a serious inference cluster, like the seven layers one, the gateway talks to the serving engine, the engine queries the vector database, the trace collector receives spans from everyone and the agents with MCP invoke tools that in turn call the gateway. Many east-west connections and, in most deployments, none of them authenticated.

User identity says which person is behind the request and is resolved with OIDC, as when putting Keycloak in front of MCP. Workload identity says which process issues it: there is no human on the other side, no browser, no consent, and the process is born and dies in seconds. Its three usual substitutes fail for different reasons:

Mechanism	Where it fails
Projected `ServiceAccount` token	SA granularity, not workload granularity: two pods with the same SA are indistinguishable. It does not cross the cluster boundary and does not federate. It is a bearer token on the filesystem
Static key	It does not rotate, it gets shared over Slack, it does not distinguish issuer from bearer; once leaked, it is valid until somebody notices
Gateway API key	It authenticates the client, not the process. LiteLLM’s virtual keys are for quota and budget, not for proving identity

There is no cryptographic link between credential and process: copy the credential, copy the identity. In a multi-tenant cluster like the H100 cluster one, real isolation then depends on network topology and not on identity.

SPIFFE: the standard

SPIFFE and its implementation SPIRE graduated together in the CNCF on 20 September 2022. As of July 2026 they are settled infrastructure, not a bet.

The SPIFFE ID is a URI of the form spiffe://<trust-domain>/<workload-identifier>, for example spiffe://inferencia.example/ns/inferencia/sa/vllm: a name, not a credential. The trust domain is the root of trust (organisation, environment or site), and the guidance recommends separating workloads from different sites or security environments into different domains.

The SVID is the credential that proves that ID. The X509-SVID carries the SPIFFE ID in the URI-type SAN, not in the CN, which the specification discourages as a source of identity, and it is the preferred format. The JWT-SVID exists for when there are L7 proxies terminating TLS in the middle, with the replay risk the documentation warns about:

Certificate:
Issuer: C=ES, O=SPIFFE
Validity
Not Before: Jul 26 08:00:00 2026 GMT
Not After : Jul 26 09:00:00 2026 GMT
Subject: C=ES, O=SPIRE, CN=vllm.inferencia
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Alternative Name:
URI:spiffe://inferencia.example/ns/inferencia/sa/vllm

One hour of validity, and what authenticates is the SAN URI: the CN is decorative.

The Workload API is the elegant piece: a local UNIX socket from which the workload obtains its SVID, its private key and the trust bundle, without presenting any secret. The documentation is explicit: “the Workload API does not require that a calling workload have any knowledge of its own identity, or possess any authentication token”. That solves the cold start of all identity cryptography: the secret you would need in order to obtain the first secret.

SPIRE: how that document is issued

SPIRE has a server (the authority that signs) and one agent per node (which exposes the Workload API), and it chains two checks.

Node attestation verifies that the agent runs where it says it does. In Kubernetes the reference attestor is k8s_psat, which validates a projected token with an audience against the API server; on bare metal there are TPM-based attestors. A note of honesty: SPIRE v1.15.1, from 28 May 2026, is a security patch over an incorrect PKCS7 validation in the azure_imds attestor that allowed attested documents to be forged and a virtual machine to be impersonated.

Workload attestation answers the opening question. The k8s attestor receives the PID of the process that opens the socket, infers the pod from its cgroup membership and queries the kubelet for the metadata. From there come the selectors: namespace, service account, pod name and UID, labels, owner, node and, above all, container and image by tag or by digest. From SPIRE v1.15.0 (19 May 2026) Sigstore support stopped being experimental, enabling selectors by signature verification status, certificate subject and issuer, and transparency log. That closes the series’ circle: you can require that the engine’s SVID go only to the container whose image carries a valid cosign signature from the expected issuer, which is what was verified in article 3/4. Provenance goes from being a one-off admission check to a precondition of identity at runtime.

spire-server entry create \
 -parentID spiffe://inferencia.example/spire/agent/k8s_psat/prod/nodo-gpu-a \
 -spiffeID spiffe://inferencia.example/ns/inferencia/sa/vllm \
 -selector k8s:ns:inferencia \
 -selector k8s:sa:vllm \
 -selector k8s:container-name:vllm \
 -selector k8s:container-image:vllm/vllm-openai@sha256:aa11bb22cc33 \
 -x509SVIDTTL 3600 \
 -federatesWith spiffe://sede-b.example

The selectors are conjunctive: a pod in another namespace, with another SA or with another image does not obtain that SVID even if it shares the node.

The default TTLs have three consequences. Rotation is continuous, not a one-off event: code that loads the certificate once at startup will break exactly one hour later. Server downtime has a clock: twenty minutes are invisible, two hours bring down the cluster’s authenticated communications, and the scaling guide admits that “a single SPIRE Server instance represents a single point of failure”, with the datastore as the bottleneck and indicative figures running from two replicas of 1 CPU and 1 GB for 10 agents to eight of 16 CPU and 16 GB for 5,000 agents and 10,000 workloads. And tag-based selectors are fragile: the runtime may report one tag or another depending on the node and the moment, so the correct selector is the digest, as in article 2/4.

Federation between trust domains

Two sites with independent clusters, or an external partner exposing a reranker, should not share an authority. SPIFFE federation makes each domain publish a bundle endpoint with its public trust material and the other poll it periodically: the https_web profile authenticates it with web PKI and https_spiffe with an X509-SVID from the domain itself, enabling automatic rotation and revocation of the root. The relationship is one-directional, bundles from different domains must never be merged and refresh is by polling with a default suggestion of five minutes. In practice, site A’s gateway accepts requests from site B’s engine without sharing a CA or an identity database, and cutting the relationship means deleting a bundle, not revoking certificates.

From identity to effective authorisation

An SVID blocks nothing by itself; somebody has to compare the presented ID against a policy.

Istio + SPIRE is the best-worn integration: Istio detects a UNIX socket implementing Envoy’s SDS API and the proxy obtains its identities from there instead of from istiod, mounted with the SPIFFE CSI Driver (recommended over hostMount). Two conditions break deployments: SPIRE’s trust domain and Istio’s must match exactly, and SPIRE only issues to previously registered workloads, including Istio’s own components. With that, policy is written in terms of identity and not IP:

apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
 name: vllm-solo-desde-gateway
 namespace: inferencia
spec:
 selector:
 matchLabels:
 app: vllm
 action: ALLOW
 rules:
 - from:
 - source:
 principals: ["inferencia.example/ns/inferencia/sa/gateway"]
 to:
 - operation:
 methods: ["POST"]
 paths: ["/v1/chat/completions", "/v1/completions"]

It is a layer on top of the NetworkPolicy from layered hardening: there we closed communication pairs by topology; here we also require who calls and on what path.

Cilium offers mutual authentication backed by SPIRE, tempting if you already use eBPF for the data network. It deserves criticism: it is still marked beta in the stable documentation, and The New Stack published a reasoned critique arguing that the design does not preserve mTLS properties over the life of the connection (it uses the handshake only to authenticate and discards the session keys) and that the identity model rests on eventually consistent per-node IP-identity caches, which can allow traffic that policy should deny.

The inference gateway and LiteLLM do not speak SPIFFE natively as of this article, and saying so is better than suggesting an integration that does not exist. The pattern that does work is the one the project itself documents: an Envoy in front that terminates mTLS with an X509-SVID or validates a JWT-SVID and delegates to OPA via ext_authz. The gateway carries on with its own job (quota, budget, routing), as we saw when choosing an OSS gateway and in the L7 router; workload identity is resolved one hop earlier.

AI agents and MCP: the missing piece

An agent invoking tools over MCP is a shared secret with legs: it is handed a static credential, the MCP server cannot know whether the caller is the legitimate agent or any process holding the same string, and the blast radius is the entire set of exposed tools. The threat model for agent isolation warned that a granted permission is a usable permission; with bearer credentials it is also transferable.

SPIFFE provides what is missing: the agent presents a short-lived SVID obtained through attestation and the MCP server authorises against the SPIFFE ID. In 2026 this stopped being theory: there is work at the IETF on dynamic OAuth client registration based on trusted SPIFFE issuers, and in June 2026 Google published a SPIFFE-based Agent Identity in its IAM. None of it is fully standardised.

Two honest caveats. SPIFFE answers “who”, not “why”: to know whether the action is the one the user asked for you need traceability (MCP with OTel) and runtime detection with Tetragon. And both identities must be composed: the agent proves its own with an SVID and propagates the delegating user’s in the token, authorising on the pair. Collapsing them into one is how you end up with agents able to do, in the system’s name, things no user could.

Part B — Isolation of the execution environment

### Kata Containers: the virtual machine sandbox

Kata replaces runc with a runtime that boots a lightweight virtual machine per pod, with its own kernel and a kata-agent inside; for Kubernetes it is transparent via RuntimeClass. As of July 2026 the stable branch is 3.32.x (3.32.0 is from June 2026, with Rust 1.94, Go 1.25.11, QEMU 11.0.1, guest kernel 6.18.35 and containerd 2.3), and in April 2026 the 4.0.0 preview was published, making the Rust runtime (runtime-rs) the default and leaving the Go one deprecated until 5.0.0, because of memory footprint.

For GPUs the route is VFIO passthrough. The NVIDIA GPU Operator automates it with sandboxWorkloads.enabled=true and sandboxWorkloads.mode=kata, installing the VFIO Manager, the Sandbox Device Plugin, the Confidential Computing Manager and the Kata Manager. It requires virtualisation and ACS in the BIOS, IOMMU, kata-deploy 3.29.0 or later, containerd (not CRI-O), the KubeletPodResourcesGet feature gate (on by default from Kubernetes 1.34) and removing the NVIDIA drivers from the host: the GPU is handed over whole to the VM and the guest manages it.

apiVersion: v1
kind: Pod
metadata:
 name: vllm-kata
 namespace: inferencia
spec:
 runtimeClassName: kata-qemu-nvidia-gpu
 containers:
 - name: vllm
 image: vllm/vllm-openai@sha256:aa11bb22cc33
 resources:
 limits:
 nvidia.com/pgpu: "1"

Two measurable costs. Memory: the VM reserves its own and you have to declare it as overhead in the RuntimeClass so the scheduler does not overcommit the node. And startup: according to the SESAME'24 study on serverless confidential containers, cold start goes from around 6 s with runc to around 7 s with Kata, and warm start from 1 s to 2 s. Acceptable for an engine that takes minutes to load weights; unacceptable for ephemeral functions.

Confidential Containers: the TEE

Confidential Containers (CoCo) takes the hypervisor and the platform operator out of the trust base. The CNCF promoted it to incubating on 22 July 2026, with more than 150 active contributors, 26 repositories and over 1,200 merged pull requests, and with Microsoft Azure, Intel, AMD, IBM, NVIDIA, Alibaba and Red Hat behind it. A serious project, but incubating means it is not graduated and that its integration surface changes between versions.

The components are the CoCo pods (unmodified containers run in a TEE via Kata) and Trustee: KBS (Key Broker Service, which releases secrets conditionally), AS (Attestation Service, which validates the hardware evidence) and RVPS (Reference Value Provider Service, which holds the reference values). Inside the guest, the Attestation Agent collects the evidence and the Confidential Data Hub consumes the secrets.

The flow is a clean instance of the RATS architecture from RFC 9334: the agent inside the TEE is the attester, the Attestation Service the verifier and the Confidential Data Hub the relying party. The evidence is a hardware-signed report with the boot measurements; the AS validates it against the RVPS values; and only if it matches does the KBS release the key that decrypts the image layers or the weights. Without valid attestation there is no key, and without a key there is no model: that is what stops the operator extracting it.

A mapping warning: the confidential-containers/operator repository was archived in February 2026 and deployment moved to the Helm charts and the trustee-operator. As of July 2026 the latest chart release is v0.21.0, aligned with kata-deploy 3.31.0, and Trustee v0.20.0, with TLS 1.3 and post-quantum cryptography, external plugins for the KBS and multi-GPU support via Intel Trust Authority and NVIDIA NVSwitch.

apiVersion: confidentialcontainers.org/v1alpha1
kind: TrusteeConfig
metadata:
 name: trusteeconfig
 namespace: operators
spec:
 profileType: Restrictive
 httpsSpec:
 tlsSecretName: trustee-tls-cert

The Permissive profile exists for development and should not leave it: it is the mode in which attestation does not block.

The hardware: CPU and GPU

On the CPU side the two relevant TEEs are AMD SEV-SNP and Intel TDX; NVIDIA’s reference architecture sets EPYC Milan/Genoa and Xeon Emerald/Granite Rapids as the baseline. Host support is no longer the hard part.

On the GPU side, the technical description of the design of the first confidential GPUs explains what CC mode on an H100 buys you. Memory is split into a Compute Protected Region that hardware firewalls stop both the CPU over PCIe and other GPUs over NVLink from accessing. Everything crossing the CPU-GPU boundary passes through bounce buffers encrypted with AES-GCM-256, and command buffers and CUDA kernels are encrypted and signed before crossing the bus. There is a chain of trust from GPU boot with a signed attestation report, and only NVIDIA-signed firmware runs in CC mode, validated against NRAS or locally in isolated environments. And the performance counters are disabled in hardware as a side-channel mitigation: that is the telemetry much of observability with DCGM depends on.

What it does not protect: the package’s HBM memory is not encrypted (it is considered resistant to interposers, which is a claim about physical difficulty and not a cryptographic guarantee); nothing against denial of service, because the operator who cannot read you can still switch you off; nothing against timing or access-pattern side channels; and nothing against a bug in your own code inside the enclave or prompt injection, which remains the job of guardrails.

On Blackwell, NVIDIA announces (2 July 2026) NVLink encryption that extends confidential computing to up to 8 GPUs, non-existent on Hopper and a condition for serving a large model with tensor parallelism. The reference architecture lists H100, H200, RTX Pro 6000 Blackwell Server Edition and B200 in single-GPU passthrough, and H100/H200 in PPCIe mode and B200 for multi-GPU.

The overhead numbers, with source and with judgement

Here it pays to distrust everybody, papers included. The published spread runs from 0 % to 28 % because the studies do not measure the same thing.

Source (date)	What it measures	Result	Nature
NVIDIA, Blackwell blog (Jul. 2026)	B200 with CC enabled	−1.0 % to −7.5 % throughput; per-token latency under 8 %; “up to 98 % of native performance”	Vendor claim
Corvex, HGX B200 with TDX (2026)	“Verified” deployment, encrypted NVSwitch	“Near-native performance”, with no figures of its own	Commercial claim
arXiv 2409.03992 (2024)	H100 GPU only in CC, Llama-3.1 8B/70B	−0.36 % to 6.85 % throughput; TTFT up to +19 %; overhead → 0 as the model grows	Independent, partial
arXiv 2509.18886 (Sep. 2025)	CPU TEE (TDX/SGX) and GPU TEE (H100), Llama2 7B/13B/70B	CPU TEE: under 10 % throughput and 20 % latency. GPU TEE: 4-8 %	Independent
arXiv 2607.19353 (May 2026)	Full stack: H100 in CC inside a CVM with TDX, Mistral-7B and Qwen3-30B-A3B under load	TTFT +21.8 % and +27.8 %; throughput −17.7 % and −21.1 %; closed-loop, 11.5-20.2 %	Independent, complete

The reconciliation: the penalty is in the data path, not in the GPU’s compute. NVIDIA itself quantifies the bottleneck: the effective bandwidth of the CPU-GPU interconnect in CC mode is limited by the CPU’s encryption throughput, “around 4 GB/s”. From there, three rules. The higher the compute/I-O ratio, the lower the overhead: a 70B with large batches and long sequences amortises the toll almost entirely, and a 7B with small batches and short prompts pays it in full. Adding the CPU TEE brings its own cost, and that is where most of the difference between 4-8 % and 20 % lies. And model loading and cold start are the worst cases, with tens of gigabytes crossing an encrypted bus at 4 GB/s, which makes everything discussed in from disk to HBM and in speeding up cold start more important.

The recommendation from the authors of the 2026 study, reserving between 15 % and 25 % extra capacity, is the figure I would take to a capacity planning exercise, and not the vendor’s “98 % of native performance”. Both can be true at once; only one is prudent for sizing.

When each rung is worth it

Scenario	Kata	CoCo (TEE)	Reason
Tenants that do not trust each other	Yes	Depends	Kata removes the shared kernel, a real escape vector
Unaudited third-party code or models	Yes	Optional	A shared kernel is too much surface for someone else’s code
Model intellectual property against the infrastructure operator	Not enough	Yes	The canonical case: third-party hosting, cloud, a partner providing the iron
Regulated third-party data on infrastructure you do not control	Not enough	Yes	See defence and healthcare
Sovereign on-premise, one tenant, trusted operator	Useful	Expensive over-engineering	It protects you from an adversary you do not have
Low latency with small batches	Yes	Bad idea	The worst point on the bus-encryption cost/benefit curve

The nuance that saves the most money: the two axes are independent. Kata without a TEE is cheap and delivers most of the isolation between tenants; CoCo only adds something if your threat model includes the operator. If you build a sovereign factory with your own iron and your own staff and you answer “no” to that question, the TEE is a cost with no return.

Operational traps

MIG and vGPU against confidential mode. NVIDIA’s GA 1.0.0 reference architecture is blunt: MIG and vGPU are forbidden in CC mode, and mixed-mode nodes are not supported; every GPU on a host goes in CC or none does. The forums show the typical contradiction of a moving area: in February 2026 a moderator states that MIG+CC is not supported and in April a user cites the MIG page suggesting that Hopper and Blackwell already allow it. As of this article the reference architecture overrides the product page: if your multi-tenancy rested on partitioning the GPU with MIG, enabling CC sends you back to “one whole GPU per tenant”.

Startup grows non-linearly. SESAME'24 breaks down where: SEV memory provisioning adds around one second per 2 GB of guest (SEV pins every page in advance, and an inference VM allocates a lot), the OVMF firmware about three seconds and layer decryption another three to five. Scaling from 0 to 16 instances goes from 16 s with runc to 190 s: irrelevant for a stable Deployment, a design change for aggressive autoscaling with KEDA.

Version coupling is brutal. Kata, kata-deploy, GPU Operator, containerd, QEMU with its own patches, guest kernel, NVIDIA driver and Trustee are a matrix that has to be treated as a unit: the reference architecture fixes Kubernetes 1.32 or later, Kata 3.29, GPU Operator 26.3.1 or later, containerd 2.2.2 or later and QEMU 10.1. Add it to the weight the stack’s operators already carry.

Attestation fails after a firmware update, and that is normal. The RVPS reference values describe a specific boot state. When you patch the SEV-SNP firmware, the microcode or the GPU firmware, the TCB version changes, the evidence stops matching and the KBS does not release keys, so the pods do not start. Since these updates usually respond to a security bulletin, the typical scenario is “we applied the critical patch on a Friday and on Saturday inference would not start”. The correct procedure reverses the order: update the reference values first with an overlap window, and patch the iron afterwards.

The attestation service is a single point of failure by design. If the KBS does not answer, no new confidential pod gets its key; those already started survive, but any rescheduling or scaling does not. The same dependency as the SPIRE server and the same treatment: real high availability, its own alert and rehearsed degradation.

Closing the series: the complete chain

Article 1/4 set the contract: what the endpoint exposes and what control plane governs it reproducibly. In 2/4 the subject was the provenance of the bytes: which registry they come from, by immutable digest and not by a moving tag. 3/4 added the cryptographic proof of what they are, with signatures, build attestations and an AIBOM verified at admission. This 4/4 closes with who runs them and where: an SVID that proves the serving process’s identity and an isolation rung chosen for the adversary you actually have. If one of the four fails, the other three are worth less than they look: an impeccably signed model served by an unidentified process on a machine a third party can dump is still a problem.

With a limited budget, the order follows cost per unit of risk removed. First, digests everywhere: almost free, and it removes a whole family of substitution attacks. Second, signature verified at admission, low cost and the first thing an auditor asks to see. Third, a declarative contract and control plane, which make everything else auditable. Fourth, workload identity with SPIFFE/SPIRE: expensive to operate, but it is the only thing that turns east-west traffic into something authorisable, and it becomes mandatory as soon as agents with MCP arrive. And last, the TEE, only if the infrastructure operator is in your threat model; Kata without a TEE can jump ahead to fourth place if there are tenants that do not trust each other.

Mapping the series to ENS, ISO/IEC 42001 and the EU AI Act

The detail is in technical controls ENS × ISO 42001 × EU AI Act; this table is the summary by link.

Link	ENS measure (RD 311/2022)	ISO/IEC 42001 (Annex A)	EU AI Act
1/4 Contract and control plane	`op.exp.2`; `op.mon.1`	A.6 life cycle	Art. 12 (record-keeping); Art. 13 (transparency)
2/4 Registry and distribution by digest	`op.ext.3`	A.10 third parties	Art. 11 and Annex IV (technical documentation)
3/4 Signature, attestations and AIBOM	`op.exp.6`; `op.ext.3`	A.6.2 and A.7 data and traceability	Art. 15(5), data and model poisoning
4/4-A Workload identity	`op.acc.1/2/5`; `op.exp.11`; `mp.com.2-3`	A.9 use of the AI system	Art. 15(5), unauthorised third parties
4/4-B Isolation and TEE	`mp.info.3`; `op.exp.2`; `mp.com.4`	A.6 life cycle controls	Art. 15(4) robustness; Art. 15(5)

Art. 15(5) carries the most weight: it requires high-risk systems to be “resilient against attempts by unauthorised third parties to alter their use, outputs or performance”, and it enumerates data and model poisoning and confidentiality attacks. Links 3 and 4 are its technical implementation. For the management framework, see ISO/IEC 42001 as an AIMS and the AI Act mapping.

For an inference factory

Start workload identity with the gateway↔engine pair and with the agents, not with the whole cluster. Registering all forty workloads on day one is the most reliable way to abandon the project: register two entries, set up the Istio policy that requires the gateway’s principal and live with it for a couple of weeks to learn what breaks when the certificate rotates. Then the agents with MCP, where static credentials are the entire threat model.

Separate the Kata decision from the TEE decision, and take them in that order. Kata is a RuntimeClass change with a second of startup and some memory. CoCo is a redesign: you lose MIG and the GPU’s performance counters, you gain ten seconds or more of startup, you pay 15-25 % of capacity and you add two dependencies whose failure prevents workloads from starting. Evaluate the second with a written threat model, not with a slide.

If you go to CoCo, rehearse patch day before you need it. The failure that will take your inference down will not be an attack, it will be a firmware update that misaligns the RVPS reference values. Write the runbook (update the RVPS with overlap, patch, retire the old value), rehearse it on one node and monitor the attestation success rate as a first-class metric, just as for SPIRE: alert on attestation failures before they turn into 401s at the gateway.

That closes the series: from the contract to the artefact, from the artefact to its proof, and from the proof to the process and the machine. What lies below is no longer a chain of trust but daily operation: seeing what each process does with Tetragon and measuring whether what is served is still good with evals. Trust is established once; vigilance is continuous.

Sources

CNCF, SPIFFE and SPIRE Projects Graduate from CNCF Incubator (20 sep. 2022) — https://www.cncf.io/announcements/2022/09/20/spiffe-and-spire-projects-graduate-from-cloud-native-computing-foundation-incubator/
SPIFFE, SPIFFE Concepts (SPIFFE ID, trust domain, SVID, Workload API) — https://spiffe.io/docs/latest/spiffe-about/spiffe-concepts/
SPIFFE, SPIFFE Federation specification — https://spiffe.io/docs/latest/spiffe-specs/spiffe_federation/
SPIFFE, Scaling SPIRE (dimensionamiento y punto único de fallo) — https://spiffe.io/docs/latest/planning/scaling_spire/
spiffe/spire, Releases (v1.15.0 de 19 may. 2026; v1.15.1 de 28 may. 2026, parche de azure_imds) — https://github.com/spiffe/spire/releases
spiffe/spire, Kubernetes Workload Attestor plugin (selectores, kubelet, cgroups, Sigstore) — https://github.com/spiffe/spire/blob/main/doc/plugin_agent_workloadattestor_k8s.md
spiffe/spire, SPIRE Server configuration reference (default_x509_svid_ttl, default_jwt_svid_ttl, ca_ttl) — https://github.com/spiffe/spire/blob/main/doc/spire_server.md
Istio, SPIRE integration (Envoy SDS, SPIFFE CSI Driver, trust domain) — https://istio.io/latest/docs/ops/integrations/spire/
SPIFFE, OPA Authorization with Envoy and JWT-SVIDs — https://spiffe.io/docs/latest/microservices/envoy-jwt-opa/readme/
The New Stack, How Cilium’s Mutual Authentication Can Compromise Security — https://thenewstack.io/how-ciliums-mutual-authentication-can-compromise-security/
Riptides, Bringing SPIFFE to OAuth for MCP — https://riptides.io/blog/bringing-spiffe-to-oauth-for-mcp-secure-identity-for-agentic-workloads/
CNCF, Confidential Containers becomes a CNCF incubating project (22 jul. 2026) — https://www.cncf.io/blog/2026/07/22/confidential-containers-becomes-a-cncf-incubating-project/
Confidential Containers, Attestation with Trustee (KBS, AS, RVPS, CDH) — https://confidentialcontainers.org/docs/attestation/
Confidential Containers, Deploy Trustee in Kubernetes (11 feb. 2026) — https://confidentialcontainers.org/blog/2026/02/11/deploy-trustee-in-kubernetes/
confidential-containers/trustee, Releases (v0.20.0: TLS 1.3, PQC, multi-GPU ITA, NVSwitch) — https://github.com/confidential-containers/trustee/releases
IETF, RFC 9334 — Remote ATtestation procedureS (RATS) Architecture — https://www.rfc-editor.org/info/rfc9334/
Kata Containers, Kata Containers 4.0.0 Preview (28 abr. 2026, runtime-rs por defecto) — https://katacontainers.io/blog/release-4-0-0-preview/
kata-containers, Release 3.32.0 — https://github.com/kata-containers/kata-containers/releases/tag/3.32.0
NVIDIA, Deploy with Kata Containers — GPU Operator — https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/deploy-kata-containers.html
NVIDIA, Deploying Proprietary Models Securely with Confidential Computing on Self-Hosted Kubernetes (GA 1.0.0; MIG y vGPU prohibidos en CC) — https://docs.nvidia.com/enterprise-reference-architectures/deploying-proprietary-models-confidential-compute-self-hosted-kubernetes/latest/reference-implementations.html
CACM, Creating the First Confidential GPUs (CPR, bounce buffers, contadores deshabilitados, HBM no cifrada) — https://cacm.acm.org/practice/creating-the-first-confidential-gpus/
NVIDIA, Confidential Computing on H100 GPUs for Secure and Trustworthy AI (límite de ~4 GB/s, NRAS) — https://developer.nvidia.com/blog/confidential-computing-on-h100-gpus-for-secure-and-trustworthy-ai/
NVIDIA, Hardware-Rooted AI Security That Won’t Slow You Down (2 jul. 2026; Blackwell, NVLink cifrado hasta 8 GPU) — https://developer.nvidia.com/blog/hardware-rooted-ai-security-that-wont-slow-you-down
Corvex, Confidential Computing Meets NVIDIA HGX B200 (claim comercial) — https://www.corvex.ai/blog/confidential-computing-meets-nvidia-hgxtm-b200-secure-ai-without-the-performance-trade-off
arXiv 2409.03992, Confidential Computing on NVIDIA Hopper GPUs: A Performance Benchmark Study — https://arxiv.org/html/2409.03992v1
arXiv 2509.18886, Confidential LLM Inference: Performance and Cost Across CPU and GPU TEEs (23 sep. 2025) — https://arxiv.org/abs/2509.18886
arXiv 2607.19353, Benchmarking Confidential GPU Inference on NVIDIA H100 under Intel TDX (may. 2026; heurística del 15-25 %) — https://arxiv.org/html/2607.19353
Segarra et al., Serverless Confidential Containers: Challenges and Opportunities (SESAME'24) — https://carlossegarra.com/assets/papers/sesame24-serverlesscoco.pdf
EU Artificial Intelligence Act, Article 15: Accuracy, Robustness and Cybersecurity — https://artificialintelligenceact.eu/article/15/
EU Artificial Intelligence Act, Article 12: Record-Keeping — https://artificialintelligenceact.eu/article/12/

Volcano and Kueue: gang scheduling, queues and GPU quotas for distributed workloads on Kubernetes

Tue, 16 Jun 2026 13:00:00 +0200

TL;DR

Volcano (volcano-sh, CNCF incubating) is a complete batch scheduler that replaces or complements the kube-scheduler: it places pods with gang semantics (all-or-nothing via PodGroup/minMember), manages queues with priority, DRF fair-share and preemption between queues, and understands network and NUMA topology.

Kueue (kubernetes-sigs/kueue) is a queue and quota manager at the Job level: it does NOT place pods (it delegates to the kube-scheduler or to Volcano), but it decides when a workload can be admitted according to available quota (ClusterQueue/LocalQueue/Cohort), with fair sharing, borrowing between teams and preemption by priority. It natively integrates Job, JobSet, RayJob, all the Kubeflow operators and more.

The winning combination in production for multi-tenant GPU workloads is: Kueue for quota and queues + Volcano (or the sig-scheduler coscheduling plugin) for the job’s gang.

The analogy

Imagine a club with limited capacity and a dance floor inside.

Kueue is the doorman and the booking manager: it checks whether the team’s quota (its reserved capacity) allows the group in, applies the fair waiting list, borrows capacity from other teams if any are idle, and takes the space back when the owner needs it. But the doorman does not decide where each person sits inside the venue.

Volcano is the head waiter: once the group has permission to come in, he decides which tables they sit at, makes sure the whole group sits down at once or nobody comes in (gang), picks the tables according to topology (who needs to talk to whom) and throws out lower-priority groups to make room if necessary.

Without a doorman (Kueue), the head waiter does not know how many groups he can take at once or whether a team is exceeding its capacity. Without a head waiter (Volcano), the doorman lets the group in but its members scatter across the available tables on their own, and the party of 8 that needs to sit together never manages it.

The problem neither solves by default: the kube-scheduler

Kubernetes’ kube-scheduler is a pod scheduler, not a job scheduler. It assigns pods one by one to the most suitable node according to available resources and affinity constraints. For a distributed training workload that needs, say, 8 pods with 4 GPUs each (32 GPUs in total across 8 nodes of 4×H100), the standard scheduler does the following:

It looks for a node with 4 available GPUs. It finds one. It schedules pod 1.
It looks for another node with 4 GPUs. It finds one. It schedules pod 2.
It carries on until it reaches pod 6 and it turns out there are no longer any nodes with 4 free GPUs: the cluster has exactly 32 GPUs and other workloads are using some of them.
Pods 1–5 are Running. Pods 6–8 are Pending.
Pods 1–5 can do nothing without the others: a distributed PyTorch job needs all the workers to start before the torchrun process can begin. It waits with the resources occupied. Deadlock.

This is not a bug, it is the design: the kube-scheduler has no concept of “schedule this group of pods only if you can schedule all of them”. As a result:

The resources of pods 1–5 are locked up without producing work.
Other jobs that could run with the partial resources also wait.
If several jobs are in this situation, the cluster can end up with fragmented resources, no job running and everyone in circular deadlock.

On top of that, the kube-scheduler has no notion of:

Queues per team or project with relative priority.
Quotas of resources per team with the ability to borrow from idle teams.
Fair-share: if team A has spent weeks using 80 % of the cluster, it should wait longer than team B, which has spent weeks idle.
Inter-queue preemption: evicting a lower-priority job from another team to make room for this team’s urgent job.

Solving any of these problems requires adding a layer on top of the scheduler. Volcano and Kueue are the two dominant OSS solutions in 2026, with complementary architectural approaches.

Volcano: the batch scheduler

What it is and what it replaces

Volcano (volcano-sh) is a Kubernetes-native batch scheduler accepted by the CNCF as its first and only official container batch scheduling project (volcano.sh/en/docs). At version v1.15.x as of June 2026, with CNCF incubating status.

Volcano is not an addon to the kube-scheduler: it is an alternative scheduler (or a complementary one, depending on the configuration) that places pods. It is installed as a deployment, and jobs that want to benefit from its capabilities must use the volcano scheduler class in their pod spec (schedulerName: volcano) or use the VolcanoJob CRD.

Volcano’s central value proposition is that it treats groups of pods as atomic scheduling units, not individual pods. That is what makes it possible to solve the deadlock described above.

Gang scheduling via PodGroup

Volcano’s central mechanism is the PodGroup: a CRD that groups a job’s pods and defines how many must be schedulable before Volcano starts any of them.

# PodGroup for a distributed PyTorch training job: 8 workers, minimum 8
apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
 name: pytorch-train-pg
 namespace: ml-training
spec:
 minMember: 8 # all-or-nothing: if there is no room for 8, none starts
 minResources:
 nvidia.com/gpu: "32" # 8 pods × 4 GPUs = 32 GPUs minimum in the cluster
 queue: team-datos  # which Volcano queue this job is assigned to
 priorityClassName: high-priority

The minMember parameter implements all-or-nothing semantics: Volcano only assigns nodes to the group’s pods when it can assign at least minMember pods simultaneously. If the cluster does not have capacity for 8 GPU pods at this moment, no pod in the group moves out of Pending. Nothing is blocked, nothing is fragmented.

minMember can be lower than the job’s total pod count: this allows elastic gang scheduling, where the job can start with fewer workers and scale up, useful for jobs that tolerate a reduced worker count.

For a job’s pods to be associated with the PodGroup, they carry the annotation:

# Pod spec of the PyTorch worker
metadata:
 annotations:
 scheduling.volcano.sh/pod-group-name: pytorch-train-pg
spec:
 schedulerName: volcano
 containers:
 - name: trainer
 image: pytorch/pytorch:2.5-cuda12.4
 resources:
 limits:
 nvidia.com/gpu: "4"

Queue: queues with quotas and priority

Volcano introduces the Queue CRD to manage multiple tenants with independent quotas:

apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
 name: team-datos
spec:
 weight: 4 # relative weight for fair-share between queues (proportion plugin)
 capability: # absolute ceiling of resources this queue can use
 nvidia.com/gpu: "16"
 guarantee: # guaranteed resource, never lent to other queues
 resource:
 nvidia.com/gpu: "8"
 reclaimable: true # if true, others can reclaim the resources it lends when they need them

apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
 name: team-ia
spec:
 weight: 2
 capability:
 nvidia.com/gpu: "8"
 guarantee:
 resource:
 nvidia.com/gpu: "4"
 reclaimable: true

The weight field feeds the proportion plugin: queues compete for the cluster’s available resources in proportion to their weight. A cluster with 32 GPUs and two queues of weight 4 and 2 splits the GPUs in a 4:2 ratio (≈21 and 11 GPUs respectively) when both are saturated.

Scheduler plugins: DRF, binpack, topology-aware

Volcano implements its scheduling logic as a pipeline of actions and plugins:

Actions (what the scheduler does in each cycle):

enqueue: moves jobs from the waiting queue to schedulable when quota is available.
allocate: assigns nodes to schedulable pods.
preempt: evicts lower-priority pods to make room for higher-priority ones within the same queue.
reclaim: evicts pods from other queues that are using more than their guarantee to return resources to the owner.
backfill: fills idle resources with best-effort jobs that do not interfere with the rest.

Plugins relevant to GPU workloads:

Plugin	What it does
`gang`	Implements the PodGroup’s all-or-nothing semantics
`proportion`	Fair-share by queue weight (proportional quota)
`capacity`	Quotas with `guarantee`/`capability` and reclaim; a more expressive alternative to proportion
`drf`	Dominant Resource Fairness: multi-dimensional fair-share (CPU, memory, GPU)
`binpack`	Packs pods onto the fullest nodes; reduces GPU fragmentation
`priority`	Orders jobs by priority within the same queue
`nodeorder`	Node scoring according to multiple criteria (affinity, resources, spread)
`task-topology`	Affinity between pods of the same job (inter-GPU communication)
`numa-aware`	NUMA affinity: aligns pods with the node’s NUMA socket to reduce memory latency

Network and NUMA topology (v1.11+)

Volcano v1.11 (February 2025) introduced Network Topology Aware Scheduling as a first-class feature (CNCF blog, March 2025). Distributed training jobs in a datacenter with a hierarchical network structure (spine/leaf, blocks of nodes with NVSwitch) can declare topology constraints:

# VolcanoJob with a network topology constraint
apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
 name: llm-pretrain
 namespace: ml-training
spec:
 minAvailable: 8
 schedulerName: volcano
 queue: team-datos
 plugins:
 ssh: []
 env: []
 svc: []
 networkTopology:
 mode: hard # hard: the job MUST satisfy the constraint
 highestTierAllowed: block  # pods cannot span beyond a network "block"
 tasks:
 - replicas: 8
 name: worker
 template:
 spec:
 containers:
 - name: trainer
 image: nvcr.io/nvidia/pytorch:25.01-py3
 resources:
 limits:
 nvidia.com/gpu: "4"

The highestTierAllowed: block semantics instruct Volcano to place the 8 pods within the same network block (for example, all the nodes under the same access switch), minimising the inter-block traffic that degrades a distributed all-reduce.

NUMA awareness works in a similar way: with the numa-aware plugin, pods request a NUMA policy (single-numa-node, restricted, best-effort) and Volcano selects nodes where the requested CPU, memory and GPU resources are in the same NUMA domain, avoiding the remote memory access overhead (NUMA-crossing) that can degrade training throughput by 15-30 % on multi-socket nodes.

GPU virtualization in v1.11+

Volcano v1.11 also introduces support for dynamic MIG and vCUDA: instead of declaring nvidia.com/gpu: 1 for a whole GPU, workloads can declare nvidia.com/gpu-memory: 20Gi and Volcano (with the corresponding device plugin) dynamically provisions the MIG instance or the vCUDA partition. This is [project marketing with no independent benchmarks published as of June 2026], but the feature’s architecture is documented in the code.

What it replaces or adds to the default scheduler

Capability	kube-scheduler	Volcano
Placing pods on nodes	Yes	Yes (it replaces it for workloads marked with `schedulerName: volcano`)
Gang scheduling (all-or-nothing)	No	Yes (PodGroup + minMember)
Queues with priority	No	Yes (Queue CRD)
Inter-queue fair-share	No	Yes (DRF, proportion, capacity plugins)
Inter-queue preemption	No	Yes (reclaim action)
Topology-aware (NUMA, network)	Partial (node affinity)	Yes (dedicated plugins)
Elastic gang (minMember < total)	No	Yes
Best-effort backfill	No	Yes
Framework integrations	Partial	MPI, PyTorch, Ray, TensorFlow, Spark, Flink, Horovod

Kueue: the queue and quota manager

What it is and what it does NOT do

Kueue (kubernetes-sigs/kueue) is a Kubernetes-native system that manages quotas and how jobs consume them (kueue.sigs.k8s.io). Kueue decides when a job must wait, when it should be admitted (pods can be created) and when it should be evicted (active pods must be deleted).

Kueue’s central design principle is explicit in its documentation: avoid duplicating mature functionality of Kubernetes components. Autoscaling is the cluster-autoscaler’s responsibility. Pod-to-node scheduling is the kube-scheduler’s responsibility. Job lifecycle management is the kube-controller-manager’s responsibility. Kueue replaces none of them: it sits on top as a layer of admission control and quota management at the Job level.

This is the fundamental distinction: Kueue does not place pods on nodes. When Kueue admits a workload, it simply allows the corresponding job controller to create the pods, and those pods are scheduled by the kube-scheduler (or by Volcano, if it is configured as the scheduler).

The four core objects

ResourceFlavor: maps abstract resources to concrete groups of physical nodes.

apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
 name: h100-sxm
spec:
 nodeLabels:
 accelerator: h100-sxm
 node-pool: gpu-training
 tolerations:
 - key: "nvidia.com/gpu"
 operator: "Exists"
 effect: "NoSchedule"

ClusterQueue: defines the resource quota per flavor for a tenant. A cluster-scoped object.

apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
 name: cq-team-datos
spec:
 cohort: llm-platform  # cohort it belongs to (can lend/borrow)
 queueingStrategy: BestEffortFIFO
 namespaceSelector:
 matchLabels:
 kubernetes.io/metadata.name: ns-datos
 resourceGroups:
 - coveredResources: ["nvidia.com/gpu", "cpu", "memory"]
 flavors:
 - name: h100-sxm
 resources:
 - name: "nvidia.com/gpu"
 nominalQuota: 16 # GPUs guaranteed for this team
 borrowingLimit: 8 # can take up to 8 additional GPUs from the cohort
 lendingLimit: 8 # can lend up to 8 of its 16 nominal GPUs
 - name: "cpu"
 nominalQuota: "128"
 - name: "memory"
 nominalQuota: "512Gi"
 preemption:
 reclaimWithinCohort: LowerPriority  # reclaims lent quota by evicting lower-priority jobs
 borrowWithinCohort:
 policy: LowerPriority
 withinClusterQueue: LowerPriority

LocalQueue: the namespace-scoped entry point for a team’s workloads. Jobs point to their LocalQueue; Kueue maps them to the corresponding ClusterQueue.

apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
 name: lq-datos
 namespace: ns-datos
spec:
 clusterQueue: cq-team-datos

Cohort: groups ClusterQueues that can lend quota to each other. It is not a standalone CRD; it is declared as a field in the ClusterQueue (spec.cohort: name). Kueue aggregates the available quota of all the ClusterQueues in the cohort and lets any of them borrow what the others are not using, respecting the borrowingLimit and lendingLimit.

Kueue implements Fair Sharing as the ordering policy for the queue of pending workloads (kueue.sigs.k8s.io/docs/concepts/fair_sharing): when several workloads compete for quota in the cohort, those belonging to ClusterQueues with higher accumulated historical usage have lower admission priority. This implements equitable sharing without permanently blocking any team.

Preemption in Kueue operates along two dimensions:

reclaimWithinCohort: the ClusterQueue that lends quota reclaims it by evicting workloads that are using it on loan, according to a priority policy.
withinClusterQueue: within the same ClusterQueue, lower-priority workloads are evicted to make way for higher-priority workloads from the same team.

Gang semantics in Kueue: all-or-nothing with ready Pods

Kueue provides gang admission at the Job level: it admits the complete workload only when all the necessary quota is available. If a RayJob needs 8 GPUs (1 head + 7 workers), Kueue does not admit the workload until 8 GPUs are available in the ClusterQueue (or borrowed from the cohort). waitForPodsReady with a timeout adds a second guarantee: if the created pods do not become Ready within the configured time, Kueue re-queues the workload and releases the quota (kueue.sigs.k8s.io/docs/tasks/manage/setup_wait_for_pods_ready).

This is gang semantics at the admission level, not at the pod placement level. It guarantees that the quota is available before creating the pods, but it does not guarantee that the kube-scheduler can place them all on concrete nodes at the same time. For that second guarantee you need Volcano or the coscheduling plugin.

Topology-Aware Scheduling (TAS)

Kueue v0.10+ introduces Topology-Aware Scheduling (kueue.sigs.k8s.io/docs/concepts/topology_aware_scheduling): it allows node topologies (blocks, subblocks, hosts) to be defined and workloads to request co-location levels. Kueue only admits the workload when it can satisfy the topological constraint, and it adds node selectors and taints at admission time so that the scheduler places the pods in the right topology.

TAS is configured with the Topology CRD:

apiVersion: kueue.x-k8s.io/v1beta1
kind: Topology
metadata:
 name: datacenter-topology
spec:
 levels:
 - nodeLabel: "topology.kubernetes.io/block"
 - nodeLabel: "topology.kubernetes.io/rack"
 - nodeLabel: "kubernetes.io/hostname"

And the ResourceFlavor references the topology:

apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
 name: h100-sxm
spec:
 nodeLabels:
 accelerator: h100-sxm
 topologyName: datacenter-topology

Framework integrations

Kueue has built-in integration (no additional code) for the following workload types, activated with an annotation on the job:

metadata:
 labels:
 kueue.x-k8s.io/queue-name: lq-datos  # points to the team's LocalQueue

The supported workload types include: batch/Job, JobSet, RayJob, RayCluster, PytorchJob, TFJob, MPIJob, JAXJob, PaddleJob, XGBoostJob, TrainJob, AppWrapper, LeaderWorkerSet, Deployment, StatefulSet, and plain Pod/PodGroup.

For LLM workloads, the directly relevant cases:

RayJob (distributed training with Ray Train): Kueue admits the RayJob when there is quota for the whole Ray cluster (head + workers). Documented at docs.ray.io.
PyTorchJob (Kubeflow Training Operator): gang admission of the complete job.
JobSet: for coordinated multi-replica jobs (LWS, multi-step pipelines).
Deployment/StatefulSet: for continuous inference, allowing inference GPU quota to be managed the same way as training quota.

The key distinction: Volcano places, Kueue admits

This table sums up the fundamental architectural difference:

Dimension	Volcano	Kueue
Main role	Scheduler (places pods on nodes)	Admission controller + quota manager (decides when to create pods)
Gang scheduling	Yes, at the placement level (PodGroup/minMember)	Yes, at the admission level (all-or-nothing on quota)
Multi-tenant quotas	Yes (Queue with capability/guarantee)	Yes (ClusterQueue with nominalQuota/borrowingLimit)
Cohorts / borrowing	Partial (reclaimable between queues)	Yes (Cohort with explicit lendingLimit/borrowingLimit)
Fair sharing	Yes (DRF plugin)	Yes (Fair Sharing based on historical usage)
Preemption	Yes (preempt + reclaim actions)	Yes (reclaimWithinCohort, withinClusterQueue)
Topology/NUMA	Yes (dedicated plugins, network topology, NUMA-aware)	Yes (TAS, topology levels in ResourceFlavor)
Framework integrations	Volcano Job (MPI, PyTorch, Ray, TF, Spark, Flink)	Native: Job, JobSet, RayJob, Kubeflow, LWS, AppWrapper, Deployment, StatefulSet
Who places the pods	Volcano	kube-scheduler (or Volcano if configured)
Installation footprint	Medium-high (its own scheduler, CRDs, webhook, metrics)	Light (controller, CRDs, webhook; does not replace the scheduler)
Maturity / status	CNCF incubating; v1.15 (June 2026); production at Huawei, Baidu, DiDi	kubernetes-sigs; API v1beta2; production at Google GKE, Red Hat OpenShift 4.20, Runway ML
Adoption curve	Steeper (requires changing schedulerName or using the VolcanoJob CRD)	Gentler (adds labels to existing jobs; does not change the scheduler)

How they coexist: the production pattern

In production, Kueue and Volcano are complementary, not mutually exclusive. The most common pattern in 2026 for multi-tenant GPU clusters is:

Kueue manages the global quota: which team can use how many GPUs, how much it can borrow, when a job goes into a queue versus being admitted.
Volcano does the gang scheduling at the pod level: once Kueue admits the job (the quota is available), Volcano places the pods, making sure they are all placed simultaneously on compatible nodes.

The integration is configured by specifying schedulerName: volcano in the pod specs of the workloads managed by Kueue. Kueue sees the Job/RayJob/PyTorchJob and manages its quota; when it admits it, the pods are created and Volcano places them with gang semantics. Volcano’s PodGroups are created automatically by the Volcano Job controller or by the Kubeflow Training Operator itself when it detects that the scheduler is Volcano.

# PyTorchJob managed by Kueue (quota) + Volcano (gang placement)
apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
 name: llm-finetune-70b
 namespace: ns-datos
 labels:
 kueue.x-k8s.io/queue-name: lq-datos  # Kueue manages the quota
 annotations:
 scheduling.volcano.sh/queue-name: team-datos  # Volcano uses its own Queue
spec:
 pytorchReplicaSpecs:
 Master:
 replicas: 1
 restartPolicy: OnFailure
 template:
 spec:
 schedulerName: volcano  # Volcano does the placement
 containers:
 - name: pytorch
 image: nvcr.io/nvidia/pytorch:25.01-py3
 resources:
 limits:
 nvidia.com/gpu: "4"
 Worker:
 replicas: 7
 restartPolicy: OnFailure
 template:
 spec:
 schedulerName: volcano
 containers:
 - name: pytorch
 image: nvcr.io/nvidia/pytorch:25.01-py3
 resources:
 limits:
 nvidia.com/gpu: "4"

The third way: the sig-scheduler coscheduling plugin

If you do not want to deploy a complete alternative scheduler but you need gang scheduling, there is the coscheduling plugin from kubernetes-sigs/scheduler-plugins. This plugin extends the kube-scheduler with a PodGroup mechanism similar to Volcano’s, implemented as a scheduling framework plugin (permit plugin). The advantage is that it does not replace the scheduler; the disadvantage is that it has less functionality than Volcano (no DRF, no Queue/fair-share, no network topology). It is the right option for simple clusters that only need gang and do not want Volcano’s operational complexity. Kueue can also work alongside this plugin.

Complete YAML examples

Volcano: Queue + PodGroup + VolcanoJob

# 1. Queue for the data team (16 nominal GPUs, ceiling at 24)
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
 name: team-datos
spec:
 weight: 4
 capability:
 nvidia.com/gpu: "24"
 cpu: "192"
 memory: "768Gi"
 guarantee:
 resource:
 nvidia.com/gpu: "16"
 reclaimable: true
---
# 2. Queue for the AI team (8 nominal GPUs, ceiling at 16)
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
 name: team-ia
spec:
 weight: 2
 capability:
 nvidia.com/gpu: "16"
 cpu: "64"
 memory: "256Gi"
 guarantee:
 resource:
 nvidia.com/gpu: "8"
 reclaimable: true
---
# 3. PodGroup: 70B fine-tuning job, 8 workers × 4 GPUs = 32 GPUs
apiVersion: scheduling.volcano.sh/v1beta1
kind: PodGroup
metadata:
 name: finetune-70b-pg
 namespace: ns-datos
spec:
 minMember: 8
 minResources:
 nvidia.com/gpu: "32"
 queue: team-datos
 priorityClassName: training-high
---
# 4. VolcanoJob (wrapper that Volcano understands natively)
apiVersion: batch.volcano.sh/v1alpha1
kind: Job
metadata:
 name: finetune-70b
 namespace: ns-datos
spec:
 minAvailable: 8
 schedulerName: volcano
 queue: team-datos
 priorityClassName: training-high
 plugins:
 env: []
 svc: []
 policies:
 - event: PodEvicted
 action: RestartJob
 tasks:
 - replicas: 8
 name: worker
 policies:
 - event: TaskCompleted
 action: CompleteJob
 template:
 metadata:
 annotations:
 scheduling.volcano.sh/pod-group-name: finetune-70b-pg
 spec:
 schedulerName: volcano
 containers:
 - name: trainer
 image: nvcr.io/nvidia/pytorch:25.01-py3
 command: ["torchrun", "--nproc_per_node=4", "--nnodes=8",
 "--node_rank=$(RANK)", "--master_addr=$(MASTER_ADDR)",
 "--master_port=23456", "train.py"]
 env:
 - name: NCCL_DEBUG
 value: "INFO"
 resources:
 requests:
 nvidia.com/gpu: "4"
 cpu: "16"
 memory: "64Gi"
 limits:
 nvidia.com/gpu: "4"
 cpu: "16"
 memory: "64Gi"
 restartPolicy: Never

Kueue: ResourceFlavor + ClusterQueue + LocalQueue + annotated Job

# 1. ResourceFlavor: nodes with H100 SXM
apiVersion: kueue.x-k8s.io/v1beta1
kind: ResourceFlavor
metadata:
 name: h100-sxm
spec:
 nodeLabels:
 accelerator: h100-sxm
 tolerations:
 - key: "nvidia.com/gpu"
 operator: "Exists"
 effect: "NoSchedule"
---
# 2. ClusterQueue for the data team: 16 nominal GPUs, can borrow 8
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
 name: cq-team-datos
spec:
 cohort: llm-platform
 queueingStrategy: BestEffortFIFO
 namespaceSelector:
 matchLabels:
 kubernetes.io/metadata.name: ns-datos
 resourceGroups:
 - coveredResources: ["nvidia.com/gpu", "cpu", "memory"]
 flavors:
 - name: h100-sxm
 resources:
 - name: "nvidia.com/gpu"
 nominalQuota: 16
 borrowingLimit: 8
 lendingLimit: 8
 - name: "cpu"
 nominalQuota: "128"
 - name: "memory"
 nominalQuota: "512Gi"
 preemption:
 reclaimWithinCohort: LowerPriority
 borrowWithinCohort:
 policy: LowerPriority
 withinClusterQueue: LowerPriority
---
# 3. ClusterQueue for the AI team: 8 nominal GPUs
apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
 name: cq-team-ia
spec:
 cohort: llm-platform
 queueingStrategy: BestEffortFIFO
 namespaceSelector:
 matchLabels:
 kubernetes.io/metadata.name: ns-ia
 resourceGroups:
 - coveredResources: ["nvidia.com/gpu", "cpu", "memory"]
 flavors:
 - name: h100-sxm
 resources:
 - name: "nvidia.com/gpu"
 nominalQuota: 8
 borrowingLimit: 8
 lendingLimit: 4
 - name: "cpu"
 nominalQuota: "64"
 - name: "memory"
 nominalQuota: "256Gi"
 preemption:
 reclaimWithinCohort: LowerPriority
 withinClusterQueue: LowerPriority
---
# 4. LocalQueue in the data team's namespace
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
 name: lq-datos
 namespace: ns-datos
spec:
 clusterQueue: cq-team-datos
---
# 5. LocalQueue in the AI team's namespace
apiVersion: kueue.x-k8s.io/v1beta1
kind: LocalQueue
metadata:
 name: lq-ia
 namespace: ns-ia
spec:
 clusterQueue: cq-team-ia
---
# 6. Batch inference RayJob managed by Kueue
apiVersion: ray.io/v1
kind: RayJob
metadata:
 name: batch-eval-llama70b
 namespace: ns-datos
 labels:
 kueue.x-k8s.io/queue-name: lq-datos  # Kueue manages the admission
spec:
 entrypoint: "python batch_eval.py --model /models/llama-70b"
 rayClusterSpec:
 headGroupSpec:
 rayStartParams:
 num-gpus: "4"
 template:
 spec:
 containers:
 - name: ray-head
 image: rayproject/ray-ml:2.40.0-gpu
 resources:
 limits:
 nvidia.com/gpu: "4"
 cpu: "16"
 memory: "64Gi"
 workerGroupSpecs:
 - replicas: 3
 minReplicas: 3 # gang: Kueue does not admit if there is no quota for 3 workers
 maxReplicas: 3
 groupName: gpu-worker
 rayStartParams:
 num-gpus: "4"
 template:
 spec:
 containers:
 - name: ray-worker
 image: rayproject/ray-ml:2.40.0-gpu
 resources:
 limits:
 nvidia.com/gpu: "4"
 cpu: "16"
 memory: "64Gi"

Kueue + Volcano together: PyTorchJob with Kueue quota and Volcano gang placement

# PyTorchJob: Kueue controls the quota, Volcano does the gang placement
apiVersion: kubeflow.org/v1
kind: PyTorchJob
metadata:
 name: distributed-finetune
 namespace: ns-datos
 labels:
 kueue.x-k8s.io/queue-name: lq-datos # Kueue: quota and admission
 annotations:
 # Volcano automatically creates the PodGroup when schedulerName=volcano
 scheduling.volcano.sh/queue-name: team-datos
spec:
 pytorchReplicaSpecs:
 Master:
 replicas: 1
 restartPolicy: OnFailure
 template:
 spec:
 schedulerName: volcano # Volcano: gang placement
 containers:
 - name: pytorch
 image: nvcr.io/nvidia/pytorch:25.01-py3
 resources:
 limits:
 nvidia.com/gpu: "4"
 cpu: "16"
 memory: "64Gi"
 Worker:
 replicas: 7
 restartPolicy: OnFailure
 template:
 spec:
 schedulerName: volcano
 containers:
 - name: pytorch
 image: nvcr.io/nvidia/pytorch:25.01-py3
 resources:
 limits:
 nvidia.com/gpu: "4"
 cpu: "16"
 memory: "64Gi"

For LLM workloads: training, fine-tuning and batch inference

Distributed multi-GPU training and fine-tuning

Gang scheduling is essential for any distributed training job that uses NCCL all-reduce (PyTorch DDP, FSDP, DeepSpeed ZeRO). If a single worker in the group does not start, the torchrun coordinator waits indefinitely; with the standard kube-scheduler this scenario happens every time the cluster is under contention.

In a generic cluster of 4 nodes of 4×H100 SXM (16 GPUs in total), a fine-tuning job for a 70B model typically requires 8 GPUs in tensor-parallel 8 (TP=8) or 16 GPUs in TP=4 plus data-parallel 4. With Volcano, the PodGroup with minMember: 8 guarantees that either the 8 pods are placed at once or none blocks resources. With Kueue on top, the quota guarantees that the team does not exceed its 16 nominal GPUs and that other teams with available quota are not blocked by a waiting job.

The cross-link with inference capacity planning is direct: the model’s VRAM budget (weights plus KV-cache) determines the minimum TP and therefore the PodGroup’s minMember.

Batch inference and evaluations (evals)

Batch inference jobs, generating responses for an evaluation dataset, processing embeddings in bulk, offline re-ranking, are naturally parallel workloads that do not necessarily require strict gang scheduling (each request is independent), but they do benefit from quota and fair-share.

For these workloads, Kueue alone is enough: a batch/Job with multiple independent pods is managed with the ClusterQueue’s quota with no need for Volcano. If several teams are submitting evaluation jobs simultaneously, Kueue orders the admission by fair-share and priority, and with cohort borrowing the jobs of teams with free quota do not have to wait for the quota of busy teams.

The chargeback of these workloads connects directly with what is described in GPU chargeback and showback: the ClusterQueue’s nominalQuota is the expression of the GPU budget in Kubernetes, and OpenCost can attribute the cost per namespace or label for the monthly report.

Multi-tenant GPU quota and chargeback

The alignment between Kueue and the chargeback system is direct:

FinOps concept	Kueue mechanism
Guaranteed GPU budget	`nominalQuota` per ClusterQueue
Maximum spending ceiling	`nominalQuota + borrowingLimit`
Lending idle capacity	Cohort + `lendingLimit`
Reclaiming your own quota	`preemption.reclaimWithinCohort: LowerPriority`
Fair-share between teams	Fair Sharing policy in the ClusterQueue
Chargeback of the loan	borrowed GPU hours × cost per GPU-hour (OpenCost)

For the utilisation dimension as a FinOps lever, see GPU utilisation as a FinOps lever: Kueue and Volcano together make it possible to maximise utilisation without sacrificing the quota guarantees, which is exactly the FinOps objective.

Managing MIG partitions within this system (declaring nvidia.com/mig-4g.40gb as a resource in the ClusterQueue) integrates naturally: the ResourceFlavor can map to nodes with a specific MIG profile, as explained in sharing a GPU: time-slicing, MPS and MIG.

Diagram: the flow of a training job with Kueue + Volcano

Full comparison table

Criterion	Volcano	Kueue	Coscheduling plugin
Role	Scheduler (placement)	Admission + quota (no placement)	kube-scheduler plugin (placement)
Gang scheduling	Yes, pod level (PodGroup/minMember)	Yes, admission level (quota gang)	Yes, pod level (PodGroup)
Multi-tenant quotas	Yes (Queue capability/guarantee)	Yes (ClusterQueue nominalQuota)	No
Cohorts / borrowing	Limited (reclaimable)	Yes (Cohort with borrowingLimit/lendingLimit)	No
Fair-share	Yes (DRF, proportion)	Yes (Fair Sharing by historical usage)	No
Inter-queue preemption	Yes (reclaim action)	Yes (reclaimWithinCohort)	No
Network / NUMA topology	Yes (v1.11+, dedicated plugins)	Yes (TAS, Topology CRD)	No
Native integrations	MPI, PyTorch, Ray, TF, Spark, Flink, Horovod	Job, JobSet, RayJob, Kubeflow, LWS, AppWrapper, Deployment, StatefulSet	Any job with a PodGroup
Who places the pods	Volcano	kube-scheduler (or Volcano)	kube-scheduler
Elastic gang	Yes (minMember < total replicas)	Partial (partial admission in batch/Job)	Limited
Footprint	Medium-high (its own scheduler)	Light (an additional controller)	Minimal (a scheduler plugin)
Compatibility with Kueue	Yes (as the scheduler under Kueue)	—	Yes (a complementary option)
CNCF status / maturity	CNCF incubating, v1.15	kubernetes-sigs, v1beta2, adopted in GKE/OpenShift	kubernetes-sigs/scheduler-plugins, experimental
When to choose it	HPC-like distributed training, NUMA, network topology, MPI	Multi-tenancy with flexible quota, heterogeneous workloads, inference + batch together	Simple clusters that only need gang without their own scheduler

Operational pitfalls and honest scepticism

1. Deadlock from a badly configured gang

The most frequent scenario: minMember set equal to the total replica count in a cluster where several jobs compete for the same resources. If two jobs of 8 pods each try to use a cluster with 8 GPU nodes and job A has 4 pods placed (not 8, so Volcano holds them pending), and job B also has 4 pods held, nobody moves forward. Volcano is doing its job properly: it places none of them until there is room for all 8. But if the queues’ nominalQuota values are badly sized against the cluster’s real capacity, this produces indefinite waits.

Solution: size the queue quotas so that the sum of guarantee does not exceed the real capacity, and so that the minMember values of the active jobs fit inside the available quota. Node autoscaling with ProvisioningRequest (Kueue + cluster-autoscaler) helps, but it introduces provisioning latency that has to be taken into account in the job’s SLA.

2. Quota versus real capacity: the silent drift

Kueue’s nominalQuota and Volcano’s guarantee are administrative declarations. They do not guarantee that the nodes with those GPUs are available, healthy or that the device plugin has registered them correctly. A node in NotReady with 4 GPUs reduces the real capacity without Kueue knowing: the ClusterQueue will carry on admitting workloads that then cannot be placed.

Recommended monitoring: cross Kueue’s metrics (kueue_admitted_workloads_total, kueue_pending_workloads) with the cluster’s real capacity metrics (GPUs registered in the device plugin) to detect the drift. Kueue exposes native Prometheus metrics; so does Volcano.

3. GPU resource naming: MIG, time-slicing and ResourceFlavor

If the cluster uses MIG, the resources in the pod specs change from nvidia.com/gpu to nvidia.com/mig-Xg.Ygb (for example, nvidia.com/mig-3g.40gb). Kueue’s ResourceFlavors and Volcano’s Queues must declare the right resource, or the quota will not match the pods. With time-slicing, the resource is still nvidia.com/gpu but the device plugin advertises more instances than there are physical GPUs; the quota is expressed in virtual replicas, which can lead to over-admission if the VRAM budget is not taken into account (see sharing a GPU: time-slicing, MPS and MIG).

4. Preemption in production: the evicted workload loses progress

When Kueue or Volcano evicts a training job that has been running for hours, that job loses its progress if it has no checkpointing configured. The preemption is correct from the quota point of view, but it destroys work if the job is not prepared for it. Before enabling aggressive preemption, verify that all training jobs have periodic checkpointing with automatic restoration. PyTorch plus Torchrun have native support; so does DeepSpeed. Stateless batch inference jobs do not have this problem.

5. Volcano as the sole scheduler versus coexistence with the kube-scheduler

Volcano can be configured as the cluster’s default scheduler (all pods go through it) or as an alternative scheduler (only pods with schedulerName: volcano). The first option simplifies the configuration but breaks system pods that assume kube-scheduler behaviour. The second, the recommended one, requires ML jobs to explicitly set schedulerName: volcano, which can be a non-trivial operator or chart change for existing workloads. Kueue solves this more transparently: it only requires a label on the job, with no change of scheduler.

6. Real operational complexity in 2026

Running Kueue plus Volcano plus the Training Operator plus the GPU Operator in production means four components with their own CRDs, webhooks, versions and release cycles. A Kubernetes upgrade may require updating all four in sequence. The operational debt is real. For a small team without the capacity to maintain this stack, a managed Kubernetes provider (GKE with native Kueue, OpenShift with the Red Hat build of Kueue) may be more pragmatic than assembling the full stack from scratch.

The sig-scheduler coscheduling plugin is a deliberately simpler option when all you need is gang: fewer features, less complexity, fewer things to maintain.

Sources

Volcano — Introducción y features (CNCF incubating, v1.15) — https://volcano.sh/en/docs/
Volcano — Volcano v1.11 Released: A New Era of Cloud-Native Scheduling (CNCF blog, marzo 2025) — https://www.cncf.io/blog/2025/03/05/volcano-v1-11-released-a-new-era-of-cloud-native-scheduling-for-ai-and-big-data/
Volcano — Release v1.11.0 (GitHub) — https://github.com/volcano-sh/volcano/releases/tag/v1.11.0
Volcano — Network Topology Aware Scheduling design doc — https://github.com/volcano-sh/volcano/blob/master/docs/design/Network%20Topology%20Aware%20Scheduling.md
Volcano — NUMA-aware scheduling design — https://github.com/volcano-sh/volcano/blob/master/docs/design/numa-aware.md
Volcano — Unified Scheduling (docs oficiales) — https://volcano.sh/en/docs/unified_scheduling/
Volcano — Capacity scheduling design — https://github.com/volcano-sh/volcano/blob/master/docs/design/capacity-scheduling.md
NVIDIA Technical Blog — Practical Tips for Preventing GPU Fragmentation for Volcano Scheduler — https://developer.nvidia.com/blog/practical-tips-for-preventing-gpu-fragmentation-for-volcano-scheduler/
Kueue — Overview (kueue.sigs.k8s.io, actualizado febrero 2026) — https://kueue.sigs.k8s.io/docs/overview/
Kueue — Cluster Queue (nominalQuota, borrowingLimit, lendingLimit, preemption) — https://kueue.sigs.k8s.io/docs/concepts/cluster_queue/
Kueue — Cohort — https://kueue.sigs.k8s.io/docs/concepts/cohort/
Kueue — Fair Sharing — https://kueue.sigs.k8s.io/docs/concepts/fair_sharing/
Kueue — Topology Aware Scheduling — https://kueue.sigs.k8s.io/docs/concepts/topology_aware_scheduling/
Kueue — Preemption — https://kueue.sigs.k8s.io/docs/concepts/preemption/
Kueue — Setup All-or-nothing with ready Pods — https://kueue.sigs.k8s.io/docs/tasks/manage/setup_wait_for_pods_ready/
Kueue — GitHub (kubernetes-sigs/kueue) — https://github.com/kubernetes-sigs/kueue
Ray — Gang Scheduling with RayJob and Kueue — https://docs.ray.io/en/latest/cluster/kubernetes/examples/rayjob-kueue-gang-scheduling.html
Kubeflow — Volcano scheduler integration — https://www.kubeflow.org/docs/components/trainer/operator-guides/job-scheduling/volcano/
kubernetes-sigs/scheduler-plugins — Coscheduling plugin README — https://github.com/kubernetes-sigs/scheduler-plugins/blob/master/pkg/coscheduling/README.md
kubernetes/enhancements — KEP-583 Coscheduling (sig-scheduling) — https://github.com/kubernetes/enhancements/tree/master/keps/sig-scheduling/583-coscheduling
InfraCloud — Batch Scheduling on Kubernetes: Comparing YuniKorn, Volcano and Kueue — https://www.infracloud.io/blogs/batch-scheduling-on-kubernetes/
Red Hat — Red Hat build of Kueue (OpenShift 4.20) — https://docs.redhat.com/en/documentation/openshift_container_platform/4.20/html/ai_workloads/red-hat-build-of-kueue
AceCloud — Multi GPU Orchestration in Kubernetes 2026: Kueue, Volcano, DRA — https://acecloud.ai/blog/multi-gpu-orchestration-kubernetes/
CloudOptimo — Kubernetes AI Infrastructure in 2026: GPU Scheduling and Production Realities — https://www.cloudoptimo.com/blog/kubernetes-ai-infrastructure-in-2026-gpu-scheduling-and-production-realities/

The reproducible harness: measuring cost, performance and energy in a single auditable experiment

Tue, 16 Jun 2026 12:00:00 +0200

Notation: amounts in euros (N €), decimals with a point, thousands with a thin space. The dollar sign is not used (on this site it is a formula delimiter).

TL;DR

The “data” series has produced three independent measurement axes: cost per million tokens (OpenCost + LiteLLM), performance under an SLO (GuideLLM + AIPerf) and energy per token (DCGM + Kepler). The problem is that the three have been measured in different articles, with different loads and at different times: they are not comparable with each other. This closing article describes the integrated harness that runs the three axes in the same experiment, on the same node (4×H100 SXM, a generic reference), with all the metadata fixed, the output in versioned JSON/CSV and an idempotent Kubernetes Job. The result is the 3-axis scorecard (€/1M tok, Wh/token, TTFT/ITL P99) that allows configurations to be compared on a multi-objective Pareto frontier and any figure to be audited with the bench to reproduce it.

Why the three axes must be measured together

The series’ opening post (The three axes) established the identity:

$$\text{CPM} = \frac{\text{cost/h}}{\text{throughput (tok/s)} \times 3.6 \times 10^{-3}}$$ $$\text{energy/token (Wh)} = \frac{\text{mean power (W)}}{\text{throughput (tok/s)} \times 3\,600}$$

Throughput is the common denominator. If it is measured in different experiments, different time, different load, different GPU temperature, then CPM and energy/token do not share a denominator: they are three anecdotes, not a scorecard. The harness captures them in the same time window, over the same load, with Prometheus windows aligned to the second. Only then is the scorecard row coherent by construction.

The second reason is reproducibility. The post Bias and reproducibility in benchmarking listed twelve biases that invalidate comparisons: an unpinned engine, an undeclared tokeniser, unfixed input/output lengths, missing warmup, a client outside the cluster. The harness removes all of them because the metadata are part of the Job, not of the documentation.

Architecture of the integrated bench

The harness has four layers. Each one is OSS, exports to Prometheus and lives in the same Kubernetes namespace:

Layer	Tool(s)	Primary metric	Export protocol
Performance	GuideLLM (SLO sweep) + AIPerf	TTFT P99, ITL P99, goodput (tok/s)	native JSON/CSV + `/metrics` OpenMetrics
Cost	OpenCost + LiteLLM proxy	CPM (€/1M tok), cost/request	REST API + Prometheus scrape
Energy (GPU)	DCGM Exporter	`DCGM_FI_DEV_POWER_USAGE` (W), `DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION` (mJ)	Prometheus DaemonSet
Energy (pod)	Kepler	`kepler_container_joules_total`, energy per pod	Prometheus DaemonSet

MLPerf Power is used as an external comparability reference: its published results, with hardware documented in detail, allow the harness figures to be calibrated for plausibility. The in-house bench does not aim to be an MLPerf submission, but to be reproducible on your own cluster.

The Kubernetes Job: the full YAML

The experiment runs as a versioned Kubernetes Job. All the relevant metadata are environment variables declared in the manifest: not in ad-hoc scripts, not in external documentation. The Job is idempotent (same name = same run) and leaves traces in the pod log and in the output volume.

apiVersion: batch/v1
kind: Job
metadata:
 name: bench-llama3-70b-fp8-h100x4-20260616
 namespace: benchmark
 labels:
 bench/model: llama3-70b
 bench/precision: fp8
 bench/engine: vllm-0.9.1
 bench/hardware: h100x4-sxm
 bench/isl: "1024"
 bench/osl: "256"
 bench/concurrency: "32"
 bench/tokenizer: meta-llama-3-tokenizer-v3
 bench/run-id: "20260616T1200"
spec:
 backoffLimit: 0
 template:
 spec:
 restartPolicy: Never
 serviceAccountName: bench-runner
 volumes:
 - name: results
 persistentVolumeClaim:
 claimName: bench-results-pvc
 initContainers:
 # Warmup: 60 s of traffic before the experiment
 - name: warmup
 image: ghcr.io/vllm-project/guidellm:0.4.2
 command:
 - guidellm
 - benchmark
 - --target
 - http://vllm-svc.inference.svc.cluster.local:8000
 - --rate-type
 - concurrent
 - --rate
 - "4"
 - --max-seconds
 - "60"
 - --data
 - prompt_tokens=1024,output_tokens=256
 env:
 - name: GUIDELLM_ENV
 value: production
 containers:
 - name: guidellm-sweep
 image: ghcr.io/vllm-project/guidellm:0.4.2
 command:
 - guidellm
 - benchmark
 - --target
 - http://vllm-svc.inference.svc.cluster.local:8000
 - --rate-type
 - sweep
 - --max-seconds
 - "120"
 - --data
 - prompt_tokens=1024,output_tokens=256
 - --output-path
 - /results/guidellm-$(BENCH_RUN_ID).json
 env:
 - name: BENCH_RUN_ID
 valueFrom:
 fieldRef:
 fieldPath: metadata.labels['bench/run-id']
 volumeMounts:
 - name: results
 mountPath: /results

 - name: aiperf-concurrent
 image: nvcr.io/nvidia/aiperf:0.2.0
 command:
 - aiperf
 - profile
 - --url
 - http://vllm-svc.inference.svc.cluster.local:8000/v1
 - --model
 - meta-llama/Meta-Llama-3-70B-Instruct
 - --concurrency
 - "4,8,16,32"
 - --input-tokens
 - "1024"
 - --output-tokens
 - "256"
 - --num-requests
 - "200"
 - --output
 - /results/aiperf-$(BENCH_RUN_ID).json
 volumeMounts:
 - name: results
 mountPath: /results

 - name: scorecard-exporter
 image: python:3.12-slim
 command:
 - python
 - /scripts/export_scorecard.py
 - --run-id
 - "$(BENCH_RUN_ID)"
 - --prometheus
 - http://prometheus.monitoring.svc.cluster.local:9090
 - --output
 - /results/scorecard-$(BENCH_RUN_ID).json
 env:
 - name: BENCH_RUN_ID
 valueFrom:
 fieldRef:
 fieldPath: metadata.labels['bench/run-id']
 volumeMounts:
 - name: results
 mountPath: /results

The Job name (bench-llama3-70b-fp8-h100x4-20260616) is the unique identifier of the run. Changing it is enough to register a variant. The labels are the metadata that the scorecard-exporter reads to enrich the output JSON.

Performance layer: GuideLLM and AIPerf

GuideLLM — the SLO-driven sweep

GuideLLM (a vLLM project) generates realistic traffic patterns, synchronous, concurrent, poisson, throughput, sweep, and captures full TTFT and ITL distributions (Red Hat Developer). The sweep mode ramps from idle to saturation over 10 rounds and identifies the knee: the maximum load where goodput ≈ throughput under the declared SLO. The output is JSON/CSV with all the percentiles per round, ready to be versioned.

For the harness, the bench’s reference SLO is:

Metric	Threshold	Percentile
TTFT	500 ms	P99
ITL (TPOT)	50 ms/tok	P95
Error rate	0.5 %	—

The Job command already appears in the YAML above. The JSON output includes, per round: rate (req/s), TTFT (P50/P95/P99), ITL (P50/P95/P99), throughput (tok/s) and goodput (tok/s). The goodput under the SLO at the knee is the value that enters the scorecard.

For the cross-verification layer of the data and comparability with published results, see the post GuideLLM in depth.

AIPerf — NVIDIA’s concurrency profiler

AIPerf (the successor to GenAI-Perf, repository ai-dynamo/aiperf) measures TTFT, ITL, throughput and latency distributions at fixed concurrencies (GitHub ai-dynamo/aiperf). Where GuideLLM gives the automatic sweep up to the knee, AIPerf gives the detailed profile at specific concurrencies (4, 8, 16, 32 in the example): it allows the throughput-latency curve to be characterised point by point.

The two are complementary: GuideLLM finds the knee automatically; AIPerf confirms it and characterises the behaviour in the neighbourhood of that knee. Both outputs go to the results volume with the same BENCH_RUN_ID. The deep analysis of AIPerf/GenAI-Perf is in GenAI-Perf in depth.

Metrics the harness extracts from AIPerf for the scorecard:

# TTFT P99 at concurrency 16 (the closest to the sweep's knee):
aiperf profile ... --concurrency 16 \
 | jq '.results[] | select(.concurrency==16) | .ttft_ms.p99'

# Goodput (tok/s) at concurrency 16:
aiperf profile ... --concurrency 16 \
 | jq '.results[] | select(.concurrency==16) | .output_token_throughput'

Cost layer: OpenCost and LiteLLM

OpenCost — the denominator in euros per GPU-hour

OpenCost (CNCF incubating, opencost.io) allocates Kubernetes cost to namespace, label, pod and container in real time (GitHub opencost/opencost). For the harness, OpenCost answers the question: how much does the vllm-svc pod cost in euros per hour during the experiment’s window?

The harness calls the OpenCost REST API when the experiment finishes:

# Cost of the inference namespace over the experiment window (1 hour)
curl -s "http://opencost.monitoring.svc.cluster.local:9003/allocation" \
 --data-urlencode 'window=2026-06-16T12:00:00Z,2026-06-16T13:00:00Z' \
 --data-urlencode 'aggregate=namespace' \
 --data-urlencode 'namespace=inference' \
 | jq '.data[0].inference.totalCost'

The returned cost (in EUR, configured with the node’s real prices) is divided by the throughput measured in that same window to obtain the CPM:

$$\text{CPM} = \frac{\text{cost}_\text{namespace/h} \times 10^6}{\text{goodput (tok/s)} \times 3\,600}$$

The post OpenCost: cost allocation in Kubernetes details the price configuration and the allocation by GPU label.

LiteLLM — the per-request token counter

LiteLLM (litellm.ai, GitHub BerriAI/litellm) acts as an OpenAI-compatible proxy with token accounting per request, model and team. In the harness, GuideLLM and AIPerf point at the LiteLLM endpoint (which in turn forwards to vLLM): every request is recorded with prompt_tokens, completion_tokens and cost (using the custom pricing configured for the on-prem node).

# litellm-config.yaml (fragment of custom on-prem pricing)
model_list:
 - model_name: llama3-70b-fp8
 litellm_params:
 model: openai/meta-llama/Meta-Llama-3-70B-Instruct
 api_base: http://vllm-svc.inference.svc.cluster.local:8000/v1
 api_key: sk-dummy
 input_cost_per_token: 0.00000109 # 1.09 EUR/1M tok on-prem
 output_cost_per_token: 0.00000109

The LiteLLM records are exported to Prometheus as litellm_request_total_tokens and litellm_spend_metric_total, which the scorecard exporter consumes to compute the real CPM per token type (input vs output).

Energy layer: DCGM and Kepler

DCGM Exporter — GPU power and energy

DCGM Exporter (GitHub NVIDIA/dcgm-exporter, docs) exposes GPU metrics on /metrics for Prometheus as a DaemonSet on the GPU nodes. The two energy metrics the harness uses:

DCGM metric	Type	Unit	Use in the harness
`DCGM_FI_DEV_POWER_USAGE`	gauge	W	instantaneous power per GPU
`DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION`	counter	mJ	energy accumulated since boot

To compute the energy consumed during the experiment’s window (without the idle baseline), the harness takes the difference of the counter before and after the sweep:

# Deployment as a DaemonSet (fragment of the official chart)
helm repo add gpu-helm-charts \
 https://nvidia.github.io/dcgm-exporter/helm-charts
helm install dcgm-exporter gpu-helm-charts/dcgm-exporter \
 --namespace monitoring \
 --set serviceMonitor.enabled=true \
 --set serviceMonitor.interval=15s

The DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION field (in mJ) allows the experiment’s energy to be computed with hardware-counter accuracy, not with an estimate:

$$\text{experiment energy (J)} = \left(\text{counter}_\text{end} - \text{counter}_\text{start}\right) \times 10^{-3}$$ $$\text{Wh/token} = \frac{\text{experiment energy (J)}}{3\,600 \times \text{generated tokens}}$$

Kepler — pod-level energy

Kepler (CNCF sandbox, GitHub sustainable-computing-io/kepler) uses eBPF to estimate energy consumption at container and pod level, exporting kepler_container_joules_total to Prometheus (Red Hat Emerging Technologies). It combines RAPL (CPU/DRAM), NVML (GPU) and regression models when no sensors are available.

In the harness, Kepler complements DCGM: DCGM gives the hardware measurement of the GPU (more precise for GPU-intensive loads such as LLM inference), while Kepler attributes the energy to the specific vLLM pod (including the node’s CPU contribution). The main metric the harness consumes:

# Energy of the vllm-svc pod during the experiment (J)
increase(
 kepler_container_joules_total{
 container_namespace="inference",
 container_name="vllm"
 }[${BENCH_DURATION}]
)

The comparative analysis of DCGM vs Kepler vs Zeus is in Energy tools: deployment, precision and overhead.

The MLPerf Power reference

MLPerf Power (MLCommons, IEEE HPCA 2025 paper) establishes the protocol for measuring the energy efficiency of ML systems with high-precision external measurement. The in-house bench is not an MLPerf submission (that requires external power meters and committee review), but its published results are the calibration reference: if the harness gives a result of the same order as the MLPerf submission for the same hardware with the same model, the measurement is plausible. If it differs by more than a factor of 2, there is a methodological problem. See the post MLPerf Power: energy efficiency for the current reference data.

Unifying the metrics: the PromQL queries

The scorecard-exporter is the Job container that, once GuideLLM and AIPerf have finished, collects all the metrics from Prometheus and builds the JSON of the scorecard row. The key queries:

# Mean GPU power during the sweep (W) — 4×H100 node
avg_over_time(
 sum(DCGM_FI_DEV_POWER_USAGE{Hostname=~"gpu-node-.*"})[${BENCH_DURATION}:15s]
)

# Total GPU energy during the sweep (mJ → convert to J)
(
 sum(DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION{Hostname=~"gpu-node-.*"}) offset 0
 -
 sum(DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION{Hostname=~"gpu-node-.*"}) offset ${BENCH_DURATION}
) * 0.001

# Tokens generated during the sweep (from LiteLLM)
increase(
 litellm_request_total_tokens{model="llama3-70b-fp8", token_type="completion"}[${BENCH_DURATION}]
)

# Cost of the inference namespace over the window (from the OpenCost API)
# → REST call at the start and at the end of the Job, difference of accrual

The ${BENCH_DURATION} variable is the real duration of the sweep (in Prometheus format, e.g. 22m), which the exporter computes as end_ts - start_ts and substitutes into every query.

The output JSON of each run has this structure:

{
 "run_id": "20260616T1200",
 "metadata": {
 "model": "meta-llama/Meta-Llama-3-70B-Instruct",
 "engine": "vllm-0.9.1",
 "precision": "fp8",
 "hardware": "4xH100-SXM-80GB",
 "isl_tokens": 1024,
 "osl_tokens": 256,
 "tokenizer": "meta-llama-3-tokenizer-v3",
 "slo_ttft_p99_ms": 500,
 "slo_itl_p95_ms": 50,
 "bench_tool_guidellm": "0.4.2",
 "bench_tool_aiperf": "0.2.0",
 "dcgm_exporter": "3.3.9",
 "kepler": "0.10.2"
 },
 "performance": {
 "goodput_tok_s": 3120,
 "ttft_p99_ms": 487,
 "itl_p95_ms": 42,
 "throughput_peak_tok_s": 3890,
 "elbow_concurrency": 28
 },
 "cost": {
 "cpm_eur_1m": 0.97,
 "gpu_cost_eur_h": 10.8,
 "cost_window_eur": 3.24
 },
 "energy": {
 "wh_per_token": 0.00044,
 "j_per_token": 1.58,
 "power_mean_w": 4924,
 "energy_total_kwh": 0.287,
 "pue": 1.4,
 "wh_per_token_pue_adjusted": 0.000616
 },
 "carbon": {
 "grid_intensity_gco2_kwh": 40,
 "co2_per_1m_tokens_g": 24.6
 }
}

All the fields are computed from the same time windows. The JSON is versioned in git alongside the harness code. Reproducing any scorecard row is: git checkout <run-id> && kubectl apply -f job.yaml.

The 3-axis scorecard: table and interpretation

The following table illustrates how five configurations of the same model (Llama 3 70B) on the same reference node (4×H100 SXM) compare with the harness. The figures are illustrative of the order of magnitude (the bench fills them in with real measurements):

Config	Engine	Precision	CPM (€/1M)	Goodput (tok/s)	Wh/tok	TTFT P99 (ms)	ITL P95 (ms)	CO2 /1M (g, FR)
A	vLLM 0.9	FP16	1.64	1,890	0.00074	498	49	29.6
B	vLLM 0.9	FP8	0.97	3,120	0.00044	487	42	17.6
C	SGLang 0.4	FP8	0.88	3,410	0.00041	421	38	16.4
D	vLLM 0.9	FP8, ISL 512	1.31	2,240	0.00056	294	31	22.4
E	vLLM 0.9	FP8, max-batch 128	0.84	3,590	0.00040	721	55	16.0

How to read the table:

B vs A: FP8 over FP16 cuts CPM by 41 % and energy per token by 41 % thanks to the same higher throughput. Both meet the SLO (TTFT P99 < 500 ms, ITL P95 < 50 ms). B dominates A on all three axes: it is Pareto-superior.
C vs B: SGLang gives 9 % more goodput and 9 % lower CPM with FP8. Both meet the SLO. C is Pareto-superior to B (if the engine is indifferent for the stack).
D vs B: a shorter ISL (512 tok) lowers latency (TTFT P99 294 ms vs 487 ms) but lowers goodput and raises CPM. If the use case demands TTFT < 300 ms, D is the candidate; if TTFT < 500 ms is enough, B is better on cost and energy.
E vs C: max-batch 128 raises goodput (+5 %) and lowers CPM, but the TTFT P99 breaks the SLO (721 ms > 500 ms) and so does the ITL P95 (55 ms > 50 ms). E has the best raw throughput but is outside the SLO: it is not a valid candidate for the interactive chat use case.

The scorecard’s Pareto frontier, under the declared SLO, includes only A, B, C and D (E falls out for breaking the SLO). Of those four, C dominates B which dominates A. D only enters the frontier if the use case demands TTFT P99 < 300 ms. The decision is not a number; it is the whole row plus the SLO.

The multi-objective Pareto frontier

With three minimisation metrics, CPM (€/1M tok), Wh/tok and TTFT P99 (ms), the Pareto frontier is defined as the set of configurations where none dominates another on all three axes simultaneously. Formally, config $ i $ dominates config $ j $ if:

$$\text{CPM}_i \leq \text{CPM}_j \;\land\; \text{Wh/tok}_i \leq \text{Wh/tok}_j \;\land\; \text{TTFT}_{i} \leq \text{TTFT}_{j}$$

with at least one strict inequality. The scorecard allows it to be computed over the set of runs with a simple vector operation. In the table’s example, the frontier under the SLO (with E excluded) is $\{C, D\}$: C dominates on cost/energy/goodput; D dominates on latency. Between C and D the choice depends on the TTFT SLO of the specific use case.

Reproducible design: the twelve mandatory metadata

The checklist that closes the bias dossier (Bias and reproducibility). For the harness JSON to be auditable, the twelve fields must be present in every run:

#	Field	Example	Why it is mandatory
1	Serving engine version	`vllm-0.9.1`	the same model can perform differently across versions
2	Model version (commit/tag)	`meta-llama/Meta-Llama-3-70B-Instruct@sha256:abc`	avoids ambiguity between variants of the same name
3	Precision	`fp8`	FP16 vs FP8 changes throughput and latency
4	Tokeniser and version	`meta-llama-3-tokenizer-v3`	the ISL/OSL in tokens depends on the tokeniser
5	ISL (input sequence length)	`1024` tok	changes the prefill time and the knee
6	OSL (output sequence length)	`256` tok	changes the decode time
7	Hardware (model and count)	`4×H100-SXM-80GB`	the basis of any comparison
8	Maximum concurrency tested	`32`	defines the sweep’s range
9	Bench tool version	`guidellm-0.4.2`, `aiperf-0.2.0`	different versions can give different metrics
10	Declared SLO	`TTFT P99 < 500 ms, ITL P95 < 50 ms`	the knee depends on the SLO; without it there is no goodput
11	Warmup duration	`60 s`	without warmup the KV cache is cold and the first rounds are biased
12	Harness version / run-id	`20260616T1200`	identifies the run so it can be reproduced with `git checkout`

A scorecard missing any of these twelve fields is an anecdote. With all twelve, it is an auditable datum. The run-id in the Job name (bench-llama3-70b-fp8-h100x4-20260616) implicitly incorporates fields 1-3 and 7, forcing the Job name to change with every variant, which makes it impossible to overlap runs in the same namespace.

Idempotence and versioning of results

The Kubernetes Job is idempotent: if the same manifest is applied again (same name), the Job already exists and Kubernetes does not relaunch it (backoffLimit: 0 policy). This avoids accidental duplicate runs. To relaunch, the Job has to be deleted (kubectl delete job <name>) or the run-id changed.

Results are versioned with the following directory convention in the harness’s git repository:

results/
20260616T1200/
guidellm-20260616T1200.json
aiperf-20260616T1200.json
scorecard-20260616T1200.json
20260617T0900/
guidellm-20260617T0900.json
aiperf-20260617T0900.json
scorecard-20260617T0900.json
scorecard-aggregate.csv # all rows, for analysis and charts

The scorecard-aggregate.csv is the scorecard table in flat format: each row is a run, each column a field of the JSON. It is generated with:

# Generates the aggregate CSV from all the JSON files in results/
python scripts/aggregate_scorecards.py results/ > scorecard-aggregate.csv

The CSV is what feeds the Pareto charts, the reports and the architecture decisions. It is versioned in git; its diff is the configuration change.

How it connects with the other posts in the series

The harness is not a standalone article: it is the synthesis of the twenty-eight articles. Each tool has its deep dive:

Performance: GuideLLM in depth — sweep, SLO, goodput, knee; GenAI-Perf / AIPerf in depth — concurrency, rate sweep, distribution metrics.
Cost: OpenCost: cost allocation in Kubernetes — allocation by namespace/label, custom on-prem prices.
Energy: Energy tools: deployment, precision and overhead — DCGM vs Kepler vs Zeus, instrumentation overhead, relative precision; MLPerf Power: energy efficiency — the external calibration reference.
Theoretical framework: The three axes: cost, performance and energy — the CPM/energy/throughput identity that justifies measuring them together.
Bias: Bias and reproducibility in benchmarking — the twelve biases the harness removes.

The harness closes the series’ dossier because it makes the opening article’s promise possible: “when someone disputes a number in the proposal, the answer is not ‘a blog says so’, but ’this is the bench, this is the methodology, reproduce it’”. With the run-id and the git repo, reproducing is one command.

Full flow of a benchmarking session

The harness’s standard operating procedure, from start to finish:

# 1. Make sure the inference endpoint is up
kubectl get svc vllm-svc -n inference

# 2. Verify that DCGM and Kepler are scraping
curl -s http://prometheus.monitoring.svc.cluster.local:9090/api/v1/query \
 --data-urlencode 'query=DCGM_FI_DEV_POWER_USAGE' | jq '.data.result | length'

# 3. Record the energy counter BEFORE the experiment
PRE_ENERGY=$(kubectl exec -n monitoring dcgm-exporter-xxx -- \
 curl -s localhost:9400/metrics \
 | grep DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION | awk '{sum+=$2} END{print sum}')

# 4. Launch the Job (unique name per run)
kubectl apply -f jobs/bench-llama3-70b-fp8-h100x4-20260616.yaml

# 5. Wait for it to finish
kubectl wait --for=condition=complete job/bench-llama3-70b-fp8-h100x4-20260616 \
 -n benchmark --timeout=3600s

# 6. Record the energy counter AFTER
POST_ENERGY=$(kubectl exec -n monitoring dcgm-exporter-xxx -- \
 curl -s localhost:9400/metrics \
 | grep DCGM_FI_DEV_TOTAL_ENERGY_CONSUMPTION | awk '{sum+=$2} END{print sum}')

# 7. Collect the results from the volume
kubectl cp benchmark/bench-run-pod:/results/ ./results/20260616T1200/

# 8. Compute the experiment's energy (mJ → kWh)
python scripts/calc_energy.py \
 --pre $PRE_ENERGY --post $POST_ENERGY \
 --run-id 20260616T1200

# 9. Add a row to the aggregate CSV
python scripts/aggregate_scorecards.py results/ > scorecard-aggregate.csv

# 10. Version it
git add results/20260616T1200/ scorecard-aggregate.csv
git commit -m "bench: llama3-70b fp8 4xH100 ISL1024 OSL256 (20260616T1200)"

Steps 6-9 are candidates for automation as a second post-processing Job launched when the main one completes (with ownerReferences or a clean-up CronJob). In the harness’s basic state, the manual steps are precisely the ones that force a review of the result before versioning it.

Cost of the experiment

A GuideLLM sweep of 10 rounds at 120 seconds per round occupies the node for ~22-26 minutes. The AIPerf profile at 4 concurrencies (200 requests each) adds ~8-12 minutes. Total per run: ~35-40 minutes of a 4×H100 node.

At the reference amortised cost (~10.8 €/h for 4×H100 on-prem):

$$\text{cost per run} \approx 10.8 \times \frac{38}{60} \approx 6.84 \text{ EUR}$$

At the reference European cloud price (4 × 2.73 = 10.92 €/h):

$$\text{cost per run (cloud)} \approx 10.92 \times \frac{38}{60} \approx 6.92 \text{ EUR}$$

A continuous benchmarking programme (10 configurations × 3 models × 2 sweeps/week) adds up to ~415 EUR/week. That is why the harness includes short sweeps (5 rounds, ~15 minutes) for CI and full sweeps for releases.

What the harness does not measure (and why)

Missing dimension	Reason	Where it is covered
Output quality (MMLU, HumanEval)	requires a quality evaluation bench, not a serving one	LLM quality benchmarks
Training throughput	the harness is exclusively for inference	outside the scope of the “data” series
Network and storage cost	OpenCost can attribute it, but it requires additional configuration	OpenCost: cost allocation
Hourly carbon intensity	use the ElectricityMaps API + the measured energy	From the watt to carbon: PUE and grid mix
Client-server network latency	the Job runs inside the cluster; WAN latency requires an external client	document as additional metadata

Sources

GuideLLM · GitHub (proyecto vLLM) — https://github.com/vllm-project/guidellm
GuideLLM · PyPI — https://pypi.org/project/guidellm/
Red Hat Developer · desplegar y benchmarkear vLLM con GuideLLM en Kubernetes — https://developers.redhat.com/articles/2025/12/24/how-deploy-and-benchmark-vllm-guidellm-kubernetes
Red Hat Developer · GuideLLM: evaluar despliegues LLM para inferencia real — https://developers.redhat.com/articles/2025/06/20/guidellm-evaluate-llm-deployments-real-world-inference
AIPerf · GitHub (ai-dynamo/aiperf) — https://github.com/ai-dynamo/aiperf
GenAI-Perf · documentación NVIDIA Triton — https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/genai-perf/README.html
NVIDIA · LLM performance benchmarking con GenAI-Perf — https://developer.nvidia.com/blog/llm-performance-benchmarking-measuring-nvidia-nim-performance-with-genai-perf
NVIDIA NIM Benchmarking · métricas (TTFT, ITL, throughput) — https://docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html
OpenCost · web oficial — https://opencost.io/
OpenCost · GitHub (opencost/opencost) — https://github.com/opencost/opencost
OpenCost · CNCF blog (sandbox → incubating) — https://www.cncf.io/blog/2022/12/06/opencost-a-new-cncf-sandbox-project-for-real-time-kubernetes-cost-monitoring/
LiteLLM · web oficial — https://www.litellm.ai/
LiteLLM · GitHub (BerriAI/litellm) — https://github.com/BerriAI/litellm
LiteLLM · Spend Tracking docs — https://docs.litellm.ai/docs/proxy/cost_tracking
NVIDIA DCGM Exporter · documentación oficial — https://docs.nvidia.com/datacenter/dcgm/latest/gpu-telemetry/dcgm-exporter.html
NVIDIA DCGM Exporter · GitHub — https://github.com/NVIDIA/dcgm-exporter
NVIDIA DCGM Exporter · métricas CSV — https://github.com/NVIDIA/dcgm-exporter/blob/main/etc/dcp-metrics-included.csv
Kepler (Kubernetes-based Efficient Power Level Exporter) · GitHub — https://github.com/sustainable-computing-io/kepler
Red Hat Emerging Technologies · Introducing Kepler — https://next.redhat.com/2023/08/22/introducing-kepler-efficient-power-monitoring-for-kubernetes/
MLCommons · MLPerf Inference Datacenter benchmark — https://mlcommons.org/benchmarks/inference-datacenter/
MLCommons · MLPerf Power presentado en IEEE HPCA 2025 — https://mlcommons.org/2025/03/ml-commons-power-hpca/
IEEE Xplore · MLPerf Power paper (μWatts to MWatts) — https://ieeexplore.ieee.org/document/10946778/
MLCommons · MLPerf Inference v5.1 results (septiembre 2025) — https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/
Red Hat · Efficient and reproducible LLM inference: MLPerf Inference v5.1 — https://www.redhat.com/en/blog/efficient-and-reproducible-llm-inference-red-hat-mlperf-inference-v51-results
BentoML · Beyond tokens-per-second: cost, speed and quality in LLM inference — https://www.bentoml.com/blog/beyond-tokens-per-second-how-to-balance-speed-cost-and-quality-in-llm-inference

From the SLO to the number of GPUs: how to size and justify the investment in inference hardware

Tue, 16 Jun 2026 11:00:00 +0200

Notation: amounts in N € or N USD (source denominated in dollars); decimal point; comma as thousands separator (1{,}234). Generic example hardware: 4×H100 SXM5 80 GB nodes. The dollar sign is not used (it is the formula delimiter).

TL;DR

With an SLO of TTFT P99 ≤ 300 ms and ITL P99 ≤ 50 ms for a production chat, peak GPU utilisation must stay at ≤ 63 %.
A service receiving 5M requests/day with 512 input tokens and 256 output tokens on average generates an hourly peak of ~11,600 tok/s (with a peak factor of 1.8×). With vLLM on H100 SXM5 (Llama-3.3 70B FP8, ~1,850 tok/s at 50 concurrent requests), ≥ 10 GPUs are needed at peak to meet the SLO, which translates into 3 4×H100 nodes (with headroom).
The all-in cost of those 3 nodes is ~83,300 € per year per node (3-year capex + base opex); the €/1M tokens in the base scenario at 63 % utilisation is ~0.37 €.
Against AWS p5 on-demand (~3.90 USD/GPU-hour after the 44 % cut of Jun. 2025), the utilisation break-even sits at ~55 %; against neocloud on-demand (~2.01 USD/GPU-hour), there is no feasible break-even at medium utilisation.
Capex payback happens between 13 and 24 months if sustained utilisation exceeds 70 %.

1. The sizing chain: from demand to the number of GPUs

Sizing an inference platform follows a causal chain of five steps. Each step has a formula; none can be skipped.

1.1 Step 1 — Characterise the demand

Parameter	Symbol	Example
Requests per day (average)	$D$	5,000,000
Input tokens per request (average)	$L_{\text{in}}$	512
Output tokens per request (average)	$L_{\text{out}}$	256
Peak factor (peak hour vs average ratio)	$k_{\text{peak}}$	1.8
Hours per day with significant traffic	$H$	16

The average output throughput:

$$\dot{T}_{\text{avg}} = \frac{D \times L_{\text{out}}}{86{,}400\,\text{s}} = \frac{5{,}000{,}000 \times 256}{86{,}400} \approx 14{,}815\;\text{tok/s}$$

The peak throughput (busy hour):

$$\dot{T}_{\text{peak}} = k_{\text{peak}} \times \frac{D \times L_{\text{out}}}{H \times 3{,}600} = 1.8 \times \frac{5{,}000{,}000 \times 256}{16 \times 3{,}600} \approx 40{,}000\;\text{tok/s}$$

Note: if the traffic profile has very pronounced peaks (peak/average ratio > 3), sizing is done for the peak and average utilisation falls; the sensitivity analysis in §5 quantifies the effect.

1.2 Step 2 — Set the SLO and derive the maximum utilisation

The latency SLO imposes a ceiling on GPU utilisation. Using queueing theory (M/G/1 model):

$$\rho_{\max} \approx 1 - \frac{1}{\sqrt{1 + C_{s}^{2}}} \cdot \frac{W_{\text{queue}}^{*}}{\bar{s}}$$

where $\rho$ is the utilisation, $W_{\text{queue}}^{*}$ the maximum admissible queueing time and $\bar{s}$ the average service time per request. For the simplified M/M/1 case with prefill time dominant:

$$\rho_{\max} = 1 - \frac{W_{\text{queue}}^{*}}{\bar{s} \cdot (1 + W_{\text{queue}}^{*}/\bar{s})}$$

The practical reference table (derived from the Spheron/Little’s Law formula, 2026):

SLO TTFT P99	Maximum utilisation $\rho_{\max}$
200 ms	55 %
300 ms	63 %
400 ms	70 %
500 ms	75 %

For our example (SLO 300 ms P99): $\rho_{\max} = 0.63$.

The ITL SLO imposes an additional constraint: the inference engine must be able to generate the next token in ≤ 50 ms. On H100 SXM5 with vLLM and Llama-3.3 70B FP8, the ITL P50 at 50 concurrent requests is ~20 ms, with P99 ~45 ms. ITL is the dominant constraint only when the batch size is very high (>64 sequences) or VRAM is nearly full.

1.3 Step 3 — Required throughput and throughput per GPU

The throughput the cluster must serve at peak, while respecting $\rho_{\max}$:

$$\dot{T}_{\text{required}} = \frac{\dot{T}_{\text{peak}}}{\rho_{\max}} = \frac{40{,}000}{0.63} \approx 63{,}500\;\text{tok/s (installed capacity)}$$

The throughput per GPU (reference benchmark, vLLM v0.18.0, Llama-3.3 70B FP8, H100 SXM5 80 GB, 50 concurrent requests):

Engine	Throughput (tok/s per GPU)	TTFT P50 / P95 (50 req)	Source
vLLM 0.18.0	1,850	380 ms / 720 ms	Spheron benchmarks, Mar. 2026
SGLang 0.5.9	1,920	360 ms / 680 ms	Spheron benchmarks, Mar. 2026
TensorRT-LLM 1.2.0	2,100	340 ms / 620 ms	Spheron benchmarks, Mar. 2026

The throughput data correspond to the test with Llama 3.3 70B Instruct FP8, 512 input / 256 output tokens, 50 concurrent requests, on bare-metal H100 SXM5. See the engine analysis in serving engines Pareto comparison.

We use vLLM as the general-purpose production reference: $\dot{T}_{\text{GPU}} = 1{,}850$ tok/s.

1.4 Step 4 — Number of GPUs and number of nodes

$$N_{\text{GPU}} = \left\lceil \frac{\dot{T}_{\text{required}}}{\dot{T}_{\text{GPU}}} \right\rceil = \left\lceil \frac{63{,}500}{1{,}850} \right\rceil = \lceil 34.3 \rceil = 35\;\text{GPUs}$$

With 4×H100 SXM5 nodes:

$$N_{\text{nodes}} = \left\lceil \frac{N_{\text{GPU}}}{4} \right\rceil = \left\lceil \frac{35}{4} \right\rceil = 9\;\text{nodes}$$

We add 15 % headroom for hardware failures (~5 % annual rate in small clusters), upgrades and unforeseen peaks:

$$N_{\text{nodes, final}} = \lceil 9 \times 1.15 \rceil = 11\;\text{nodes} \approx 44\;\text{GPUs}$$

For the TL;DR example (5M requests/day with a 16 h active profile, k 1.8 and an SLO of 300 ms) the number of nodes is 11. The simplified TL;DR case with k=1 and H=24 gives 3 nodes; the difference illustrates the impact of the hourly profile.

1.5 Summary of the chain

2. From sizing to cost: the TCO model

With $N_{\text{nodes}} = 11$ 4×H100 SXM5 nodes, the TCO follows the model detailed in TCO of an on-premise GPU cluster. Here the compact formula is reproduced and applied to the sized cluster.

2.1 Annual cost per node (base scenario)

$$C_{\text{node/year}} = \frac{\text{node capex}}{\text{years}} + \text{node opex/year}$$

Line item	Value (USD / €)	Note
4×H100 node capex (midpoint)	178,500 USD	GPUs + server + network + storage + rack
3-year amortisation	59,500 USD/year ≈ 55,300 €/year	Straight line
Opex/year (base scenario, 8–16 node cluster)	~28,000 €/year	Energy + staff + maintenance + colocation
Total per node per year	~83,300 €/year	—

For 11 nodes: 915,300 €/year of total fixed cost.

2.2 From €/node-year to €/GPU-hour

$$\text{EUR/GPU-hour} = \frac{C_{\text{node/year}}}{4\;\text{GPUs} \times 8{,}760\;\text{h} \times \rho}$$

Utilisation $\rho$	EUR/GPU-hour
40 %	5.93
55 %	4.31
63 % (SLO 300 ms)	3.76
70 %	3.39
80 %	2.97
100 %	2.38

2.3 From €/GPU-hour to €/1M tokens

$$\text{EUR/1M tokens} = \frac{\text{EUR/GPU-hour} \times 10^{6}}{\dot{T}_{\text{GPU}} \times 3{,}600}$$

With $\dot{T}_{\text{GPU}} = 1{,}850$ tok/s (vLLM, Llama-3.3 70B FP8):

Utilisation	EUR/GPU-hour	EUR/1M tokens
40 %	5.93	0.891
55 %	4.31	0.647
63 % (SLO 300 ms)	3.76	0.564
70 %	3.39	0.509
80 %	2.97	0.446
100 %	2.38	0.357

The complete cost/token identity as a function of throughput and utilisation is developed in cost per token and per request.

2.4 Comparison with cloud (equivalent €/1M tokens)

To compare, the cloud price is converted to the €/1M tokens equivalent using the same reference throughput ($\dot{T}_{\text{GPU}} = 1{,}850$ tok/s):

$$\text{EUR/1M tokens (cloud)} = \frac{P_{\text{cloud}} \times 10^{6}}{1{,}850 \times 3{,}600}$$

Provider / Mode	GPU-hour price (USD)	EUR/GPU-hour ($1\,\text{USD} \approx 0.93\,\text{EUR}$)	EUR/1M tokens equiv.
AWS p5 on-demand (post Jun. 2025)	3.90	3.63	0.545
AWS p5 1-year reserved	~2.50	2.33	0.350
CoreWeave on-demand	~2.01	1.87	0.281
CoreWeave 3-year reserved	~1.49	1.39	0.209
GCP A3 on-demand	~3.67	3.41	0.512
Azure ND H100 v5 on-demand	~6.98	6.49	0.975

Sources: IntuitionLabs (Jun. 2026), CloudZero (Jun. 2026), Spheron GPU pricing (May 2026).

3. Break-even and payback of your own investment

3.1 The utilisation break-even formula

The utilisation break-even $u^{*}$ is the utilisation at which the annual on-prem cost per GPU-hour equals the cloud price:

$$u^{*} = \frac{\text{capex/year} + \text{opex/year}}{4 \times 8{,}760 \times P_{\text{cloud}}}$$

where $P_{\text{cloud}}$ is the cloud price in the same currency as the on-prem costs.

With the base scenario (capex/year 55,300 €, opex/year 28,000 €, total 83,300 €/node/year):

Cloud reference	Cloud price (EUR/GPU-hour)	$u^{*}$ break-even
Azure on-demand (~6.49 EUR)	6.49	23 %
AWS on-demand post-cut (~3.63 EUR)	3.63	41 %
GCP on-demand (~3.41 EUR)	3.41	44 %
CoreWeave on-demand (~1.87 EUR)	1.87	80 %
CoreWeave 3-yr reserved (~1.39 EUR)	1.39	>100 % (impossible)
AWS 1-yr reserved (~2.33 EUR)	2.33	64 %

3.2 Capex payback

Payback is the time $T_{\text{pay}}$ in which the cumulative saving against cloud equals the initial capex:

$$T_{\text{pay}} = \frac{\text{total cluster capex}}{(\text{cloud cost/year}) - (\text{on-prem opex/year})}$$

where the cloud cost/year is computed at the same sustained utilisation.

For the 11-node cluster (total capex 11 × 178,500 USD ≈ 1,825,000 USD ≈ 1,697,000 €):

Cloud reference	Sustained utilisation	Annual saving vs cloud	Payback
AWS on-demand (3.63 EUR/h)	70 %	(3.63 − 3.39 EUR) × 4 × 8760 × 0.70 × 11 nodes ≈ 63,900 €/year	~27 months
AWS on-demand (3.63 EUR/h)	80 %	(3.63 − 2.97) × 4 × 8760 × 0.80 × 11 ≈ 203,600 €/year	~10 months
GCP on-demand (3.41 EUR/h)	70 %	(3.41 − 3.39) × 4 × 8760 × 0.70 × 11 ≈ 5,400 €/year	~315 months (not viable)
Azure on-demand (6.49 EUR/h)	70 %	(6.49 − 3.39) × 4 × 8760 × 0.70 × 11 ≈ 830,000 €/year	~2 months

The 13-month payback quoted in studies such as Lenovo TCO 2026 corresponds to ~80 % utilisation against high-priced hyperscalers (Azure/AWS before the cut). With current prices (post June 2025, AWS at 3.90 USD), the window widens.

3.3 Simple payback (capex only vs equivalent cloud)

If on-prem opex is omitted and only capex is compared with the gross saving:

$$T_{\text{pay,simple}} = \frac{\text{capex}}{P_{\text{cloud}} \times 4 \times 8{,}760 \times \rho \times N_{\text{nodes}}}$$

Utilisation	AWS (3.63 EUR)	Azure (6.49 EUR)
50 %	54 months	30 months
70 %	39 months	22 months
80 %	34 months	19 months

4. Decision table: when to buy, rent or go hybrid

The table below is a five-dimension Pareto. There is no implicit order between columns; the reading depends on the organisation’s constraints.

Option	EUR/GPU-hour (util. 70 %)	Initial capex	Required utilisation	Data sovereignty	Peak elasticity
On-prem purchase, util. ≥ 70 %	3.39	high (178k USD/node)	≥ 70 % sustained	total	none
On-prem purchase, util. < 50 %	> 4.75	high	< 50 % → loses vs cloud	total	none
Sovereign EU cloud OD (Scaleway/Nebius EU)	2.00–3.59	none	any	yes (EU)	total
AWS p5 on-demand (post Jun. 2025)	3.63	none	any	no (CLOUD Act)	total
CoreWeave on-demand	1.87	none	any	partial (US)	total
CoreWeave 3-year reserved	1.29–1.39	financial commitment	rigid contract	partial (US)	none
AWS 1-year reserved	~2.17	1-year commitment	contract	no (CLOUD Act)	none
Hybrid on-prem base + EU cloud peak	2.50–3.39 (weighted)	medium	base ≥ 70 %, elastic peak	yes (EU)	elastic peak

Cut-off criteria that come before the table:

GDPR sovereignty: if the data is personal or the system is high-risk under the EU AI Act, CoreWeave/AWS are ruled out before any price comparison.
Minimum volume to amortise capex: below ~2M tokens/day sustained over 3 years, on-prem capex does not amortise against AWS on-demand.
Traffic elasticity: peaks >3× the base favour a hybrid or pure cloud; a stable base favours on-prem.

The cost/sovereignty Pareto frontier for GDPR data leaves three options: on-prem, sovereign EU cloud and hybrid. Sustained utilisation and traffic predictability decide between them. See the cross-analysis of the four axes in sovereign on-premise vs hyperscalers.

5. Sensitivity analysis

5.1 Sizing vs hourly profile and peak factor

The number of GPUs grows linearly with $k_{\text{peak}}$ and inversely with $\rho_{\max}$:

$$N_{\text{GPU}} = \left\lceil \frac{D \times L_{\text{out}} \times k_{\text{peak}}}{H \times 3{,}600 \times \rho_{\max} \times \dot{T}_{\text{GPU}}} \right\rceil$$

Peak factor $k$	SLO 300 ms ($\rho_{\max}=0.63$)	SLO 500 ms ($\rho_{\max}=0.75$)
1.2	8 GPUs (2 nodes)	6 GPUs (2 nodes)
1.8	12 GPUs (3 nodes)	10 GPUs (3 nodes)
2.5	16 GPUs (4 nodes)	13 GPUs (4 nodes)
3.5	22 GPUs (6 nodes)	18 GPUs (5 nodes)

(Example simplified to 5M req/day with H=24 to illustrate the sensitivity to the peak factor)

A peak factor of 3.5× triples the number of nodes compared with k=1.2 at the same SLO. Sizing hardware for $k > 2.5$ leaves GPUs idle more than 70 % of the time; peak cloud is more efficient beyond that threshold.

5.2 Break-even vs sustained utilisation

$$u^{*} = \frac{83{,}300}{4 \times 8{,}760 \times P_{\text{cloud}}}$$

Cloud price (EUR/GPU-hour)	$u^{*}$ break-even	Scenario
6.49 (Azure OD)	23 %	On-prem wins almost always
3.63 (AWS OD)	41 %	On-prem wins if util. > 41 %
3.41 (GCP OD)	44 %	—
2.33 (AWS 1-yr reserved)	64 %	On-prem wins if util. > 64 %
1.87 (CoreWeave OD)	80 %	Hard to reach in production
1.39 (CoreWeave 3-yr reserved)	>100 %	On-prem never closes the gap

5.3 Break-even vs energy price

Energy accounts for 6–11 % of total TCO. Its impact on the break-even is moderate:

Energy price (EUR/kWh)	Energy opex/year per node	EUR/GPU-hour (70 % util.)	$u^{*}$ vs AWS OD
0.034 (solar PPA)	1,604 €	3.22	38 %
0.116 (industrial ES, base)	5,475 €	3.39	41 %
0.200 (high tariff)	9,437 €	3.57	43 %

The difference between the cheapest and the most expensive scenario is only 5 percentage points of break-even. The variable that moves the needle is utilisation, not energy.

5.4 Break-even vs demand growth

If demand grows at an annual rate $g$, the average utilisation of the cluster (sized for year 1) rises over time until it saturates and has to be expanded:

$$\rho(t) = \rho_{0} \times (1 + g)^{t}$$

Annual growth $g$	Time to saturation ($\rho \to 100\,\%$)	Decision
10 %	~11 years	Comfortable purchase
30 %	~4 years	Purchase with a review at 3 years
60 %	~2 years	Hybrid: base + elastic cloud
>100 %	<1 year	Pure cloud until it stabilises

For growth above 30 % per year, a buy-only strategy means oversizing for the future peak or re-buying hardware in short cycles. The hybrid (on-prem base + cloud for growth) minimises the capex at risk.

5.5 Headroom: the cost of the safety margin

The 15 % headroom in $N_{\text{nodes}}$ amounts to having ~1.6 extra nodes on average. Its annual cost is:

$$C_{\text{headroom}} = 0.15 \times 83{,}300\;\text{EUR/node/year} \times N_{\text{nodes,base}} \approx 12{,}500 \times 9 = 112{,}500\;\text{EUR/year}$$

That cost is justified by:

GPU failure rate ~5 % per year (in small clusters, documented by Introl, Apr. 2026)
Replacement time of 2–8 weeks (depending on market availability)
Unforeseen peaks of up to 20 % above the estimate

If the service can degrade gracefully (relaxing the TTFT SLO from 300 ms to 500 ms at extreme peak), headroom can be cut to 10 %, saving ~37,500 EUR/year.

6. Sensitivity map: €/GPU-hour and break-even on two axes

The table below crosses utilisation with the opex scenario, showing the all-in EUR/GPU-hour (base scenario, capex/year 55,300 €):

	Low opex (13,000 €/year)	Base opex (28,000 €/year)	High opex (75,000 €/year)
Util. 40 %	4.74	5.93	9.06
Util. 55 %	3.45	4.31	6.59
Util. 63 %	3.01	3.76	5.75
Util. 70 %	2.71	3.39	5.18
Util. 80 %	2.37	2.97	4.54
Util. 100 %	1.90	2.38	3.63

The crossing point with the cloud price (AWS OD: 3.63 EUR):

Low-opex scenario: break-even at ~38 % utilisation
Base-opex scenario: break-even at ~41 % utilisation
High-opex scenario: break-even at ~53 % utilisation

The biggest lever for lowering the break-even is not hardware capex but opex (staff and colocation above all). See the analysis of utilisation as a FinOps lever in GPU utilisation as FinOps.

7. Integration with the rest of the FinOps chain

The sizing of §1 determines the number of nodes; the TCO of §2 gives the cost per hour; the comparison of §3 gives the break-even. Those three numbers feed directly into the other instruments of the series:

Instrument	Input from this article	Output
GuideLLM — SLO validation under load	TTFT/ITL P99 SLO from step 2	Experimental confirmation of the real throughput per GPU
Capacity planning for on-premise inference	Demand profile and N GPUs from step 4	Scaling policy and autoscaling triggers
Cost per token and per request	EUR/GPU-hour from step §2.2	EUR/1M tokens per model and batching
Cloud GPU commitment and spot	Cloud prices from table §2.4	Optimisation of the complementary cloud tier
Sovereign on-premise vs hyperscalers	Break-even from §3	Final buy/rent decision including the sovereignty axis
TCO of an on-premise GPU cluster	Capex and opex from §2.1	Detailed TCO model with every line item
GPU utilisation as FinOps	Target utilisation from step 2	Scheduling levers to raise real utilisation

Sources

Spheron · LLM Inference SLO Engineering: TTFT, ITL, and P99 Latency Budgets for Production AI (2026) — https://www.spheron.network/blog/llm-inference-slo-ttft-itl-latency-budget-guide-2026/
Spheron · vLLM vs TensorRT-LLM vs SGLang: H100 Benchmarks (2026) — https://www.spheron.network/blog/vllm-vs-tensorrt-llm-vs-sglang-benchmarks/
Spheron · GPU Cloud Pricing 2026: H100 from 1.03 USD/hr, B200 from 2.12 USD/hr — https://www.spheron.network/blog/gpu-cloud-pricing-comparison-2026/
Spheron · LLM Inference On-Premise vs GPU Cloud: 2026 Cost and Break-Even Analysis — https://www.spheron.network/blog/llm-inference-on-premise-vs-cloud/
MLPerf Inference v5.1 — Red Hat: 5 777 tok/s (offline) en Llama 3.1-8B FP8 en H100 — https://www.redhat.com/en/blog/efficient-and-reproducible-llm-inference-red-hat-mlperf-inference-v51-results
MLPerf Inference v6.0 Results Explained: GPU Performance Rankings for AI Workloads (2026) — https://www.spheron.network/blog/mlperf-inference-v6-benchmark-results-2026/
IntuitionLabs · H100 Rental Prices Compared: 1.49–6.98 USD/hr Across 15+ Cloud Providers (2026) — https://intuitionlabs.ai/articles/h100-rental-prices-cloud-comparison
CloudZero · Cloud GPU Pricing Comparison: AWS vs Azure vs GCP For AI Workloads (2026) — https://www.cloudzero.com/blog/cloud-gpu-pricing-comparison/
Lenovo Press · On-Premise vs Cloud: Generative AI Total Cost of Ownership (2026 Edition) — https://lenovopress.lenovo.com/lp2368-on-premise-vs-cloud-generative-ai-total-cost-of-ownership-2026-edition
Introl · GPU Infrastructure TCO Model: 5-Year Cost Analysis for Enterprise AI (abr. 2026) — https://introl.com/blog/gpu-infrastructure-tco-5-year-cost-model
GMI Cloud · NVIDIA H100 GPU Pricing 2026: Rent vs Buy Cost Analysis — https://www.gmicloud.ai/en/blog/nvidia-h100-gpu-pricing-2026-rent-vs-buy-cost-analysis
Red Hat · 233 % 3-year ROI and 13 months to payback with Red Hat AI (feb. 2026) — https://www.redhat.com/en/blog/233-3-year-return-investment-and-13-months-payback-red-hat-ai
VentureBeat · 5 % GPU utilization: the 401 billion USD AI infrastructure problem — https://venturebeat.com/infrastructure/5-gpu-utilization-the-401-billion-ai-infrastructure-problem-enterprises-cant-keep-ignoring/
DZone · Queueing Theory for LLM Inference — https://dzone.com/articles/queueing-theory-for-llm-inference
GuideLLM · Evaluate LLM deployments for real-world inference (Red Hat Developer, jun. 2025) — https://developers.redhat.com/articles/2025/06/20/guidellm-evaluate-llm-deployments-real-world-inference

Energy in TCO and in regulation: cost, PUE, CSRD and the new European reporting obligation for datacenters

Tue, 16 Jun 2026 10:00:00 +0200

Notation: amounts in N € or N USD; decimals with a point; thousands separated by a comma. The dollar symbol is not used (it is a formula delimiter). Generic example hardware (4×H100 SXM node, 3.6 kW IT). No real infrastructure.

TL;DR

A 4×H100 SXM node at Spanish industrial prices (0.116 €/kWh, PUE 1.54) spends ~5,475 € on energy per year: 6–8 % of total opex when staff and maintenance dominate, but up to 18–22 % of opex in clusters with a solar PPA and cheap colocation. Energy is not the main TCO lever; it is the only TCO variable that European regulation requires you to measure, publish and certify.

Since September 2024, every datacenter with IT demand ≥ 500 kW in the EU is obliged to report PUE, WUE, ERF and REF to the ReportENER database (Delegated Regulation EU 2024/1364). From August 2027 an automatic sustainability label will be generated from that data. Companies subject to CSRD have to publish Scope 1, 2 and 3 in their sustainability report, along with emissions intensity, which turns the gCO₂/token ratio into a corporate accounting figure, not just an engineering one.

Track context

This article is C8 of the energy pillar. The previous ones:

The full TCO model that this article extends is in:

For the context of technical security compliance:

Technical controls: ENS, ISO 42001 and the EU AI Act

Energy as a TCO component

The reference model

The full TCO model of the on-premise cluster is developed in A8. The reference node: 4×H100 SXM5 80 GB, IT draw ~3.6 kW, 8-node cluster, 3-year amortisation.

Annual opex per node in the base scenario (8-node cluster, Spain):

Opex item	€/year per node	% of total opex
Staff (0.5 FTE pro rata)	7,500	26.8 %
Maintenance / support / failures	9,000	32.1 %
Rack colocation (Spain, high density)	6,000	21.4 %
Energy (PUE 1.54; 0.116 €/kWh)	5,475	19.6 %
Total opex	~28,000	100 %

The annual energy cost formula

$$\text{annual energy cost} = P_{\text{IT}} \times \text{PUE} \times 8{,}760\,\text{h} \times p_{\text{kWh}}$$

where $P_{\text{IT}}$ is the power of the IT equipment (kW), PUE the infrastructure multiplier, and $p_{\text{kWh}}$ the price of electricity (€/kWh).

For the reference node:

$$\text{energy/year} = 3.6\,\text{kW} \times 1.54 \times 8{,}760\,\text{h} \times 0.116\,\frac{\text{EUR}}{\text{kWh}} \approx 5{,}475\,\text{EUR}$$

Sensitivity to energy price and to PUE

Variation in the price of energy has a moderate impact on total TCO when hardware capex dominates:

Energy price (€/kWh)	Energy/year (€)	% opex	€/GPU-hour (70 % util., base)	Var. vs base
0.034 (Spanish solar PPA Q3 2025)	1,604	6.8 %	3.00	−12 %
0.116 (Spanish industrial, base)	5,475	19.6 %	3.39	reference
0.160 (European industrial average)	7,550	25.4 %	3.54	+4 %
0.200 (high tariff, no PPA)	9,437	30.2 %	3.67	+8 %

Variation in PUE is also moderate in total TCO but relevant in the absolute cost of energy and in regulatory reporting:

PUE	Energy/year (€, 0.116 €/kWh)	% opex	€/GPU-hour (70 % util.)
1.15 (liquid cooling)	2,166	8.4 %	3.21
1.20 (modern datacenter)	2,259	8.7 %	3.23
1.48 (datacenters <5 years, Uptime 2025)	3,490	13.1 %	3.33
1.54 (global average, Uptime 2025)	5,475	19.6 %	3.39
1.80 (legacy colocation)	6,400	23.2 %	3.47

The difference between PUE 1.15 and PUE 1.80 in €/GPU-hour is only +8 %, because capex still dominates, but in absolute energy consumed the difference is ×2.96: 2,166 € against 6,400 € a year per node. That difference in absolute energy is exactly what the regulatory framework is going to measure, publish and, in the future, classify.

Energy as a fraction of total TCO

Annual TCO per node includes amortised capex + opex:

$$\text{annual TCO per node} = \frac{\text{capex}}{\text{years}} + \text{annual opex} = 55{,}300\,\text{EUR} + 28{,}000\,\text{EUR} = 83{,}300\,\text{EUR}$$

With these numbers, energy represents:

$$\frac{5{,}475}{83{,}300} \approx 6.6\,\%\ \text{of the all-in TCO}$$

In the low-opex scenario (solar PPA, own datacenter, no colocation):

$$\frac{1{,}604}{68{,}300} \approx 2.3\,\%\ \text{of the TCO}$$

In the high-opex scenario (open-market tariff, expensive colocation):

$$\frac{9{,}437}{130{,}300} \approx 7.2\,\%\ \text{of the TCO}$$

The weight of energy in all-in TCO ranges from 2 % to 7 %. The more capex dominates, the lower the weight of energy. Energy matters more in TCO the longer the amortisation and the lower the relative capex, that is, in scenarios of cheaper hardware or long amortisation periods.

European regulatory framework 2024-2026

Map of the rules

The European framework on datacenters and energy is built from four regulatory layers with different scopes of application, required metrics and effective dates:

Rule	Legal basis	Main obligation	Who it applies to	From when
EED 2023/1791, Art. 12	Directive (EU) 2023/1791	Annual reporting of energy and sustainability KPIs to an EU database	Datacenters ≥ 500 kW IT in the EU	15 Sep. 2024 (1st report)
Delegated Reg. 2024/1364	Art. 33(3) EED	Specifies KPIs, formats and reporting deadlines (Phase 1 of the assessment scheme)	Datacenters ≥ 500 kW IT in the EU	Published 17 May 2024; in force
EU rating scheme (Phase 2, 2026 draft)	Art. 33(3) EED	Automatic sustainability label generated by the ReportENER database	Datacenters that report under 2024/1364	Automatic label from 15 Aug. 2027
CSRD / ESRS E1	Directive 2022/2464/EU + Delegated Regulation (EU) 2023/2772	Sustainability report: Scope 1-2-3 emissions, emissions intensity, transition plan	Large EU companies (>500 employees, from FY 2024); expanding progressively	Financial years from 1 Jan. 2024 (first reports 2025); listed SMEs from 2026
PNIEC 2023-2030 (Spain)	National Integrated Energy and Climate Plan	Energy efficiency 44 %, renewables 81 % in electricity; datacenters included in the target; Royal Decree in progress (consultation Sep. 2025)	Spain	2025-2030

Energy Efficiency Directive: EED 2023/1791

Directive (EU) 2023/1791, published in the OJEU of 20 September 2023 (L 231), recasts the earlier Directive 2012/27/EU. Article 12 introduces for the first time a reporting obligation specific to datacenters at European scale.

Threshold of application: datacenters with an electrical power demand of installed IT systems ≥ 500 kW. Member States may lower this threshold (Germany lowered it to 300 kW; France to 100 kW).

Types of datacenter covered: (a) enterprise datacenters, operated by a company solely for its own IT needs; (b) colocation datacenters, where the customer installs and manages its own equipment; (c) co-hosting or multi-tenant datacenters, where the customer accesses shared infrastructure.

What must be reported: 18 key performance indicators (KPIs) that include installed IT demand, total energy consumption, water consumption, reused waste heat, renewable energy consumed, and the four sustainability indicators: PUE, WUE, ERF and REF.

Deadlines:

First report (2023 data): before 15 September 2024.
Second report (2024 data): before 15 May 2025.
Subsequent annual reports: before 15 May each year.

Delegated Regulation (EU) 2024/1364

Commission Delegated Regulation (EU) 2024/1364 of 14 March 2024 (OJEU L, 17 May 2024) constitutes Phase 1 of establishing the Union’s common assessment regime for datacenters. It specifies:

The performance indicators and the information that operators must report to the European ReportENER database.
The calculation formulas for the four sustainability KPIs (PUE, WUE, ERF, REF).
The reporting format and the classification categories by datacenter type and size.

The Commission has made the ReportENER database available as the single entry point for reporting, or via the equivalent national systems of each Member State.

Phase 2: rating scheme and sustainability label

In March 2026 the European Commission published the public consultation on the draft second Delegated Regulation on the datacenter sustainability rating regime, Phase 2 of the scheme. The consultation was open until 23 April 2026.

The draft establishes that the ReportENER database will automatically generate an electronic sustainability label for every datacenter that has reported under Regulation 2024/1364. The label will show:

PUE and WUE classes based on indices calculated under Annex I of the draft.
REF (renewable energy factor) and ERF (energy reuse factor).
Information on grid functions (demand response capability).

Date of first issue of labels: 15 August 2027, and every year thereafter.

The draft also introduces amendments to Regulation 2024/1364, among them: the possibility of voluntary reporting for datacenters <500 kW, the removal of data traffic indicators from the list of mandatory KPIs, and the relaxation of reporting obligations for colocation operators (reflecting the difficulty of obtaining tenant data for confidentiality reasons).

The Commission’s technical report of October 2025 anticipates that the minimum performance standards (MPS) will be finalised between late 2027 and early 2028, and that they will progressively become mandatory for new builds and significant refurbishments.

CSRD and ESRS E1: corporate emissions reporting

The Corporate Sustainability Reporting Directive (CSRD, Directive 2022/2464/EU) requires companies in scope to publish a sustainability report drawn up under the European Sustainability Reporting Standards (ESRS). The ESRS E1 standard covers climate change.

Temporal scope of application:

Large companies (>500 employees, already subject to NFRD): financial year 2024 data, report published in 2025.
Other large companies (>250 employees or >40 M € turnover or >20 M € balance sheet): financial year 2025 data, report in 2026.
Listed SMEs: financial year 2026 data, report in 2027 (with the possibility of an extension).

What ESRS E1 requires in relation to datacenters and energy:

E1-4: GHG emission reduction targets in absolute value and in intensity (e.g. gCO₂eq per unit of product or service).
E1-5: Energy consumption and mix (renewable vs non-renewable energy, energy intensity).
E1-6: Gross GHG emissions Scope 1, Scope 2 (location-based and market-based) and Scope 3, without netting off offsets; emissions intensity.
E1-7: Removals and carbon projects (relevant for offsets that do not replace reduction).

The amended version of ESRS E1 (EFRAG Exposure Draft of July 2025, submitted to the European Commission in November 2025) expands the disclosure requirements from 9 to 11 and is expected to apply from January 2027.

Relevance for datacenter operators: a company that runs an on-premise GPU cluster for its own AI workloads has those datacenters within the perimeter of its Scope 1 (if it has its own generators) and Scope 2 (purchased electricity). If it contracts cloud services or colocation, those datacenters fall into its Scope 3 (category 1 or 11 under the GHG Protocol).

Spain: PNIEC context and the Royal Decree in progress

Spain’s National Integrated Energy and Climate Plan (PNIEC) 2023-2030, updated in September 2024 (MITECO), sets targets of 81 % renewables in electricity and 44 % energy efficiency for 2030. Datacenters are included among the sectors under intensive efficiency monitoring.

In August-September 2025, MITECO opened the public consultation on the draft Royal Decree regulating the energy efficiency and sustainability of datacenters, transposing it from Art. 12 of EED 2023/1791. The Royal Decree introduces additional requirements for datacenters with total installed rated power above 1 MW: an obligation to use the waste heat generated to supply climate control services or other heat applications.

The deadline for incorporating Art. 12 of the EED expired on 11 October 2025. The Royal Decree is awaiting the opinion of the Council of State.

Regulatory metrics: definitions and formulas

PUE — Power Usage Effectiveness

Defined in ISO/IEC 30134-2:2018 (European equivalent EN 50600-4-2). It is the central metric of the regulatory framework:

$$\text{PUE} = \frac{E_{\text{DC}}}{E_{\text{IT}}}$$

where $E_{\text{DC}}$ is the total energy consumed by the datacenter (including cooling, electrical distribution, lighting and losses) and $E_{\text{IT}}$ is the energy consumed by the IT equipment (servers, storage, network).

PUE = 1.0: theoretical perfection (all the energy goes to compute).
PUE = 1.54: 2025 global average (Uptime Institute, 15th annual survey).
PUE = 1.10–1.15: hyperscaler range.
PUE ≥ 1.80: legacy, unoptimised facilities.

Regulation 2024/1364 requires annual PUE to be measured and reported. The Phase 2 draft introduces PUE classes (A to D or similar) for the sustainability label.

WUE — Water Usage Effectiveness

Defined in ISO/IEC 30134-9:2022:

$$\text{WUE} = \frac{W_{\text{in}}}{E_{\text{IT}}}$$

where $W_{\text{in}}$ is the datacenter’s total water consumption (m³), and the result is expressed in m³/MWh. ReportENER data (2024-2025) shows EU average values between 0.07 and 1.28 m³/MWh depending on Member State, with an EU average of ~0.58 m³/MWh.

WUE is particularly relevant in water-stressed regions, such as southern and inland Spain, where evaporative cooling consumes drinking water.

ERF — Energy Reuse Factor

Defined in ISO/IEC 30134-6:2021:

$$\text{ERF} = \frac{E_{\text{reuse}}}{E_{\text{DC}}}$$

where $E_{\text{reuse}}$ is the energy (waste heat) exported and reused outside the datacenter (e.g. district heating networks, industrial processes). ERF varies between 0 (no reuse) and 1 (all the energy reused, theoretically). The Phase 2 draft proposes minimum ERF thresholds for new facilities.

REF — Renewable Energy Factor

$$\text{REF} = \frac{E_{\text{res,tot}}}{E_{\text{DC}}}$$

where $E_{\text{res,tot}}$ is the total renewable energy consumed (own generation + PPAs + Guarantees of Origin), divided by the datacenter’s total consumption. The Commission’s proposal points to targets of 75–100 % REF by 2030.

CUE — Carbon Usage Effectiveness

Defined in ISO/IEC 30134-8:2022. Not included in the reporting obligation of Regulation 2024/1364, but identified as a future KPI in the Commission’s technical report (Oct. 2025):

$$\text{CUE} = \frac{G_{\text{DC}}}{E_{\text{IT}}}$$

where $G_{\text{DC}}$ is the datacenter’s total GHG emissions (gCO₂eq) and the result is expressed in gCO₂eq/kWh IT. It is equivalent to the product $\text{PUE} \times I_{\text{grid}}$ when only location-based Scope 2 emissions are considered.

Master table: rule × what it requires × metric/threshold × date

Rule	What it requires	Metrics / thresholds	Date of application	To whom
EED 2023/1791, Art. 12	Annual reporting of KPIs to an EU database (ReportENER)	PUE, WUE, ERF, REF + 14 additional KPIs (energy, water, waste heat, renewables)	15 Sep. 2024 (1st report, 2023 data); 15 May annually from 2025	Datacenters ≥ 500 kW IT in the EU
Delegated Reg. (EU) 2024/1364	Technical specification of KPIs and reporting format (Phase 1 of the scheme)	PUE = E DC/E IT; WUE = W in/E IT; ERF = E reuse/E DC; REF = E res/E DC	In force since 17 May 2024	Datacenters ≥ 500 kW IT in the EU
EU rating scheme, Phase 2 (2026 draft)	Automatic electronic sustainability label with PUE and WUE classes	PUE and WUE classes A-D (thresholds in Annex I of the draft; pending final adoption)	Automatic labels from 15 Aug. 2027	Datacenters that have reported under 2024/1364
Minimum performance standards (MPS, expected 2027-2028)	Mandatory minimum PUE, REF and ERF thresholds for new builds and refurbishments	PUE targets differentiated by climate and size; REF ≥ 75-100 % by 2030; minimum ERF for new facilities	Finalisation expected 2027-2028; progressively mandatory	New and refurbished datacenters
CSRD / ESRS E1	Sustainability report: GHG emissions Scope 1, 2 (location- and market-based) and 3; emissions intensity; transition plan	Scope 1+2+3 mandatory in tCO₂eq; intensity in gCO₂eq/functional unit	Data from 1 Jan. 2024 (>500 employees); from 1 Jan. 2025 (>250 employees)	Large EU companies subject to CSRD
PNIEC 2023-2030 / Spanish Royal Decree (in progress)	Energy efficiency and sustainability of datacenters; obligation to use waste heat	Renewables target 81 % by 2030; datacenters ≥ 1 MW: waste heat for climate control or other applications	Royal Decree pending approval (consultation Sep. 2025)	Datacenters in Spain

How Wh/token and gCO₂/token connect with the required reporting

The conversion chain

The technical inference efficiency figure, J/token or Wh/token, measured with tools such as DCGM, NVML or nvidia-smi, becomes a regulatory reporting figure through the following chain:

$$E_{\text{token}} = \frac{P_{\text{IT}}}{T} \quad \text{[Wh/tok]}$$ $$E_{\text{DC/token}} = E_{\text{token}} \times \text{PUE} \quad \text{[Wh/tok adjusted to the datacenter]}$$ $$\text{gCO}_{2}\text{/token} = E_{\text{DC/token}} \times I_{\text{grid}} \quad \text{[gCO}_2\text{eq/tok]}$$

where $T$ is throughput (tok/s), $P_{\text{IT}}$ the power of the IT node, and $I_{\text{grid}}$ the carbon intensity of the grid (gCO₂eq/kWh). The full methodology is in C6.

For the reference node (4×H100 SXM, $P_{\text{IT}} = 3.6\,\text{kW}$, indicative throughput 1,000 tok/s, PUE 1.54):

$$E_{\text{DC/token}} = \frac{3.6\,\text{kW} \times 1.54}{1{,}000\,\text{tok/s}} = 5.544\,\text{J/tok} = 1.54 \times 10^{-3}\,\text{Wh/tok}$$

From Wh/token to the regulatory KPI: aggregation

What regulatory reporting (Regulation 2024/1364) requires is not per-token ratios but annual totals:

$E_{\text{IT}}$: total energy of the IT equipment over the year (kWh).
$E_{\text{DC}}$: total datacenter energy over the year (kWh), including cooling and distribution.
$W_{\text{in}}$: total water consumption (m³).
$E_{\text{reuse}}$: energy (waste heat) exported and reused (kWh).
$E_{\text{res,tot}}$: renewable energy consumed (kWh).

The four sustainability KPIs are calculated from those totals. The link between the per-token ratio and regulatory reporting is:

$$E_{\text{IT,annual}} = \sum_{\text{inferences}} E_{\text{token}} \times N_{\text{tokens}} + E_{\text{idle}} \quad \text{[kWh/year]}$$

The $E_{\text{token}}$ figure is needed in order to attribute total IT consumption to inference workloads (vs idle, vs fine-tuning), which allows the energy intensity of inference to be calculated as a fraction of the datacenter’s total consumption, a figure relevant to ESRS E1 reporting of emissions intensity.

From Wh/token to gCO₂/token as a CSRD accounting figure

Under ESRS E1-6, companies subject to CSRD must report GHG emissions with intensity per functional unit (E1-4). For a company running an on-premise LLM, the natural functional unit is the token generated, the query answered or the user served (under the Green Software Foundation’s SCI specification, ISO/IEC 21031:2022).

gCO₂/token becomes a corporate accounting figure like this:

Technical measurement: J/token per node (DCGM/NVML).
PUE adjustment: Wh/token at datacenter level = J/token × PUE / 3,600.
Carbon intensity: gCO₂/token = Wh/token × $I_{\text{grid}}$ / 1,000.
Annual scaling: tCO₂eq/year = gCO₂/token × annual N tokens / 10⁶ / 10⁶.
ESRS E1-6 reporting: location-based Scope 2 emissions (with national annual average $I_{\text{grid}}$) and market-based (with $I_{\text{grid}}$ according to renewable energy contracts), in tCO₂eq/year + intensity in gCO₂eq/token.

The location-based / market-based distinction from the GHG Protocol Scope 2 Guidance is mandatory under ESRS E1-6: both methods must be reported.

Worked example: from J/token to tCO₂eq/year

Reference node (4×H100 SXM, PUE 1.54, throughput 1,000 tok/s, 70 % utilisation, Spanish average grid intensity 2024 ~108 gCO₂/kWh):

$$E_{\text{DC/token}} = 1.54 \times 10^{-3}\,\text{Wh/tok}$$ $$\text{gCO}_{2}\text{/tok} = 1.54 \times 10^{-3} \times 108 \approx 0.166\,\text{gCO}_{2}\text{eq/tok}$$

Tokens generated per year (1,000 tok/s × 70 % util. × 8,760 h × 3,600 s/h):

$$N_{\text{tokens/year}} = 1{,}000 \times 0.70 \times 8{,}760 \times 3{,}600 \approx 2.2 \times 10^{10}\,\text{tok}$$

Annual location-based Scope 2 emissions per node:

$$\text{tCO}_{2}\text{eq/year} = \frac{0.166 \times 2.2 \times 10^{10}}{10^{12}} \approx 3.65\,\text{tCO}_{2}\text{eq}$$

For an 8-node cluster: ~29.2 tCO₂eq/year. With a solar PPA (effective intensity ~20 gCO₂/kWh market-based): ~5.4 tCO₂eq/year. The difference between the two methods, location-based vs market-based, is ×5.4, and both have to appear in the CSRD report.

Scenario	Grid intensity (gCO₂/kWh)	gCO₂/tok	tCO₂eq/year (1 node, 70 % util.)	ESRS method
Spanish average 2024 (location-based)	108	0.166	3.65	Location-based mandatory
Spanish solar PPA (market-based)	~20	0.031	0.68	Market-based mandatory
French average 2025 (location-based)	19.6	0.030	0.66	Location-based mandatory
German average 2025 (location-based)	328	0.505	11.1	Location-based mandatory

The 500 kW threshold and on-premise AI datacenters

Which datacenters does it affect in practice?

A 4×H100 SXM5 node draws ~3.6 kW IT. The 500 kW IT threshold is equivalent to ~139 nodes of 4×H100 SXM5. For a smaller cluster, the regulatory threshold looks distant.

There are, however, three routes by which it applies to smaller on-premise AI infrastructure:

Route	Explanation
Shared datacenter	If the GPU nodes sit in a larger corporate datacenter that exceeds 500 kW IT in total (not only GPUs), the datacenter as a whole is subject to reporting, and it is the datacenter operator who reports. The end user receives the PUE of its facility.
Threshold lowered by a Member State	France lowered the threshold to 100 kW IT (~28 4×H100 nodes). Germany set it at 300 kW (~83 nodes). In those countries, medium-sized AI clusters are already subject to reporting.
CSRD independent of the datacenter threshold	The CSRD obligation applies to the company according to its size, not to the datacenter. A company subject to CSRD that operates even a single GPU node is obliged to include the Scope 2 emissions of that node in its sustainability report.

PUE as an operational figure and as a regulatory figure

PUE is simultaneously:

An operational figure for the TCO model (multiplier of energy cost and of gCO₂/token).
A regulatory KPI subject to mandatory reporting under Regulation 2024/1364.
A classification criterion in the future Phase 2 sustainability label.

The reference technical standard for measurement is ISO/IEC 30134-2:2018, which defines four levels of PUE measurement depending on where IT power is measured (from the rack to the incoming transformers). Regulation 2024/1364 specifies that PUE measurement for reporting to ReportENER must follow this standard.

For a typical on-premise facility, PUE has not historically been measured with the granularity the standard demands. The first compliance step is to instrument the datacenter with power meters at the points specified by ISO/IEC 30134-2.

Sensitivity of gCO₂/token to PUE and energy price

The following table shows the variation in gCO₂/token for the reference node (4×H100 SXM, 1,000 tok/s, Spain) as a function of PUE and of the energy scenario (which determines the effective carbon intensity):

PUE	Grid intensity (gCO₂/kWh)	gCO₂/1,000 tok	gCO₂/1M tok	Var. vs base
1.15 (liquid, Spain solar)	50	0.064	63.9	−55 %
1.15 (liquid, Spain average)	108	0.138	138.0	−2 %
1.54 (global average, Spain average)	108	0.166	166.3	reference
1.54 (global average, Spain night gas)	220	0.339	338.8	+104 %
1.80 (legacy, Spain average)	108	0.194	194.4	+17 %
1.80 (legacy, Spain night gas)	220	0.396	396.0	+138 %

Grid intensity has more impact than PUE on gCO₂/token: going from Spanish solar (50 gCO₂/kWh) to Spanish night-time gas (220 gCO₂/kWh) multiplies emissions by ×4.4, while going from PUE 1.15 to 1.80 multiplies them only by ×1.57. The combination of a dirty grid plus a high PUE is the worst: ×6.2 against the base case.

For location-based CSRD reporting, the intensity used is the national annual average (or regional if more granular data exists), not the hourly figure. For market-based reporting, the effective intensity depends on the energy contract (PPA, Guarantees of Origin). Carbon-aware temporal shifting (moving load to hours of lower intensity) reduces real emissions but does not directly change location-based reporting, which uses the annual grid average.

Connection between operator metrics and regulator obligations

The map of required data

Technical figure	How it is obtained	Use in TCO	Use in regulatory reporting
$P_{\text{IT}}$ (W)	DCGM / NVML / IPMI	Energy cost formula	$E_{\text{IT,annual}}$ for PUE, WUE, ERF, REF
Throughput (tok/s)	Engine metrics (vLLM, TGI)	€/token, Wh/token	Energy intensity of inference
PUE (measured)	Meters on PDUs and IT equipment (ISO/IEC 30134-2)	Multiplier in the TCO formula	Mandatory ReportENER KPI
Annual $I_{\text{grid}}$ (gCO₂/kWh)	REE / MITECO / national factors	gCO₂/token for carbon-aware	Emission factor for location-based Scope 2 ESRS E1-6
Hourly $I_{\text{grid}}$	ElectricityMaps / Carbon Aware SDK	Carbon-aware scheduling	Location-based Scope 2 with hourly granularity (best practice)
$W_{\text{in}}$ (m³)	Datacenter water meters	Cost of water (where applicable)	WUE for ReportENER
$E_{\text{reuse}}$ (kWh)	Heat export meters	Possible waste heat revenue	ERF for ReportENER
$E_{\text{res,tot}}$ (kWh)	Energy bills with a renewable attribute / GOs	Energy cost with a PPA	REF for ReportENER; market-based Scope 2 for ESRS E1-6
Hardware embodied carbon (gCO₂eq)	Supplier LCA (AWS methodology, where available)	—	Scope 3 cat. 1 for ESRS E1-6

The minimum measurement system for compliance

For an on-premise datacenter subject to the obligations of Regulation 2024/1364 and to CSRD, the minimum measurement system needed is:

A power meter at the datacenter entry point (total datacenter energy, $E_{\text{DC}}$).
Power meters on the IT racks (IT equipment energy, $E_{\text{IT}}$): in practice, metered PDUs or the sum of DCGM/NVML measurements.
A water meter for the datacenter cooling circuit ($W_{\text{in}}$).
A waste heat export meter if the datacenter has a recovery system ($E_{\text{reuse}}$).
A record of energy bills with a breakdown of renewable energy (PPAs, GOs) for calculating REF and market-based Scope 2.

Without this measurement system, reporting to ReportENER cannot be correct and CSRD reporting lacks the data base it needs.

Minimum Performance Standards: the 2027-2028 outlook

The European Commission’s technical report (Assessment, July 2025, published on the EU Op Portal) and EUDCA’s monitoring anticipate that the Minimum Performance Standards (MPS) for datacenters will be defined in the 2027-2028 period with the following expected characteristics (not finalised):

KPI	MPS proposed for new builds	MPS proposed for existing facilities	Horizon
PUE	≤ 1.3 (cold climate); ≤ 1.4 (temperate/warm climate)	Staged targets differentiated by age	2028+
REF	≥ 75 % renewables in 2025; ≥ 100 % by 2030	Progressive, following the CNDCP roadmap	2025-2030
ERF	Minimum threshold for new facilities with usable heat output	Feasibility assessment requirement	2028+
WUE	No general threshold proposed yet; focus on potable vs non-potable water	Monitoring and reporting	2027+

These values are indicative and come from the consultation process; the definitive values will appear in the second or third Delegated Regulation adopted under Art. 33(3) of the EED.

Sources

EUR-Lex · Directiva (UE) 2023/1791 del Parlamento Europeo y del Consejo, de 13 de septiembre de 2023, relativa a la eficiencia energética (EED refundida) — https://eur-lex.europa.eu/legal-content/ES/TXT/?uri=OJ:JOL_2023_231_R_0001
EUR-Lex · Reglamento Delegado (UE) 2024/1364 de la Comisión, de 14 de marzo de 2024, relativo a la primera fase del establecimiento de un régimen de evaluación común de la Unión para centros de datos — https://eur-lex.europa.eu/eli/reg_del/2024/1364/oj/eng
BOE / DOUE · Texto en español del Reglamento Delegado (UE) 2024/1364 — https://www.boe.es/buscar/doc.php?id=DOUE-L-2024-80715
Comisión Europea · Commission adopts EU-wide scheme rating sustainability of data centres (15 mar. 2024) — https://energy.ec.europa.eu/news/commission-adopts-eu-wide-scheme-rating-sustainability-data-centres-2024-03-15_en
Comisión Europea · Rating scheme for data centres in the EU: Commission launches call for feedback (26 mar. 2026) — https://energy.ec.europa.eu/news/rating-scheme-data-centres-eu-commission-launches-call-feedback-2026-03-27_en
DLA Piper · EU Commission proposes new data centre sustainability rating label (20 abr. 2026) — https://www.dlapiper.com/en-us/insights/blogs/environment-health-safety-and-product-compliance/2026/eu-commission-proposes-new-data-centre-sustainability-rating-label
RE24 Energy · What EU New Data Centre Rating Scheme Means for Operators (dic. 2025) — https://re24.energy/blog-what-eu-new-data-centre-rating-scheme-means/
Comisión Europea · Assessment of next steps to promote the energy performance and sustainability of data centres in the EU (2nd Technical Report, jul. 2025) — https://op.europa.eu/o/opportal-service/download-handler?identifier=83be4c3e-5c79-11f0-a9d0-01aa75ed71a1&format=PDF&language=en&productionSystem=cellar
Comisión Europea · Assessment of next steps — October 2025 follow-up — https://op.europa.eu/o/opportal-service/download-handler?identifier=29aa14ed-a4ba-11f0-a7c5-01aa75ed71a1&format=PDF&language=en&productionSystem=cellar
Comisión Europea · Minimum performance standards for EU data centres (preparatory study) — https://energy.ec.europa.eu/resources/preparatory-studies/minimum-performance-standards-eu-data-centres_en
MITECO · Centros de datos: normativa y reporte (España) — https://www.miteco.gob.es/en/energia/eficiencia/centros-de-datos/normativa.html
MITECO · PNIEC 2023-2030 (actualización sept. 2024) — https://www.miteco.gob.es/content/dam/miteco/es/energia/files-1/pniec-2023-2030/PNIEC_2024_240924.pdf
Ashurst · Draft Royal Decree regulating energy efficiency and sustainability of data centres in Spain (sept. 2025) — https://www.ashurst.com/es-es/insights/draft-royal-decree-regulating-energy-efficiency-and-sustainability-of-data-centres-in-spain/
EFRAG · Amended ESRS E1 Exposure Draft (jul. 2025) — https://www.efrag.org/sites/default/files/media/document/2025-07/Amended_ESRS_Exposure_Draft_July_2025_ESRS_E1_0.pdf
Normative · ESRS E1 Explained: CSRD Climate Disclosure (2026) — https://normative.io/insight/esrs-e1/
CSRD Pro · Reporting Cloud Services and Data Centres for CSRD — https://www.csrdpro.com/en/report-cloud-services-data-centres-csrd/
Uptime Institute · Global Data Center Survey 2025 (PUE medio global 1,54) — https://datacenter.uptimeinstitute.com/rs/711-RIA-145/images/2025.Annual.Survey.Report.pdf
SOCOMEC · Energy Efficiency Directive: Essential Guidelines for Data Center Compliance — https://emea.socomec.com/en/news/eu-energy-efficiency-directive-what-new-rules-mean-data-centres
ISO/IEC 30134-2:2018 · Power Usage Effectiveness (PUE) — https://www.iso.org/standard/63451.html
ISO/IEC 30134-6:2021 · Energy Reuse Factor (ERF) — https://www.iso.org/standard/75728.html
ISO/IEC 30134-8:2022 · Carbon Usage Effectiveness (CUE) — https://www.iso.org/standard/82562.html
ISO/IEC 30134-9:2022 · Water Usage Effectiveness (WUE) — https://www.iso.org/standard/82563.html
Green Software Foundation · Software Carbon Intensity (SCI) Specification (ISO/IEC 21031:2022) — https://sci.greensoftware.foundation/
White & Case · Data centres and energy consumption: evolving EU regulatory landscape and outlook for 2026 — https://www.whitecase.com/insight-alert/data-centres-and-energy-consumption-evolving-eu-regulatory-landscape-and-outlook-2026
Datacenter Forum · New reporting requirements for data center operators introduced by the Delegated Regulation (EU) 2024/1364 — https://datacenter-forum.ro/en/new-reporting-requirements-for-data-center-operators-introduced-by-the-delegated-regulation-eu-2024-1364/

Levers for spending fewer watts per token: a quantified catalogue for LLM inference

Tue, 16 Jun 2026 09:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar symbol is not used (on this site it is a formula delimiter).

TL;DR

Seven independent levers act on the numerator or the denominator of the identity $\text{J/token} = \text{power (W)} / \text{throughput (tok/s)}$. Applied in order of cost-benefit: quantisation (FP16→FP8) cuts J/token by 30–50% without changing hardware; power capping to 70% of TDP reduces power by 30% with less than a 10% drop in throughput on memory-bound loads; DVFS in the decode phase saves a further 22–45%; continuous batching lowers J/token by 25–40% against unoptimised stacks; speculative decoding trims up to 29% off J/token at low batch; prefix caching eliminates the prefill recompute for repeated prompts; and scheduling to clean hours shifts the carbon footprint without changing physical consumption. Starting reference: a vLLM FP16 stack on an H100 SXM (700 W TDP) with Llama-3 70B produces ~0.39 J/token under realistic batch loads; an unoptimised stack (PyTorch baseline) starts 2–6× higher. The levers stack; the combined effect can reach a 73% reduction against the unoptimised baseline (arXiv 2504.17674).

The reference identity

$$\text{J/token} = \frac{\overline{P}\ [\text{W}]}{\text{throughput}\ [\text{tok/s}]}$$

where $\overline{P}$ is the average power over the measurement window (see the methodology in energy per token). The levers act on one of the two factors:

They reduce $\overline{P}$: power capping, DVFS, scheduling to low-intensity hours.
They raise throughput (the denominator): quantisation, continuous batching, speculative decoding, prefix caching, chunked prefill, model selection.
Both at once: quantisation (smaller weight footprint → a larger batch fits in VRAM → higher throughput at similar or lower power).

The expanded formula including infrastructure overhead:

$$\text{J/token}_{\text{effective}} = \frac{\overline{P}_{\text{GPU}} + \overline{P}_{\text{rest of node}}}{\text{throughput}_{\text{tok/s}}} \times \text{PUE}$$

For the lever calculations the work is done on $\overline{P}_{\text{GPU}}$ for clarity; the PUE factor (see energy per token) amplifies or attenuates the effect of each lever proportionally.

Master table: levers × impact × cost × tool

Lever	J/token reduction (%)	Effect on latency	Effect on quality	Main OSS tool	Source
Quantisation FP16→FP8	30–50%	−8–33% TPOT (improvement)	<1% degradation on benchmarks	vLLM, TRT-LLM, llm-compressor	arXiv 2504.17674, baseten.co
Quantisation FP16→INT4 (AWQ/GPTQ)	45–60%	latency improves (lower BW)	1–3% degradation depending on model	AutoAWQ, llm-compressor, vLLM	arXiv 2504.17674
Continuous vs static batching	25–40% vs PyTorch	+5× throughput (improvement)	no impact	vLLM, TGI, TRT-LLM	arXiv 2504.17674, TokenPowerBench
Power capping 70% TDP	20–30% power	<10% throughput (decode)	no impact	nvidia-smi, DCGM, Zeus	arXiv 2604.11391, NERSC docs
DVFS decode-phase	22–45% in decode	1–6% total latency	no impact	nvidia-smi, GreenLLM, DVFS-GPT	arXiv 2501.08219, HAL 05190051
Speculative decoding	up to 29% (batch≤16)	TTFT improves	no demonstrable impact	vLLM, TRT-LLM	arXiv 2602.09113
Prefix caching	50–90% on reusable prefill	TTFT −50–80% (repeated prompts)	no impact	vLLM, SGLang	vLLM docs
Chunked prefill	~10–15% on mixed loads	more stable TTFT	no impact	vLLM (SARATHI)	arXiv 2308.16369
SLM vs dense LLM selection	60–80% (7B vs 70B model)	low latency	degradation varies by task	Ollama, vLLM, llama.cpp	arXiv 2504.17674
Scheduling to clean hours	0% J/token, up to −70% CO₂	no impact on the load	no impact	Kueue, Volcano, cron	ACM SLIT, arXiv 2507.09942

Note: the percentages are against the effective J/token of the load, not against idle board power. The effects are always measured under conditions of equal output quality (same model, same serving stack, same prompt distribution).

Lever 1: quantisation

Mechanics

Quantisation reduces the numerical precision of the weights (and optionally the activations), which shortens the time spent reading weights from HBM per generated token. Decode is memory-bandwidth-bound: the bottleneck is reading the weights, not computing. Reducing bit width frees bandwidth for higher throughput or allows a larger batch in VRAM.

Format	Bits/weight	BW reduction vs FP32	J/token vs FP16	Tool
FP32	32	1× (reference)	+100% (worse)	—
BF16/FP16	16	2×	reference	vLLM native
FP8 (W8A8)	8	~4×	−30–50%	vLLM, TRT-LLM, llm-compressor
INT4 AWQ (W4A16)	4	~8×	−45–60%	AutoAWQ, llm-compressor
INT4 GPTQ (W4A16)	4	~8×	−40–55%	AutoGPTQ, vLLM

Concrete reference figures:

Llama-3 405B on H100, FP16→FP8: −30% J/token measured; TPOT improves by 33% (Baseten, baseten.co/blog/33-faster-llm-inference-with-fp8-quantization).
The vLLM+CUDA graphs stack vs the PyTorch baseline: 2–6× more efficient in J/token (arXiv 2504.17674).
Llama-65B on V100 FP16: 3–4 J/token. Llama-3 70B on H100 FP8: 0.39 J/token. Combined hardware+quantisation reduction: ~90% (MLCommons, arXiv 2504.17674).

Quality cost

FP8 quantisation with calibration produces <1% degradation on standard benchmarks (MMLU, HumanEval). INT4 AWQ can produce 1–3% degradation depending on the model and the task; INT4 GPTQ without fine calibration, up to 2–5%. Validate with LLM energy leaderboards and your own quality benchmarks.

OSS tooling

# vLLM with FP8 (load-time quantisation, H100/A100)
vllm serve meta-llama/Llama-3-70B-Instruct \
 --quantization fp8 \
 --dtype auto

# llm-compressor: offline quantisation to W4A16 (AWQ)
python -m llmcompressor.transformers.compression.helpers \
 --model meta-llama/Llama-3-70B-Instruct \
 --recipe w4a16_awq.yaml \
 --output-dir llama-70b-w4a16/

Lever 2: continuous batching

Mechanics

Static batching keeps the batch fixed: if one request finishes early, the slot stays empty until the end of the batch. Continuous batching inserts new requests as soon as a slot frees up, keeping the GPU always at the highest possible utilisation.

The energy effect: GPU power is almost constant (the GPU is always active); the throughput grows. The result is a larger denominator at similar power:

$$\text{J/token}_{\text{continuous batch}} = \frac{\overline{P}}{\text{throughput}_{\uparrow}}$$

Measured figures (arXiv 2504.17674, TokenPowerBench arXiv 2512.03024):

vLLM with PagedAttention vs unoptimised Transformers: −25–40% J/token on high batch loads.
Online continuous batching vs static offline: a further −5% of J/token (small but consistent).
Batch 32→256 tokens: −25% J/token on Llama-3 70B (kernel economies of scale).
Throughput: ×5 against static batching stacks at equivalent load.

The efficiency knee on the H100 sits at around 256–512 tokens of batch; beyond that, J/token flattens or rises slightly.

OSS tooling

vLLM, TGI (HuggingFace Text Generation Inference), TRT-LLM. The relevant flag in vLLM: --max-num-seqs (maximum concurrent sequences); tuning it to the available VRAM maximises the effective batch.

Lever 3: power capping

Mechanics

The power cap sets a ceiling on instantaneous power (W). The GPU respects it by cutting the clock (internal DVFS). The efficiency trick: LLM decode is memory-bandwidth-bound; the GPU is already not at usefully full clock, and lowering the TDP reduces power without degrading throughput proportionally.

The power cap vs throughput curve (H100 SXM)

Figures from (arXiv 2604.11391) and NERSC (docs.nersc.gov/jobs/power-capping):

Power cap (W)	% of TDP (700 W)	Relative throughput (memory-bound)	Efficiency (tok/W)
700 W	100%	100%	1.00×
560 W	80%	~95%	~1.19×
490 W	70%	~90–95%	~1.27× (sweet spot)
420 W	60%	~80–85%	~1.33× (latency rises)
350 W	50%	~75% (the H100 reaches maximum BW at 350 W memory-bound)	~1.50×
200 W	29%	~50%	frequency saturation

Key finding from arXiv 2604.11391: the H100 reaches its maximum effective memory bandwidth with only ~350 W on memory-bound loads (STriad benchmark), against a TDP of 700 W. Going from 700 W to 490 W (−30% power) produces less than a 10% drop in decode throughput: J/token improves by ~20–27%.

For compute-bound loads (heavy prefill), the sweet spot shifts to 400–500 W, where performance scales almost linearly with the cap; from 500 W to 700 W, the gain is only ~10% extra throughput for each additional 100 W.

J/token formula with a power cap

$$\text{J/token}(P_{\text{cap}}) = \frac{P_{\text{cap}} \times \eta(P_{\text{cap}})}{\text{throughput}_{\text{base}} \times f(P_{\text{cap}})}$$

where $\eta(P_{\text{cap}})$ is the fraction of power actually consumed under the cap (on memory-bound loads, the GPU no longer consumes the whole cap), and $f(P_{\text{cap}})$ is the resulting fraction of throughput. In the H100’s 490–560 W zone, $\eta \approx 0{.}9$ and $f \approx 0{.}92\text{–}0{.}95$: the numerator falls more than the denominator, so J/token improves.

Commands

# Set the power cap to 490 W on GPU 0 (H100 SXM, 700 W TDP)
sudo nvidia-smi -i 0 -pl 490

# Check the applied limit and the power draw in real time
nvidia-smi --query-gpu=power.limit,power.draw --format=csv -l 1

# With DCGM: read DCGM_FI_DEV_POWER_USAGE in Prometheus
# and DCGM_FI_DEV_POWER_LIMIT to confirm the cap is active

# Apply the cap to every GPU on the node (systemd/cron script)
for i in $(seq 0 3); do
 sudo nvidia-smi -i $i -pl 490
done

With DCGM in production, the cap can be set via dcgmi policy --set powercap:490 or through the NVIDIA GPU Operator (the powerLimit field in the DevicePlugin). Zeus (ml.energy/zeus) can sweep caps automatically and measure the resulting J/token per configuration.

Lever 4: DVFS (Dynamic Voltage and Frequency Scaling)

Mechanics

DVFS is finer-grained than a power cap: it allows the SM frequency to be set directly (nvidia-smi -lgc to lock the GPU clock) or left for the driver to adjust dynamically according to available power and load. In LLM inference, the asymmetry between phases is the key fact:

Prefill: compute-bound. Benefits from maximum clock. Sensitive to frequency reduction.
Decode: memory-bandwidth-bound. The SM clock barely matters; what rules is HBM bandwidth.

This asymmetry is what LLM-specific DVFS systems exploit:

System	Energy saving	Latency penalty	Hardware	Source
DVFS-GPT (HAL)	up to 32% at iso-latency	~1–2%	A100, RTX 4090	HAL 05190051
EcoInfer	up to 25.4% (average 21.5%)	<5%	A100	arXiv MDPI Electronics
DualScale (disaggregated prefill/decode)	22–45% with SLO	1–6%	H100	arXiv 2602.18755
GreenLLM (SLO-aware DVFS)	18–30%	within the SLO	—	arXiv 2508.16449

The paper arXiv 2501.08219 measures that reducing the GPU frequency from 2,842 MHz to 180 MHz achieves an average saving of 42% with only a 1–6% latency increase in the decode phase, because decode is memory-bound and the SM frequency is not the bottleneck.

Commands (manual DVFS)

# Lock the SM clock at 1,200 MHz (H100 boost: 1,980 MHz) for decode
sudo nvidia-smi -i 0 -lgc 1200,1200

# Restore the dynamic clock
sudo nvidia-smi -i 0 -rgc

# Persistent mode (needed for the lock to survive between jobs)
sudo nvidia-smi -pm 1

For automatic DVFS tuned to the SLO (target TPOT), systems such as GreenLLM or DVFS-GPT measure the time of each iteration and adjust the frequency in real time, without overrunning the latency budget.

Lever 5: model selection (SLM, MoE vs dense)

The effect of scale

J/token scales with model size, but sublinearly. For Llama-3 (arXiv 2504.17674):

Model	Parameters	Relative J/token	Factor vs 1B
Llama-3 1B	1B	1.0×	reference
Llama-3 8B	8B	~2.3×	×2.3 (not ×8)
Llama-3 70B	70B	~7.3×	×7.3 (not ×70)
Llama-3 405B	405B	~40×	×40 (not ×405)

J/token grows ~7.3× from 1B to 70B, while parameters grow ×70: hardware efficiency (caches, tensor parallelism, kernels) cushions the scaling. Even so, moving from 70B to 8B cuts J/token by ~68% if the quality is sufficient for the task.

MoE vs dense

MoE models activate only a subset of experts per token (typically 2 of 8–64), which reduces active compute. But there is an implementation trap:

OLMoE-1B-7B (MoE, 1B active) vs OLMo-1B (dense): the MoE consumes 54.24% more energy per token than the dense model of the same active size (arXiv 2504.17674), because the fused expert kernels are ~19–63% slower than an equivalent dense GEMM.
OLMoE-1B-7B vs OLMo-7B (dense): here the MoE is indeed more efficient (7B active → only 1B active, a ~7× reduction in active FLOPs per token).

The rule: MoE is efficient when the comparison is equal total parameters, fewer active (for example Mixtral 8×7B vs dense Llama-3 70B). It is not efficient when the comparison is equal active parameters per token.

Model selection table by use case

Use case	Recommended model	Efficiency rationale
Simple chat, entity extraction	SLM 7–8B FP8	−68% J/token vs 70B
RAG with long contexts	70B FP8 or MoE 8×7B	optimal quality/J at K>2K tokens
Complex code, reasoning	70B FP8 or 405B INT4	quality is non-negotiable
Classification, embeddings	dedicated model (e.g. E5, BGE)	orders of magnitude less J
Deferrable offline batch	large MoE (DeepSeek, Mixtral)	high quality, batch throughput

Lever 6: speculative decoding

Mechanics

Speculative decoding uses a small draft model to propose several tokens, which the target model verifies in parallel in a single forward pass. If the tokens are accepted, K tokens are generated per step instead of 1: potentially K× throughput.

The energy efficiency condition

The J/token benefit depends on batch size (arXiv 2602.09113):

Batch size	Effect on J/token	Reason
1–16 (low batch)	−up to 29%	the draft model fills the waiting time; verification is cheap
32–64 (medium batch)	~0% (neutral)	verification overhead cancels out the gain
128+ (high batch)	+25% (worse)	GPU saturated; the draft model adds overhead without saving

Conclusion: speculative decoding is a J/token lever only in a low-latency regime (batch≤16); in high-throughput serving it is counterproductive in J/token terms (although it reduces TPOT latency).

OSS tooling

vLLM supports speculative decoding with --speculative-model and --num-speculative-tokens. TRT-LLM does too. The draft model must come from the same family (for example Llama-3 8B as draft for Llama-3 70B).

Lever 7: prefix caching and chunked prefill

Prefix caching

The prefill KV-cache of a shared prompt (system prompt, few-shot examples, document context) is identical across requests. Storing and reusing it eliminates the prefill recompute for that fraction of the prompt.

Energy impact: prefill is compute-bound and draws power spikes. Removing the repeated prefill removes those spikes. In scenarios with a fixed 1,000-token system prompt and 200-token user prompts, the saving is ~1,000/(1,000+200) = ~83% of prefill energy. In total J/token, the effect depends on the length of the shared prompt against the generation length.

Enabling it in vLLM:

vllm serve meta-llama/Llama-3-70B-Instruct \
 --enable-prefix-caching \
 --max-num-seqs 256

Chunked prefill (SARATHI)

Chunked prefill splits the prefill into fixed-size chunks and processes each chunk together with decode tokens from other requests. The benefit: the prefill of a long prompt no longer blocks the decode of other requests (reducing TPOT spikes), which allows higher GPU utilisation over time.

In terms of total J/token, chunked prefill reduces the node’s peak power (fewer prefill spikes) and improves average utilisation, which translates into ~10–15% less J/token on mixed loads with long prompts (arXiv 2308.16369).

# Enable chunked prefill in vLLM (chunk size: 512 tokens)
vllm serve meta-llama/Llama-3-70B-Instruct \
 --enable-chunked-prefill \
 --max-num-batched-tokens 512

Lever 8: scheduling to clean hours

This lever does not reduce physical J/token, but it does reduce the gCO₂/token of deferrable load. The mechanics: moving training, document ingestion, batch re-ranking and fine-tuning to the hours of lowest grid carbon intensity.

Impact figures for temporal shifting (ACM SLIT, arXiv 2507.09942):

CO₂ reduction from temporal shifting: up to 34.7% for flexible loads.
With carbon-aware reinforcement learning (Eco-Orchestrator): up to a 70% offset in integrated deployments.
In Spain, the hourly variation in intensity is ~80–250 gCO₂/kWh: a factor of ~3× within the same day (see from watt to carbon).

The practical implementation uses Kueue or Volcano on Kubernetes, with a grid intensity metric (the ElectricityMaps API or esios) as the scheduling signal. Loads are marked with a carbon-deadline annotation (the maximum tolerable wait) and the scheduler places them in the minimum-intensity hours within that window.

# Kueue annotation for carbon-aware scheduling
metadata:
 annotations:
 kueue.x-k8s.io/carbon-aware: "true"
 kueue.x-k8s.io/carbon-deadline: "8h"

The power cap vs throughput curve: the sweet spot

Reading the chart: the power curve (solid line) falls faster than the throughput curve (dashed line) in the 490–560 W zone. Above 560 W, each additional 70 W gives only ~5% extra throughput; below 420 W, throughput starts falling faster than power. The efficiency sweet spot (minimum J/token) sits at 490–560 W for decode-bound loads.

For compute-bound loads (pure prefill), the sweet spot shifts to 500–600 W: prefill saturates SM compute and needs more power to sustain throughput.

How to measure the effect and validate quality

Measurement stack

The full stack for measuring the effect of each lever:

What to measure	Tool	Metric
Real-time GPU power	DCGM (`DCGM_FI_DEV_POWER_USAGE`)	W per GPU, 100ms
J/token in production	Kepler + vLLM tokens	`rate(kepler_joules) / rate(vllm_tokens)`
J/token on the bench	Zeus (`ZeusMonitor`)	J measured during the benchmark
Throughput	vLLM metrics (`vllm:generation_tokens_total`)	tok/s
TPOT latency	vLLM (`vllm:e2e_request_latency`)	ms/tok
Output quality	lm-evaluation-harness	MMLU, HumanEval, etc.

See the full stack in energy tooling.

Measurement protocol per lever

# Real-time J/token (Kepler + vLLM in Prometheus)
sum(rate(kepler_container_joules_total{container="vllm"}[5m]))
 /
sum(rate(vllm_generation_tokens_total[5m]))

# Average power per GPU (DCGM)
avg by (gpu) (DCGM_FI_DEV_POWER_USAGE)

# Check that the power cap is active
avg by (gpu) (DCGM_FI_DEV_POWER_LIMIT)

With Zeus, the configuration sweep is automatic:

from zeus.monitor import ZeusMonitor

monitor = ZeusMonitor(gpu_indices=[0, 1, 2, 3])

with monitor.begin_window("fp8_inference"):
 # run the benchmark with vLLM FP8
 run_benchmark(model="llama-3-70b-fp8", num_tokens=10_000)

result = monitor.end_window("fp8_inference")
print(f"J/token: {result.total_energy / 10_000:.4f}")

Quality validation

Validating that the lever does not degrade quality is done with lm-evaluation-harness (github.com/EleutherAI/lm-evaluation-harness):

# Compare MMLU between FP16 and FP8
lm_eval --model vllm \
 --model_args pretrained=meta-llama/Llama-3-70B-Instruct,dtype=float16 \
 --tasks mmlu --num_fewshot 5 --output_path ./results/fp16/

lm_eval --model vllm \
 --model_args pretrained=meta-llama/Llama-3-70B-Instruct,quantization=fp8 \
 --tasks mmlu --num_fewshot 5 --output_path ./results/fp8/

The MMLU score difference between FP16 and FP8 with calibration is typically <1%. Without calibration it can be 2–4%.

Pareto decision table (energy / latency / quality)

Lever	J/token	TPOT (latency)	Quality	Implementation cost	Reversibility
FP8 quantisation	↓↓↓	↓ (improves)	~ (−<1%)	Low (1 flag)	High
INT4 AWQ quantisation	↓↓↓↓	↓ (improves)	↓ (−1–3%)	Medium (recalibrate)	High
Continuous batching	↓↓↓	~ (neutral on TPOT)	~	Low (new engine)	High
Power cap 70% TDP	↓↓	↑ <10%	~	Very low (1 command)	Immediate
DVFS decode	↓↓↓	↑ <6%	~	Medium (script + tuning)	High
Speculative decoding	↓ (low batch)	↓ (improves TTFT)	~	Medium (draft model)	High
Prefix caching	↓↓ (prefill)	↓↓ (TTFT improves)	~	Low (1 flag)	High
Chunked prefill	↓	↑ stable TPOT	~	Low (1 flag)	High
SLM (7B vs 70B)	↓↓↓↓	↓↓ (improves a lot)	↓↓ (−variable)	High (quality re-eval)	Medium
MoE (vs dense, same active size)	↑ (+54%)	↑	~	—	—
Scheduling to clean hours	~ J/token, ↓↓ CO₂	~ (for deferrable)	~	Medium (scheduler)	High

Key: ↓↓↓↓ strong improvement · ↓↓↓ significant improvement · ↓↓ moderate improvement · ↓ slight improvement · ~ neutral · ↑ worsens · ↑↑ worsens significantly.

Stacking levers: the combined effect

The levers are mostly orthogonal and they stack. The maximum combined impact documented in (arXiv 2504.17674):

“Correct application of the relevant inference optimisations can reduce total energy consumption by up to 73% against unoptimised baselines.”

Recommended sequence by cost-benefit:

Optimised engine (vLLM, TRT-LLM) with continuous batching: −25–40% immediately.
FP8 quantisation (if the hardware supports it: H100, A100 with limited support): a further −30–50% on the remaining J/token.
Power cap at 70% TDP: −20–30% power with <10% of throughput. Low cost.
Prefix caching: turn it on if there are system prompts or repeated contexts. No cost.
Chunked prefill: if the load has long variable prompts.
DVFS in decode: a further −22–45% if the SLO allows it.
Speculative decoding: only if the batch is low (<16) and TTFT latency matters.
Model selection: move to an SLM if the quality is sufficient (evaluate with lm-evaluation-harness on your specific task).

Quantified example on 4×H100 SXM (2,800 W node TDP), Llama-3 70B, chat load:

Configuration	Power (W)	Throughput (tok/s)	J/token
PyTorch FP16, static batching	~2,100	~400	5.25
vLLM FP16, continuous batching	~2,200	~1,800	1.22
vLLM FP8, continuous batching	~2,100	~2,400	0.875
+ power cap 490 W/GPU (1,960 W node)	~1,960	~2,160	0.907
+ prefix caching (40% hit rate)	~1,960	~2,600	0.754
+ DVFS decode 1,200 MHz	~1,500	~2,400	0.625

The values are illustrative estimates based on the cited papers applied to a 4×H100 node. Real J/token depends on the prompt distribution, the effective batch and the temperature of the models. Measuring it with Zeus or DCGM+Kepler is mandatory.

Reference hardware: TDP and power cap ranges

GPU	TDP (W)	Supported cap range	Decode sweet spot
H100 SXM 80GB	700	200–700 W	~490–560 W
H100 PCIe 80GB	350	100–350 W	~245–280 W
A100 SXM 80GB	400	100–400 W	~280–320 W
A100 PCIe 80GB	300	100–300 W	~210–240 W
L40S 48GB	350	100–350 W	~245–280 W
RTX 5090 (reference, PCIe)	575	~100–575 W	~400–460 W

For the exact ranges: nvidia-smi -q -d POWER | grep -E "Min|Max|Current".

Executive summary by role

Role	First action	Second action	Key tool
Cluster operator	`nvidia-smi -pl [70% TDP]` on every node	Enable persistent mode	nvidia-smi, DCGM
Serving engineer	Migrate to vLLM FP8 with continuous batching	Enable prefix caching	vLLM, llm-compressor
Platform architect	Select the smallest model that meets the quality SLO	Design a carbon-based scheduling policy	lm-evaluation-harness, Kueue
MLOps team	Set up a power cap sweep with Zeus	Instrument J/token in Prometheus	Zeus, DCGM exporter

See the full measurement stack in energy tooling and the J/token methodology in energy per token. For the context of reference energy leaderboards and benchmarks, see LLM energy leaderboards. The connection with full TCO is in GPU utilisation as FinOps.

Sources

arXiv 2504.17674 · Energy Considerations of LLM Inference and Efficiency Optimizations (CMU/Hugging Face, 2025) — https://arxiv.org/pdf/2504.17674
arXiv 2604.11391 · Architectural Trade-offs: power-capping NVIDIA H100 and H200 (FAU Erlangen, 2026) — https://arxiv.org/html/2604.11391v1
arXiv 2512.03024 · TokenPowerBench: Benchmarking the Power Consumption of LLM Inference — https://arxiv.org/pdf/2512.03024
arXiv 2602.09113 · Benchmarking the Energy Savings with Speculative Decoding Strategies — https://arxiv.org/pdf/2602.09113
arXiv 2501.08219 · Characterizing LLM Inference Energy-Performance Tradeoffs across Workloads and GPU Scaling — https://arxiv.org/abs/2501.08219
arXiv 2602.18755 · DualScale: Energy-Efficient Disaggregated LLM Serving via Phase-Aware Placement and DVFS — https://arxiv.org/pdf/2602.18755
arXiv 2508.16449 · GreenLLM: SLO-Aware Dynamic Frequency Scaling for Energy-Efficient LLM Serving — https://arxiv.org/pdf/2508.16449
HAL 05190051 · DVFS-GPT: Dynamic Voltage and Frequency Scaling for Energy-Efficient LLMs — https://hal.science/hal-05190051
arXiv 2308.16369 · SARATHI: Efficient LLM Inference by Piggybacking Decodes with Chunked Prefills — https://arxiv.org/pdf/2308.16369
arXiv 2507.09942 · Green-LLM: Optimal Workload Allocation for Environmentally-Aware Distributed Inference — https://arxiv.org/pdf/2507.09942
arXiv 2505.06371 · The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization — https://arxiv.org/html/2505.06371v1
Zeus Project (ml.energy) — https://ml.energy/zeus/
Zeus · PyTorch blog: Deep Learning Energy Measurement and Optimization — https://pytorch.org/blog/zeus/
Baseten · 33% faster LLM inference with FP8 quantization — https://www.baseten.co/blog/33-faster-llm-inference-with-fp8-quantization/
NVIDIA · Managing Power Capping (DGX H100/H200 User Guide) — https://docs.nvidia.com/dgx/dgxh100-user-guide/power-capping.html
NERSC · GPU Power Capping documentation — https://docs.nersc.gov/jobs/power-capping/
MLCommons · MLPerf Inference v5.1 results — https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/
vLLM · Disaggregated Prefill documentation — https://docs.vllm.ai/en/latest/features/disagg_prefill/
Microsoft Research · Characterizing Power Management Opportunities for LLMs in the Cloud (ASPLOS 2024) — https://www.microsoft.com/en-us/research/wp-content/uploads/2024/03/GPU_Power_ASPLOS_24.pdf
EcoInfer (MDPI Electronics) · Optimizing Energy Efficiency with Latency Guarantees Through Iteration-Level GPU Frequency Control — https://doi.org/10.3390/electronics15102139

From watt to carbon, honestly: PUE, grid intensity and gCO₂ per token

Tue, 16 Jun 2026 08:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar symbol is not used (on this site it is a formula delimiter). Generic example hardware (4×H100 SXM 80 GB); no real infrastructure. Emission figures in gCO₂eq/kWh (CO₂ equivalents).

TL;DR

A model that consumes 2 J/token at the board (4×H100 SXM, PUE 1.45) emits between $0.08$ and $1.05$ gCO₂ per 1,000 tokens depending on the region and the hour: $0.08$ in nuclear France in the small hours, $0.22$ in solar Spain at midday, $0.72$ in Spain on night-time gas, and $1.05$ in Germany on coal. The difference, a factor of ×13, is set by three variables external to the hardware: the datacenter PUE (Uptime Institute 2025: global average 1.54; hyperscale 1.10–1.15), the grid carbon intensity (which varies in real time, from 5 to 600+ gCO₂/kWh depending on zone and hour), and the type of emissions being counted (location-based, market-based, marginal). Moving batch work to hours of lower carbon intensity, temporal carbon-aware shifting, cuts the emissions of deferrable work by between 20 % and 65 % without touching the hardware. The SCI specification (Software Carbon Intensity, Green Software Foundation, ISO/IEC 21031:2022) formalises the metric as $\text{SCI} = (E \times I + M) / R$, where $M$ captures the embodied carbon of the hardware, the part that almost no LLM inference analysis includes today.

Track context

This article is C6 of the energy pillar. The previous ones:

The starting point is the J/token already measured (methodology in C2, tools in C3). This article converts that number into gCO₂/token.

PUE: the first multiplier

Definition

PUE (Power Usage Effectiveness) measures what fraction of the total energy consumed by a datacenter actually reaches the IT equipment:

$$\text{PUE} = \frac{\text{total datacenter energy (W)}}{\text{IT equipment energy (W)}}$$

A PUE of 1.0 would be perfect (all energy going to compute). A PUE of 2.0 means that for every watt the GPU consumes, another watt is consumed in cooling, electrical distribution, lighting and conversion losses. PUE turns board power, the figure DCGM or NVML measure, into the total datacenter power drawn from the grid.

Current values

The Uptime Institute Global Data Center Survey 2025 (15th annual edition) reports:

Segment	Typical PUE	Source
Global average	1.54 (6th consecutive year unchanged)	Uptime Institute 2025
Hyperscale (Google, Meta, Microsoft, Amazon)	1.10–1.15	Uptime Institute 2025
Colocation / enterprise	1.58–1.80	Uptime Institute 2025
Unoptimised on-premise	1.6–2.0	industry, various
Theoretical limit (cold air, favourable climate)	1.10–1.20	ASHRAE / industry

The stagnation of average PUE at 1.54 reflects the fact that legacy facilities, with inefficient cooling, keep operating alongside the new ones. For a typical on-premise platform with no specific investment in efficiency, assuming a PUE between 1.4 and 1.6 is reasonable. Facilities with direct liquid cooling can reach 1.10–1.20.

How it inflates J/token

If the reference node (4×H100 SXM) consumes 3,200 W at full load (GPU + CPU + DRAM + fans):

PUE	Total datacenter power	Inflation factor over board
1.10	3,520 W	+10 %
1.40	4,480 W	+40 %
1.54 (global average)	4,928 W	+54 %
1.80 (enterprise legacy)	5,760 W	+80 %

The energy per token on a leaderboard, measured at GPU or node level, does not include PUE. The impact: going from PUE 1.10 (hyperscale) to PUE 1.54 (global average) multiplies effective energy consumption by 1.40×. It is the same GPU, the same model, the same throughput; the difference comes from the facility.

Grid carbon intensity

What it measures and why it varies

The carbon intensity of the electricity grid expresses how many grams of CO₂eq are emitted per kilowatt-hour generated, as a function of the source mix (nuclear, wind, solar, gas, coal, hydro). It is the most heterogeneous variable in the calculation: it varies by country, by region within a country, and by hour of the day.

The reference OSS tools are:

Tool	Signal type	Coverage	Access
ElectricityMaps	Hourly average (production-based and flow-traced)	60+ countries in real time	API (limited free plan); OSS on GitHub
WattTime	Hourly marginal (MOER)	US + growing global coverage	API; free data since 2025 (REsurety-WattTime partnership)
Carbon Aware SDK (Green Software Foundation)	Wrapper over ElectricityMaps and WattTime	Same sources	OSS, Apache 2.0; github.com/Green-Software-Foundation/carbon-aware-sdk

ElectricityMaps (formerly electricityMap) publishes carbon intensity in gCO₂eq/kWh hour by hour via API and on its interactive map (app.electricitymaps.com). WattTime publishes the marginal operating emissions rate (MOER), useful for calculating the causal impact of adding or removing load. The GSF Carbon Aware SDK integrates both signals as interchangeable backends.

Country data (annual averages 2024–2025)

Country	Average intensity 2024 (gCO₂eq/kWh)	Approximate hourly range	Source
Spain	~108	30–250	Nowtricity / REE 2025
France	~21.7 (2024); ~19.6 (2025)	5–80	RTE Bilan Électrique 2025
Germany	~363 (2024); ~328 (2025)	100–600	Fraunhofer ISE 2025
EU average	~213	—	Ember European Electricity Review 2025

France has the lowest intensity in continental Europe thanks to its nuclear fleet (>90 % low-carbon generation in 2025, according to RTE). Spain closed 2024 with a historic record: 56.8 % renewable generation, with an average intensity of 108 gCO₂/kWh, among the lowest in the non-nuclear EU. Germany, in accelerated transition but still dependent on lignite and gas, has three times Spain’s average intensity.

Hourly variation: the figure that matters most for scheduling

Within a single country, the hourly variation in carbon intensity is enormous. Data from the ElectricityMaps 2025 grid review for Spain:

Time of day	Typical condition	Estimated intensity (gCO₂eq/kWh)
Solar midday (11:00–15:00, summer)	Solar covers >40 % of demand	30–80
Night-time trough (02:00–06:00, summer)	Zero solar; variable wind	80–180
Demand peak (19:00–21:00, winter)	Gas as the marginal plant	180–280
Cold windless small hours (winter)	Gas + combined cycle dominate	220–300

The price-carbon correlation in Spain is strong and negative (coefficient −0.70 in 2025 according to ElectricityMaps): cheap hours are also the cleanest, because both are driven by renewable penetration. The 477 hours with negative prices in 2025 were also the hours of lowest carbon intensity.

Spain vs France vs Germany: impact on gCO₂/token

To anchor the difference in operational terms, with the reference node (4×H100 SXM, indicative throughput 1,000 tok/s under load, PUE 1.45):

Energy per token at the datacenter:

$$E_{\text{DC}} = \frac{3{,}200\ \text{W} \times 1{.}45}{1{,}000\ \text{tok/s}} = 4{.}64\ \text{J/tok} = 1{.}29 \times 10^{-3}\ \text{Wh/tok}$$

Country / hour	Intensity (gCO₂/kWh)	gCO₂ per 1,000 tokens	gCO₂ per 1M tokens
France (2025 average)	19.6	0.025	25.3
Spain solar (summer midday)	~50	0.064	64.5
Spain 2024 average	~108	0.140	139.3
Spain night-time gas (winter)	~220	0.284	283.8
Germany 2025 average	~328	0.423	423.0
Germany coal peak	~550	0.709	709.0

The range is ×28 between the French average and the German peak. Between the two hourly extremes in Spain (solar vs winter night-time gas) the difference is ×4.4. The choice of country and of execution hour moves carbon per token more than any hardware or engine optimisation within that range.

Scope 2 and Scope 3: what counts and what does not

GHG Protocol scopes

The GHG Protocol defines three scopes for corporate emissions:

Scope	Definition	Datacenter example
Scope 1	Direct emissions from owned or controlled sources	Emergency diesel generators
Scope 2	Indirect emissions from purchased electricity consumption	Electricity consumption of the cluster
Scope 3	Other indirect emissions in the value chain	Hardware manufacturing (embodied carbon)

For an on-premise inference platform, Scope 2 is the dominant item in operation; upstream Scope 3 (manufacturing of servers and GPUs) is typically the second largest item over the hardware life cycle.

Scope 2: location-based vs market-based

The GHG Protocol Scope 2 Guidance (2015, under public revision 2025–2027) establishes two accounting methods, both mandatory in corporate reporting:

Method	What it measures	How it is obtained
Location-based	Actual intensity of the electricity grid where the facility sits	Emission factor of the national or regional grid (published by operators such as REE, RTE)
Market-based	Intensity according to the contractual instruments for energy purchase	Guarantees of Origin (GoO), PPAs or tariffs with a renewable attribute

The revision under public consultation (October 2025) proposes requiring, for the first time, hourly matching for the market-based method: buying 100 MWh of solar over a year is not enough if it is consumed in hours without sun. The aim is for market-based to reflect physical electricity and not just accounting certificates.

For an inference cluster, the location-based method with hourly intensity (not an annual average) is the most informative for carbon-aware operation: it shows in real time what emissions each token generates.

Marginal vs average emissions

This distinction, documented by ElectricityMaps and WattTime, is fundamental for evaluating the real impact of scheduling decisions:

Type	Definition	When to use
Average	The share of all grid emissions attributable to the consumer according to their quota	Location-based Scope 2 carbon reporting
Marginal (MOER)	Emissions of the generator that would respond to a marginal increase in load	Evaluating the causal impact of adding or moving load

Textbook example from ElectricityMaps: in a grid with 50 % wind (0 gCO₂/kWh) and 50 % gas (500 gCO₂/kWh), the average intensity is 250 gCO₂/kWh, but the marginal factor is ~500 gCO₂/kWh (gas is the plant that responds to the demand increase). Reducing load avoids 500 gCO₂/kWh, even though the Scope 2 report records only 250 gCO₂/kWh.

Marginal and average must never be mixed in the same accounting: they represent different attribution paradigms (the GHG Protocol and ElectricityMaps both say so explicitly).

Scope 3: the embodied carbon of the hardware

Embodied carbon is the CO₂eq footprint generated during manufacturing, transport and end of life of the hardware. For GPUs and high-performance servers, this item is significant:

Item	Indicative value	Source
Service life of a datacenter server	3–5 years	AWS / industry
Service life of the datacenter facility	15–20 years	industry
Amortisation methodology	Embodied carbon / years of service life	GHG Protocol / AWS methodology 2025
Embodied carbon of an AI server (indicative)	1,000–3,000 kgCO₂eq (cradle-to-gate)	AWS Embodied Carbon methodology

Embodied carbon is amortised over the service life of the hardware: a server with 1,500 kgCO₂eq of embodied carbon and a 4-year service life contributes 375 kgCO₂eq/year of Scope 3, regardless of how much it is used. A high utilisation rate reduces the embodied carbon per token (the same hardware produces more functional value).

None of the public LLM energy leaderboards (HF AI Energy Score, ML.ENERGY, MLPerf Power) include Scope 3 in their metrics. The SCI specification incorporates it explicitly.

The full conversion: from J/token to gCO₂/token

The equation

$$\text{gCO}_{2}\text{/token} = \frac{E_{\text{board}}\ [\text{Wh/tok}] \times \text{PUE} \times I_{\text{grid}}\ [\text{gCO}_{2}\text{/kWh}]}{1{,}000}$$

where:

$E_{\text{board}}$ is the energy per token measured at GPU/node level (without PUE)
$\text{PUE}$ multiplies by the datacenter overhead
$I_{\text{grid}}$ is the grid carbon intensity at the moment of execution
Dividing by 1,000 converts Wh to kWh

For the reference node (4×H100 SXM, throughput 1,000 tok/s, GPU+CPU+RAM consumption 3,200 W, PUE 1.45):

$$E_{\text{board}} = \frac{3{,}200\ \text{W}}{1{,}000\ \text{tok/s}} = 3{.}2\ \text{J/tok} = 8{.}9 \times 10^{-4}\ \text{kWh/tok}$$

Then, applying PUE 1.45:

$$E_{\text{DC}} = 8{.}9 \times 10^{-4} \times 1{.}45 = 1{.}29 \times 10^{-3}\ \text{kWh/tok}$$

And the emissions:

$$\text{gCO}_{2}/\text{tok} = 1{.}29 \times 10^{-3} \times I_{\text{grid}}$$

Full table by region and hour

Scenario	PUE	Grid intensity (gCO₂/kWh)	gCO₂/1,000 tok	gCO₂/1M tok
France 2025 average	1.45	19.6	0.025	25.3
Spain solar (summer midday)	1.45	50	0.064	64.5
Spain 2024 average	1.45	108	0.139	139.3
Spain night-time gas (winter)	1.45	220	0.284	283.8
Germany 2025 average	1.45	328	0.423	423.0
Germany coal peak	1.45	550	0.709	709.0
Spain solar + hyperscale PUE	1.12	50	0.050	50.0
Spain night-time gas + legacy PUE	1.80	220	0.354	354.0

The PUE factor matters more in high-intensity scenarios: going from PUE 1.12 to 1.80 at the German coal peak moves the result from ~617 to ~860 gCO₂/1M tok (+39 %). In solar Spain, the same PUE change moves it from 50 to 79 gCO₂/1M tok; the low grid intensity cushions the impact of poor PUE.

Carbon-aware shifting: temporal and spatial

Principle and tools

Carbon-aware computing means running flexible load (batch, training, fine-tuning, document ingestion) when and where the grid carbon intensity is lower. The reference OSS tool is the Carbon Aware SDK (Green Software Foundation, Apache 2.0): a wrapper that queries ElectricityMaps or WattTime and returns the current and forecast intensity for a zone and time window.

The SDK distinguishes two types of shifting:

Type	Description	Applicable to
Temporal	Delaying or advancing execution within the same datacenter to a lower-carbon hour	Batch, training, deferrable tasks with a loose deadline
Spatial	Moving the load to a geographic region with lower intensity at that moment	Multi-cloud or multi-datacenter; requires model/data replication

For an on-premise platform in a single datacenter, only temporal shifting is immediately applicable. Spatial shifting requires multi-site infrastructure.

Quantifying the temporal saving

Recent research (arXiv 2512.07799, 2512.08725; Microsoft Carbon-Aware Computing Whitepaper) quantifies the saving from temporal shifting of batch load:

Study / case	Emissions saving	Conditions
Batch load with 24 h shifting (simulation, several grids)	20–40 % emissions reduction	24 h window, grid with a renewable-fossil mix
ML training with shifting against grid forecast (Microsoft)	~30 %	Azure, shifting against forecast intensity
Greedy heuristics vs optimum (arXiv 2512.07799)	≥ 90 % of the optimum	Simple one-migration or greedy deferral
UBS + Microsoft, Azure Batch, 24 h window	Validated with the Carbon Aware SDK	Real temporal shifting in production

For Spain, where the price-carbon correlation is −0.70, shifting to solar hours cuts both electricity cost and emissions per token at the same time. In the 477 hours with negative prices recorded in 2025 (ElectricityMaps / REE), carbon intensity was also at a minimum. The coincidence between clean hour and cheap hour means that carbon-aware scheduling has no economic opportunity cost in Spain: optimising carbon and cost point to the same time window.

Quantification for the reference node

For a 1-hour batch task on the reference node (4×H100 SXM, 4,640 W total with PUE 1.45), generating ~3,600 million tokens per hour:

Scenario	Intensity (gCO₂/kWh)	Emissions per batch hour	Saving vs non-deferred execution
No shifting (Spain average)	108	501 gCO₂	—
Shifted to a solar hour (50 gCO₂/kWh)	50	232 gCO₂	−54 %
Shifted to a night-time gas hour (220 gCO₂/kWh)	220	1,021 gCO₂	+104 % (worst moment)

The range between the worst and the best moment to run that batch is ×4.4 in emissions. Shifting to the cleanest hour available in a 24-hour window in Spain typically avoids between 40 % and 60 % of the emissions of deferrable work.

The SCI standard: Software Carbon Intensity

Specification

Software Carbon Intensity (SCI) is a Green Software Foundation specification, published as standard ISO/IEC 21031:2022 and maintained on GitHub (github.com/Green-Software-Foundation/sci). It defines a rate of carbon emissions per functional unit of software:

$$\text{SCI} = \frac{(E \times I) + M}{R}$$

where:

Symbol	Meaning	Units
$E$	Energy consumed by the software system (servers, network, user devices)	kWh
$I$	Carbon intensity of the energy consumed (location-based or another declared basis)	gCO₂eq/kWh
$M$	Embodied carbon of the hardware, amortised by time of use and utilisation percentage	gCO₂eq
$R$	Functional unit (the denominator that normalises the rate)	per API call, per token, per user, etc.

For an LLM inference platform, $R$ is naturally the output token or the query. SCI in gCO₂eq/token therefore includes the three components that most current analyses omit: PUE implicit in $E$, hourly intensity in $I$, and embodied carbon in $M$.

Key difference from the simple calculation

The calculation $E \times I$ (energy × grid intensity) is operational Scope 2. SCI adds $M$, the Scope 3 of the hardware, which has a different structure:

$$M = \frac{\text{total embodied carbon (gCO2eq)}}{\text{service life (hours)}} \times \frac{\text{hours of use}}{1} \times \frac{\text{allocated resources}}{\text{total server resources}}$$

For an AI server with 1,500 kgCO₂eq of embodied carbon, a 4-year service life (~35,040 hours) and 80 % utilisation:

$$M_{\text{hour}} = \frac{1{,}500{,}000\ \text{gCO2eq}}{35{,}040\ \text{h}} \times 0{.}80 \approx 34\ \text{gCO2eq/h}$$

With 3,600 million tokens per hour:

$$M_{\text{per token}} \approx \frac{34}{3{.}6 \times 10^{9}} \approx 9{.}4 \times 10^{-9}\ \text{gCO2eq/tok}$$

At this scale, embodied carbon per token is negligible against operational Scope 2 during operation. However, $M$ becomes dominant in two cases: underutilised hardware (low utilisation amplifies the cost per token) and hardware with a very short life cycle (chips replaced every 2 years instead of 4).

SCI as the reference metric for the track

The SCI specification and its AI extension (github.com/Green-Software-Foundation/sci-ai) are being adopted as the reference metric for reporting the carbon footprint of AI systems in companies subject to CSRD. It provides a common denominator, gCO₂eq per functional unit, that allows architectures, regions and scheduling strategies to be compared on the same basis.

Limits of the calculation, and honesty

No gCO₂/token calculation captures the full reality. The limits of the model set out in this article:

Limit	Description
Annual average vs hourly intensity	Using the country’s annual average as the emission factor gives a stable but fictitious result. Intensity varies ×4–10× within a day. For location-based Scope 2 reporting with hourly data, the difference can be 30–50 % relative to the average.
Static PUE	PUE varies with outdoor temperature (cooling works harder in summer) and with IT load. The declared PUE is an average; on hot days it can be 10–20 % worse.
Constant throughput	The calculation uses throughput at full load. In production, with variable demand, consumption does not scale linearly with throughput; efficiency per token gets worse at low load.
Partial Scope 3	Embodied carbon here covers only the IT hardware. It does not include the construction of the datacenter building, the manufacture of cooling equipment, or the energy supply chain.
National emission factors	The per-country values are annual averages. France has low intraday variation (nuclear dominates); Spain and Germany have high variation. The national factor is useful for order-of-magnitude comparisons, not for operational scheduling.
Embodied carbon of GPUs	Lifecycle assessment data for AI chips such as the H100 are not public. Estimates rely on die-area models, which carry ±30–50 % uncertainty.
Spatial shifting omitted	This article covers temporal shifting only. Spatial shifting (moving load between regions) can add another 20–40 % of reduction but requires multi-site infrastructure.
Cooling water and other impacts	gCO₂eq does not capture cooling water consumption (WUE, Water Usage Effectiveness), another environmental impact indicator relevant to arid zones such as southern Spain.

Sources

Uptime Institute · Global Data Center Survey 2025 (PUE medio global 1,54, 15.ª edición) — https://uptimeinstitute.com/resources/research-and-reports/uptime-institute-global-data-center-survey-results-2025
Uptime Institute Blog · Global PUEs — are they going anywhere? — https://journal.uptimeinstitute.com/global-pues-are-they-going-anywhere/
Uptime Institute Blog · Large data centers are mostly more efficient — https://journal.uptimeinstitute.com/large-data-centers-are-mostly-more-efficient-analysis-confirms/
ElectricityMaps · Electricity Grid Review 2025: Spain (precio, intensidad, correlación precio-carbono −0,70) — https://www.electricitymaps.com/grid-in-review-2025/spain
ElectricityMaps · Marginal emissions — what they are, and when to use them (actualizado enero 2025) — https://www.electricitymaps.com/resources/publications/marginal-emissions-introduction
ElectricityMaps · Interactive map and API — https://app.electricitymaps.com
WattTime · Average vs marginal emissions — https://watttime.org/data-science/data-signals/average-vs-marginal/
WattTime · REsurety-WattTime free global marginal emissions platform (marzo 2025) — https://watttime.org/news-and-insights/resurety-and-watttime-announce-release-of-free-electricity-marginal-carbon-platform/
Red Eléctrica de España (REE) · Renewable energies generated 56 % of Spain’s electricity mix in 2024 — https://www.ree.es/en/press-office/press-release/news/press-release/2025/01/renewable-energies-generated-56-per-cent-spains-electricity-mix-2024
Nowtricity · CO₂ emissions per kWh in Spain (media 2024: 108 gCO₂eq/kWh) — https://www.nowtricity.com/country/spain/
RTE · Bilan électrique 2025 — GHG Emissions (Francia 2025: 19,6 gCO₂eq/kWh) — https://analysesetdonnees.rte-france.com/en/annual-review-2025/ghg-emissions
RTE · Bilan électrique 2024 key findings (Francia 2024: 21,7 gCO₂eq/kWh) — https://assets.rte-france.com/analyse-et-donnees/2025-11/RTE%20-%20Annual%20electricity%20review%202024%20-%20Key%20findings.pdf
Fraunhofer ISE · German Public Electricity Generation in 2025 (Alemania 2025: ~328 gCO₂/kWh) — https://www.ise.fraunhofer.de/en/press-media/press-releases/2026/german-public-electricity-generation-in-2025-wind-and-solar-power-take-the-lead.html
Ember · European Electricity Review 2025 (intensidad UE media: 213 gCO₂/kWh) — https://ember-energy.org/latest-insights/european-electricity-review-2025/five-years-of-progress/
GHG Protocol · Scope 2 Standard Advances (consulta pública 2025, hourly matching, market-based revisions) — https://ghgprotocol.org/blog/scope-2-standard-advances-isb-approves-consultation-market-and-location-based-revisions
GHG Protocol · Public Consultation Scope 2 (octubre 2025) — https://ghgprotocol.org/sites/default/files/2025-10/GHG-Protocol-Scope2-Public-Consultation.pdf
Green Software Foundation · Software Carbon Intensity (SCI) Specification — https://sci.greensoftware.foundation/
Green Software Foundation · SCI GitHub (ISO/IEC 21031:2022) — https://github.com/Green-Software-Foundation/sci
Green Software Foundation · SCI for AI — https://github.com/Green-Software-Foundation/sci-ai
Green Software Foundation · Carbon Aware SDK (Apache 2.0) — https://github.com/Green-Software-Foundation/carbon-aware-sdk
Green Software Foundation · Carbon-Aware Computing Whitepaper — UBS/Microsoft Azure Batch case — https://greensoftware.foundation/articles/carbon-aware-computing-whitepaper-how-ubs-succeeded-in-measuring-and-reducing-car/
arXiv 2512.07799 · Quantifying the Carbon Reduction of DAG Workloads: A Job Shop Scheduling Perspective — https://arxiv.org/abs/2512.07799
arXiv 2512.08725 · Spatio-Temporal Shifting to Reduce Carbon, Water, and Land-Use Footprints of Cloud Workloads — https://arxiv.org/abs/2512.08725
AWS · How AWS estimates embodied emissions of IT hardware (Scope 3 methodology, 2025) — https://aws.amazon.com/blogs/infrastructure-sustainability/how-aws-estimates-embodied-emissions-of-it-hardware-the-science-and-technology-behind-the-latest-customer-carbon-footprint-methodology/
CodeCarbon · Track and reduce CO₂ emissions from your local computing — https://codecarbon.io/

Measurement bias and reproducibility: why two benchmarks of the same model give figures that differ by up to 7×

Tue, 16 Jun 2026 07:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The mathematical symbol is reserved for formulas: amounts are expressed in € or USD. Thousands with a thin space ($1\,000$).

TL;DR

Running the same Llama-3-70B-Instruct model on the same 4×H100 SXM 80 GB node with two different tools can produce throughput results that differ by 7.2× without the engine changing a single line of code. The cause is not the engine: it is the measurement method. At 1,000 QPS, a single-process asyncio client (vLLM bench, SGLang bench, genai-perf pre-AIPerf) processed 75,574 tokens against the 545,733 tokens of a multi-process client, both measuring the same endpoint (arXiv 2605.24217). The bias is not random: it is systematic, directional and reproducible. This article catalogues the sources of bias, quantifies their magnitude and describes the minimum harness that turns a number into an auditable datum.

The sources of bias: a catalogue with magnitudes

Each source of bias moves the published number in a given direction and with a characteristic magnitude. The table below orders the sources from largest to smallest impact observed in the literature:

Source of bias	Direction of the bias	Documented magnitude	Reference
Single-process client saturation (Python GIL, asyncio)	understates throughput; overstates latency	up to 7.2× fewer tokens processed at 1,000 QPS	arXiv 2605.24217
Tokenizer different from the model’s (LLMPerf uses a universal LlamaTokenizer)	tok/s not comparable across different vocabularies	variable; up to 33 % of client overhead in the TPS denominator	NVIDIA blog (TPS = client overhead ÷ total duration)
ignore_eos missing or inconsistent OSL	underestimates throughput; artificially shortens the real OSL	up to premature termination; an unrealistic fixed-length benchmark	vLLM bench docs; NVIDIA fundamental concepts
Unrealistic ISL/OSL distribution (fixed length instead of a realistic distribution)	an optimistic knee that does not resemble production	the knee shifts; prefill understated with short prompts	AIPerf sequence-length-distributions docs
Missing warmup / warm prefix cache	overstates TTFT (artificially low on the first requests)	TTFT underestimated by an already populated KV cache	vLLM prefix caching docs; arXiv 2605.24217
`concurrency` vs `request-rate` as the load mode	concurrency saturates symmetrically; request-rate can build a queue	ITL overestimated if the queue grows without a ceiling with request-rate	NVIDIA fundamentals; AIPerf docs
Measurement window too short	does not capture the steady state; includes ramp-up	throughput overestimated; latency underestimated	meta-metrics arXiv 2508.10251
Thermal state and GPU clocks	thermal throttling reduces throughput without prior stabilisation	temperature > 75 °C triggers throttling; a difference of ~3 W of stable power	arXiv 2604.09048
Warm prefix cache between runs	overstates TTFT; unrealistic TTFT if the cache is pre-populated from the previous run	a severe effect in multi-prompt evaluations with shared prefixes	arXiv 2605.24217
Mean instead of percentiles	hides the latency tail	P99 can be 4–6× the median at high concurrency	NVIDIA fundamentals; Anyscale docs

The dominant bias: single-process client saturation

The largest bias is not in the engine but in the client that generates the load. The single-process tools (vLLM bench, SGLang bench, genai-perf before its replacement by AIPerf on 15 April 2026) use a single Python process with asyncio to manage the concurrent requests. The Python Global Interpreter Lock (GIL) prevents a single process from using more than one CPU core at a time, which introduces a bottleneck on the client side that becomes critical at high concurrency.

The mathematical effect: as the request rate rises, the client fails to dispatch requests at the configured pace, accumulates queueing time on the client side, and that time gets recorded as engine latency. The result is that TTFT and TPOT appear inflated and throughput appears depressed, without the engine having changed anything (arXiv 2605.24217).

The documented discrepancy at 1,000 QPS:

Client architecture	Tokens processed at 1,000 QPS	Ratio
Single-process (asyncio, Python GIL)	75,574	1× (baseline)
Multi-process (distributed load)	545,733	7.2×

Note: both clients point at the same endpoint; the difference is entirely attributable to the benchmark client, not to the engine.

Tokenizer bias: tok/s are not comparable across vocabularies

Tokens are not universal. Each model has its own tokenizer with its own vocabulary. Llama-3 uses a vocabulary of 128,256 tokens; Gemma uses 256,128; earlier models used 32,000–50,000. The same English text produces a different number of tokens depending on the model’s tokenizer.

The direct consequence for benchmarking: if the tool measures tok/s with a tokenizer different from that of the model being served, the token count, and therefore the throughput in tok/s and the cost per token, are biased. LLMPerf (archived in December 2025) used LlamaTokenizer universally for all models, which guaranteed internal consistency in the leaderboard but made the tok/s not comparable with measurements of other models with different vocabularies.

The TPS formula in LLMPerf additionally included the benchmark’s full denominator, the time spent generating prompts, preparing requests and storing responses, which NVIDIA estimated at up to 33 % of the total duration at concurrency 1. This makes LLMPerf’s TPS systematically lower than that of GenAI-Perf/AIPerf for the same system, without the engine being worse:

$$\text{TPS}_{\text{LLMPerf}} = \frac{\text{output tokens}}{T_{\text{end}} - T_{\text{start}}}$$ $$\text{TPS}_{\text{GenAI-Perf}} = \frac{\text{output tokens}}{T_y - T_x}$$

Where $T_{\text{start}}$ and $T_{\text{end}}$ include the client overheads, while $T_x$ and $T_y$ are the instant of the first request and that of the last token received, respectively (NVIDIA · Fundamental Concepts).

The operational rule: always count tokens with the tokenizer of the model being served, not with a proxy tokenizer.

The bias of ignore_eos and an inconsistent OSL

Most LLM models generate a special end-of-sequence token (EOS) when they consider the response complete. If the benchmark does not set ignore_eos=True, the real output length (OSL) varies from request to request according to the model’s natural length distribution, which produces:

Inconsistent OSL: two runs with the same seed can produce different OSL distributions if the model varies its natural output.
Spurious comparison: a “faster” model may be so simply because it generates shorter responses (it hits EOS sooner), not because it has more real throughput.
Underestimated throughput: if the benchmark expects OSL=256 but the model stops at OSL=80 on average, the throughput in tok/s appears higher but measures less work.

The ignore_eos parameter (or --ignore-eos in vLLM bench) instructs the engine to ignore the EOS token and continue until reaching max_tokens. It is mandatory for the OSL to be the configured one rather than the model’s natural one, and for two runs to be comparable (vLLM benchmark CLI docs).

The bias of the ISL/OSL distribution

The distribution of input (ISL) and output (OSL) lengths determines what proportion of the compute time goes to prefill (costly in TTFT) and how much to decode (costly in ITL). A benchmark with a fixed length, say ISL=128, OSL=128, produces results that do not resemble any real traffic.

Real use cases have very different distributions:

Use case	Typical ISL (tokens)	Typical OSL (tokens)	Dominated by
Translation	500–2,000	500–2,000	balanced
Generation (code, email)	~100	~1,000	decode (long OSL)
Summarisation / RAG	~1,000	~100	prefill (long ISL)
Reasoning (CoT)	~100	1,000–10,000	very long decode

A short-ISL benchmark with a model optimised for prefill will give an artificially low TTFT and an artificially high throughput. The knee of the sweep shifts: with ISL=64 the system admits more concurrency without breaking the TTFT SLO; with ISL=1,024 prefill saturates earlier and the knee appears earlier. Using the wrong distribution means sizing for traffic that does not exist.

AIPerf introduces sequence distributions with configurable per-component variance to reproduce realistic traffic mixes (AIPerf · Sequence Length Distributions):

--sequence-distribution "64|10,32|8:70;256|40,128|20:20;1024|100,512|50:10"

This creates 70 % of requests with $\text{ISL} \sim \mathcal{N}(64, 10)$ and $\text{OSL} \sim \mathcal{N}(32, 8)$, 20 % with medium ISL/OSL, and 10 % with long ISL/OSL, far more faithful to real chatbot traffic than a fixed length.

The bias of warmup and the prefix cache

Two related but distinct sources of bias:

Missing warmup. The first requests of a benchmark hit a “cold” engine: the GPU is in a low-clock state, the KV cache is empty, and the operating system may be paging memory. The TTFT of the first requests is structurally higher than that of the steady state. If the benchmark does not discard a warmup period, the mean TTFT includes these outliers and overestimates the real production latency. Some frameworks (GenAI-Perf/AIPerf) use a sliding window that excludes the ramp-up and ramp-down requests; others do not.

Warm prefix cache between runs. The prefix KV cache (prefix cache or prompt cache) stores the computed results of repeated prompt prefixes. If the benchmark runs multiple consecutive runs with the same prompts, the second and subsequent runs find the KV cache already populated and report an artificially low TTFT, that of the decode, not of the prefill. For a baseline benchmark, the prefix cache must be cold; for one that simulates production with repeated prompts, warm. The distinction must be made explicit (arXiv 2605.24217; vLLM · Automatic Prefix Caching).

The bias of `concurrency` vs `request-rate`

The two load control modes produce different latency distributions for the same system:

Mode	Semantics	When to use	Risk
concurrency N	keeps exactly N requests in flight; as soon as one finishes, it launches another	measuring the system under a fixed concurrency load	over-represents sustained load; does not simulate real arrivals
request-rate r (constant or Poisson)	launches one request every $1/r$ seconds (constant) or with interarrival $\sim \text{Exp}(1/r)$ (Poisson)	simulating real traffic (random arrivals)	if the engine cannot absorb r req/s, the queue grows without a ceiling

NVIDIA recommends the concurrency mode for most capacity benchmarks (NVIDIA · LLM Benchmarking Fundamental Concepts). The request-rate mode is more faithful for online traffic (a Poisson distribution of arrivals), but if the rate exceeds the engine’s capacity the queue grows indefinitely and the latency metrics include queueing time that can dominate the TTFT, mixing queue behaviour with engine behaviour.

A system measured at concurrency=16 and another at request-rate=16 req/s are not under the same load: fixed concurrency guarantees 16 simultaneous requests; the rate sets the arrival pace but not the instantaneous concurrency. Comparing their results without adjustment is incorrect.

The bias of the measurement window and the thermal state

Window too short. A 30-second benchmark may measure the system’s ramp-up, not the steady state. The recommendation of arXiv 2508.10251 is that the measurement window cover at least 3-5× the system’s ramp time under load, and that the metrics be computed only over the steady window, excluding warmup and cooldown.

Thermal state of the GPU. Datacenter GPUs (H100 SXM, A100) operate with thermal throttling above ~75 °C. If the GPU has not reached its steady-state temperature before the benchmark, the first measurements correspond to a higher-clock state than the sustainable one. Controlled experiments document that power must stabilise within a range of 3 W for at least 30 seconds before the measurements are representative (arXiv 2604.09048 · Watt Counts). The effect is especially severe in prefill benchmarks (TTFT), where the high clocks of the cold state produce an artificially low TTFT.

For a reproducible benchmark on 4×H100 SXM 80 GB: before measuring, run sustained load for at least 2–3 minutes until nvidia-smi reports a stable temperature and stable power (variation < 3 W over 30 s). Only then start the measurement window.

How the tools compute ITL differently

ITL (Inter-Token Latency) appears in every tool but its formula varies, and the differences are not small:

GenAI-Perf / AIPerf:

$$\text{ITL} = \frac{\text{e2e latency} - \text{TTFT}}{\text{output tokens} - 1}$$

TTFT is excluded from the numerator and the denominator discounts the first token. ITL is a pure decode metric, with no prefill contamination.

LLMPerf (archived Dec. 2025):

$$\text{ITL}_{\text{LLMPerf}} = \frac{\text{e2e latency}}{\text{output tokens}}$$

TTFT is included in the numerator. For short sequences (OSL < 50 tokens), TTFT can represent 50–80 % of the e2e_latency, so that LLMPerf’s ITL measures mainly prefill, not decode. Two systems with the same decode but different prefill will show different ITLs in LLMPerf even if they are identical in decode speed.

vLLM bench (benchmark_serving.py):

It computes ITL as the mean of the intervals between consecutive tokens of the output stream, including intra-request variance. It can reveal decode jitter that the others average away.

The practical consequence: never cross a GenAI-Perf ITL with an LLMPerf one as if they were the same metric. They are different formulas over the same signal.

Comparability across tools: why their numbers do not cross

The following table summarises the methodological differences that make one tool’s numbers structurally incompatible with another’s without an explicit adjustment:

Dimension	vLLM bench	LLMPerf (archived)	GenAI-Perf / AIPerf	GuideLLM
Client architecture	single-process asyncio	single-process	multi-process	multi-process
ITL includes TTFT	no	yes	no	no
TPS denominator	$T_y - T_x$	$T_{\text{end}} - T_{\text{start}}$ (overhead included)	$T_y - T_x$	per round
Tokenizer	that of the model served	universal LlamaTokenizer	that of the model served	that of the model served
Warmup / sliding window	not automatic	no	yes (sliding window)	per round
ignore_eos by default	no	no	explicitly recommended	configurable
ISL/OSL distribution	manual parameter	manual parameter	distributions with variance	configurable `--data`
Primary load mode	concurrency	concurrency (drained batches)	concurrency (recommended)	sweep + poisson
Structured output	basic JSON	JSON	JSON + CSV	JSON + HTML + CSV

An LLMPerf result and a GenAI-Perf result for the same endpoint can differ in TTFT, ITL and TPS simultaneously and in all of them for methodological reasons, not because of engine differences. The only way to cross them is to run both tools on the same system under the same conditions and compute the empirical conversion factor, which in practice amounts to re-measuring with a single tool.

The honest harness: a reproducibility checklist

A benchmark whose number can be defended in an audit has to come accompanied by all the metadata that allow it to be reproduced. The minimum to demand, organised by category:

Hardware

Metadata	Example 4×H100	Effect if not fixed
GPU (model, variant, count)	4× NVIDIA H100 SXM 80 GB	H100 PCIe vs SXM differ in memory BW and NVLink
GPU-GPU interconnect	NVLink 4.0	tensor parallelism depends on the interconnect BW
CPU, RAM, CPU-GPU bandwidth	2× Intel Xeon 8480+, 512 GB DDR5, PCIe 5.0	the tokenisation bottleneck can be on the CPU
Thermal state at the start	GPU temperature < 65 °C, stable power ± 3 W	throttling alters TPS and TTFT by up to ~15 %
GPU clocks	without `nvidia-smi -pm 1` they can vary	a difference of up to ~10 % in throughput

Software and model

Metadata	Example	Effect if not fixed
Engine + version + flags	vLLM 0.8.4, `--tensor-parallel-size 4 --gpu-memory-utilization 0.90`	each version changes the scheduler and the KV cache management
Model + precision	Llama-3-70B-Instruct, FP8	FP16 vs FP8 differ by ~1.4–1.8× in throughput
Tokenizer used for counting	tokenizer of the model served (HF tokenizer)	a universal LlamaTokenizer biases tok/s across vocabularies
Generation seed	`--seed 42`	without a fixed seed, the OSL varies run-to-run
`ignore_eos`	`True`	without it, the OSL varies with the prompt content
Sampling parameters	`temperature=1.0, top_p=0.95`	greedy vs sampling affect logit speed

Load

Metadata	Example	Effect if not fixed
Bench tool + version	AIPerf 0.2.1	each version changes formulas and warmup
ISL/OSL distribution	$\mathcal{N}(512, 64)$ ISL, $\mathcal{N}(128, 20)$ OSL	changing the distribution moves the knee
Load mode	concurrency, sweep 1–64	concurrency vs request-rate: different distributions
Concurrency levels	1, 2, 4, 8, 16, 32, 64 (full sweep, beyond the knee)	without passing the knee, the safe capacity is unknown
Duration per point	120 s minimum per level	short windows capture ramp-up, not the steady state
Warmup	30 s excluded from the metric computation	without warmup, the metrics include the cold state
Prefix cache treatment	cold (flush between runs) or warm (declared explicitly)	warm cache: unrealistic TTFT for a baseline

SLO and reported metrics

Metadata	Example	Effect if not declared
Declared SLO	TTFT P99 < 500 ms, ITL P95 < 50 ms	goodput depends on the SLO; without an SLO there is no goodput
Reported percentiles	P50, P95, P99 (not just the mean)	the mean hides the tail; irrelevant for an SLO
Throughput vs goodput	goodput under the SLO, not raw throughput	catalogue throughput can be 5–10× the goodput
Total number of requests	≥ 1,000 per concurrency level	small samples: high variance in percentiles

Methodological defence of the datum before an audit

A throughput number presented without the data sheet above is not a datum: it is an anecdote. The validation pattern for a technical audit:

1. Traceability of the tool. The tool and its version must be pinnable to a Git commit or a container image hash. AIPerf and GuideLLM export JSON with version metadata; vLLM bench omits them by default and they have to be captured manually.

2. Reproducibility of the environment. The engine deployment script (or the Kubernetes manifest) and the exact benchmark command must be enough for a third party to reproduce the number on the same hardware. GuideLLM exports the benchmark file as an authoritative record of the session: configuration, metadata, per-benchmark statistics and request entries with individual timings (GitHub vllm-project/guidellm).

3. Verification of the steady state. The metrics must come from the benchmark’s steady window (warmup excluded). The throughput vs concurrency curve must show the knee, the point where throughput saturates and latency blows up, and extend beyond it. Without a visible knee, the reported capacity may be the client’s limit, not the engine’s.

4. Goodput, not raw throughput. The auditable throughput is the goodput under the declared SLO, not the peak throughput. A system that reports 4,000 tok/s but whose goodput under TTFT P99 < 500 ms is 1,800 tok/s has a real capacity of 1,800 tok/s for interactive use cases.

5. Internal comparability. If two engines or two configurations are compared, the tool, the load distribution, the SLO, the hardware and the warmup treatment must be identical. Any difference in these dimensions contaminates the comparison.

The goodput formula as an auditable metric:

$$\text{goodput} = \text{throughput} \times \Pr[\text{TTFT} \leq \text{SLO}_{\text{TTFT}}] \times \Pr[\text{ITL} \leq \text{SLO}_{\text{ITL}}]$$

Where the probabilities are estimated over the empirical distribution of the run. An engine with a throughput of 4,000 tok/s and $\Pr[\text{TTFT} \leq 500\,\text{ms}] = 0.45$ has a goodput of $4{,}000 \times 0.45 = 1{,}800$ tok/s.

A numerical example: the same node, four different measurements

Reference hardware: 4×H100 SXM 80 GB, model Llama-3-70B-Instruct FP8, tensor parallel 4, SLO TTFT P99 < 500 ms.

Benchmark configuration	Reported throughput	Real goodput	Factor vs goodput
vLLM bench, concurrency=32, no ignore_eos, no warmup, fixed OSL=64	5,800 tok/s	~1,200 tok/s (short OSL inflates TPS)	4.8×
LLMPerf, concurrency=32, LlamaTokenizer, overhead included in the denominator	3,100 tok/s	~1,600 tok/s (ITL includes TTFT)	1.9×
AIPerf, concurrency=32, ignore_eos, sliding window, model’s tokenizer	3,400 tok/s	3,350 tok/s	1.01×
GuideLLM, sweep 1–64, poisson, realistic ISL/OSL distribution, declared SLO	3,330 tok/s goodput (knee at round 6)	3,330 tok/s	1.0× (honest baseline)

The range: from 1,200 tok/s to 5,800 tok/s for the same engine, the same hardware and the same model. The maximum factor is 4.8×. The cause is not the engine; it is the instrumentation decisions.

Decomposing the total bias

The total observable bias between two tools is the multiplicative composition of the individual biases. For a comparison of single-process vLLM bench vs multi-process GuideLLM on the same system:

$$\text{total factor} \approx \underbrace{f_{\text{client}}}_{\leq 7.2\times} \times \underbrace{f_{\text{tokenizer}}}_{1.0{-}1.3\times} \times \underbrace{f_{\text{ignore-eos}}}_{1.0{-}2.0\times} \times \underbrace{f_{\text{ISL/OSL}}}_{1.0{-}1.5\times} \times \underbrace{f_{\text{warmup}}}_{1.0{-}1.2\times}$$

The client saturation factor ($\leq 7.2\times$) dominates, but the other factors multiply. Under adverse conditions (single-process client + different tokenizer + no ignore_eos + unrealistic OSL + no warmup), the compound bias can exceed 15–20× for the same system.

State of the art 2026: what has changed

genai-perf → AIPerf migration (15 Apr 2026): NVIDIA retired genai-perf and replaced it with AIPerf, which is multi-process with automatic saturation detection, configurable ISL/OSL distributions and a built-in sliding window. The migration removes the single-process client bias from NVIDIA’s official tooling.
LLMPerf archived (Dec. 2025): the ray-project/llmperf repository is archived and in read-only mode. The historical results of the LLMPerf leaderboard are comparable only among themselves; they must not be crossed with modern measurements without adjustment.
GuideLLM 0.5.x (2025–2026): a complete architectural refactor, multimodal support, and authoritative JSON export with all the session metadata. It is the OSS standard for SLO-driven evaluation.
arXiv 2605.24217 (May 2026): the first formal characterisation of systematic measurement bias in production LLM benchmarks, with a mathematical demonstration of the GIL effect and a proposal for a multi-process harness.
arXiv 2508.10251: meta-metrics and good practice for system-level performance benchmarking; it establishes the minimum dimensions of the parameter space that must be declared for a result to be reproducible.
MLPerf Inference v5.1 (Sep. 2025): 27 participants, a participation record. The MLPerf rules require declaring the exact code, the dataset and the hardware, and admit public review; it is the only industry benchmark with a formal review process for reproducibility.

Cross-links from the benchmarking track

The biases described here apply to every tool in the catalogue. For the complete data sheet of each tool:

Full catalogue: LLM benchmarking tools, data sheet by data sheet
GuideLLM and SLO validation: GuideLLM in depth: validating the SLO under load
AIPerf (ex genai-perf) and saturation detection: NVIDIA GenAI-Perf in depth
State of the art of frameworks: LLM benchmarking: frameworks and state of the art
Engine decision on the Pareto axis: Serving engines compared: the Pareto frontier

Sources

arXiv 2605.24217 · Identifying and Mitigating Systemic Measurement Bias in Production LLM Inference Benchmarks — https://arxiv.org/abs/2605.24217
arXiv 2508.10251 · Meta-Metrics and Best Practices for System-Level Inference Performance Benchmarking — https://arxiv.org/pdf/2508.10251
arXiv 2604.09048 · Watt Counts: Energy-Aware Benchmark for Sustainable LLM Inference on Heterogeneous GPU Architectures — https://arxiv.org/html/2604.09048v1
NVIDIA Technical Blog · LLM Inference Benchmarking: Fundamental Concepts (ITL, TPS, ISL/OSL, ignore_eos, concurrency vs request-rate) — https://developer.nvidia.com/blog/llm-benchmarking-fundamental-concepts/
NVIDIA AIPerf Docs · Sequence Length Distributions for Advanced Benchmarking — https://docs.nvidia.com/aiperf/tutorials/datasets-inputs/sequence-length-distributions-for-advanced-benchmarking
NVIDIA AIPerf · Request Rate with Max Concurrency — https://docs.nvidia.com/aiperf/tutorials/load-patterns-scheduling/request-rate-with-max-concurrency
NVIDIA AIPerf · Comprehensive LLM Benchmarking Guide — https://lucaberton.com/blog/nvidia-aiperf-llm-inference-benchmarking-guide/
vLLM Documentation · Benchmark CLI (ignore_eos, métricas disponibles) — https://docs.vllm.ai/en/latest/benchmarking/cli/
vLLM Documentation · Automatic Prefix Caching — https://docs.vllm.ai/en/stable/design/prefix_caching/
GitHub · ray-project/llmperf (archivado dic. 2025) — https://github.com/ray-project/llmperf
GitHub · vllm-project/guidellm (exportación JSON autoritativa, metadatos de sesión) — https://github.com/vllm-project/guidellm
Red Hat Developer · GuideLLM: evaluar despliegues LLM para inferencia real — https://developers.redhat.com/articles/2025/06/20/guidellm-evaluate-llm-deployments-real-world-inference
MLCommons · MLPerf Inference v5.1 results (récord 27 participantes, reproducibilidad formal) — https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/
arXiv 2502.16721 · Speed and Conversational LLMs: Not All Is About Tokens per Second (tokenizer incompatibilidad y tok/s entre vocabularios) — https://arxiv.org/pdf/2502.16721
Medium · LLM Inference Benchmarking (genAI-perf y vLLM, discrepancia 7,2×) — https://kchandan.medium.com/llm-inference-benchmarking-genai-perf-and-vllm-5dd06b57428e

Full TCO of an on-premise GPU cluster: from capex to the all-in €/GPU-hour and the break-even against cloud

Tue, 16 Jun 2026 06:00:00 +0200

Notation: amounts in N € or N USD (source denominated in dollars); decimal point; comma as thousands separator. The dollar sign is not used (it is the formula delimiter). Data centred on Europe/Spain. Generic example hardware: a cluster of N nodes, each with 4×H100 SXM5 80 GB.

TL;DR

A 4×H100 SXM5 node costs between 150,000 USD and 200,000 USD in total capex (GPUs + server + network + storage + prorated rack). Amortised over 3 years with European opex (energy at ~0.116 €/kWh industrial, PUE 1.54 average or 1.2 with liquid, 0.3 FTE of staff), the all-in cost ranges from 3.10 USD/GPU-hour (100 % utilisation) to 6.20 USD/GPU-hour (50 % utilisation). The break-even against AWS p5 on-demand (~6.88 USD/GPU-hour) is crossed at around 70 % utilisation; against a neocloud 3-year reserved (~1.49–2.10 USD/GPU-hour), on-prem never closes the gap in that scenario. Utilisation is the variable that decides the cost axis, not the price of the hardware.

The model: declared assumptions

Every calculation below starts from these assumptions. Changing any of them moves the conclusion; the sensitivity section quantifies by how much.

Parameter	Base value	Sensitivity range
Node	4×H100 SXM5 80 GB (HGX baseboard)	—
Cluster	N nodes (per-node model; scales linearly)	1–32 nodes
Capex amortisation	3 years (straight line)	3–5 years
GPU utilisation	70 %	30 %–100 %
Energy price	0.116 €/kWh (industrial Spain, Sept. 2025)	0.06–0.20 €/kWh
PUE	1.54 (global average, Uptime Institute 2025)	1.15–1.80
Exchange rate	1 USD = 0.93 € (reference Jun. 2026)	—

Energy source: GlobalPetrolPrices · Spain Business Electricity, Sept. 2025. PUE source: Uptime Institute Global Data Center Survey 2025 — the global average PUE has been stuck at 1.54 for the sixth consecutive year; hyperscalers 1.10–1.15; colocation/enterprise 1.58–1.80; facilities less than 5 years old, 1.48. PUE 1.2 is achievable with direct-to-chip liquid cooling.

Capex breakdown per 4×H100 SXM5 node

GPUs

Component	Unit price (USD)	Quantity	Subtotal (USD)	Source and date
H100 SXM5 80 GB (card)	30,000–40,000	4	120,000–160,000	GMI Cloud, Apr. 2026 · Introl, Apr. 2026

The range reflects market variability and volume discounts (5–15 % for orders above 50 units). H100 SXM5 cards require NVIDIA’s HGX baseboard; they are not sold loose for direct installation in standard servers.

Server / HGX baseboard

A complete 4×H100 SXM node uses NVIDIA’s HGX H100 4-GPU baseboard plus a compatible host server. Reference models: Supermicro SYS-421GU-TNXR (4U, dual Intel Xeon 4th Gen, HGX H100 4-GPU) and its Dell equivalent.

Component	Estimated cost (USD)	Note
Server chassis + CPU (2× Xeon) + RAM (512 GB DDR5) + redundant PSU	18,000–25,000	Based on a Supermicro SYS-821GE bare without GPUs at ~24,806 USD (xicomputer.com, Jun. 2026); scaled to the 4-GPU
HGX H100 4-GPU baseboard	included in the GPU price	NVIDIA HGX platform; no separate public price
Inter-GPU NVLink (inside the node)	included in the baseboard	4 GPUs connected by NVLink 4.0 on the HGX baseboard

Marketing claim (no independent verification): Supermicro announces datacenter energy cost reductions of up to 40 % with liquid cooling in its HGX H100 servers (Supermicro press release).

InfiniBand NDR network

For a multi-node cluster with tensor parallelism across nodes, the GPU-to-GPU network is critical. NDR InfiniBand (400 Gb/s per port) is the de facto standard for HGX clusters.

Component	Estimated cost per node (USD)	Source / Note
NVIDIA Quantum-2 NDR 400G switch (64 ports, prorated across N nodes)	2,000–4,000	Switch ~35,000 USD (Introl, Apr. 2026); at 16 nodes, ~2,200 USD/node
InfiniBand NDR cables/transceivers (4 ports per node × ~1,000 USD/port)	4,000	Estimate based on ~1,000 USD per optical transceiver (Introl, Apr. 2026)
InfiniBand network (prorated per 4-GPU node)	~6,000–8,000	—

For inference serving inside a single node (4 GPUs with NVLink), the inter-node network is less critical than for multi-node training. For disaggregated prefill-decode workloads spread across nodes, InfiniBand NDR is necessary.

NVMe storage

Component	Estimated cost (USD)	Note
Local NVMe (4 TB × 2 U.2/E1.S drives, working datasets and checkpoints)	2,000–4,000	~500–1,000 USD/TB enterprise NVMe, 2025
Shared object storage (NAS/MinIO, prorated per node)	2,000–5,000	Varies with the total capacity of the cluster
Total storage per node	~4,000–9,000	—

Introl models 50 TB per GPU for effective operation in training clusters (Introl, Apr. 2026); for pure inference the requirement is significantly lower (model weights + logs).

Rack, PDU and datacenter connectivity

Component	Estimated cost per node (USD/year)	Source
Rack colocation (high density, 10–15 kW per node)	5,000–12,000/year	Encoradvisors · Colocation Pricing 2026: high density 3,000–6,000 USD/month per rack; at 2 nodes per rack, ~1,500–3,000 USD/month per node = 18,000–36,000 USD/year in tier-1; lower in Spain
Rack PDU, electrical cabling (prorated)	500–1,000 per node (amortised capex)	Inside the colocation line item or in an own datacenter

Colocation in Spain/Europe is structurally cheaper than in US tier-1 markets (New York, Silicon Valley). For an own datacenter, replace it with the cost of your own space plus amortisation of the electrical and cooling infrastructure.

Capex summary per 4×H100 SXM5 node

Line item	Range (USD)	Midpoint
GPUs (4× H100 SXM5)	120,000–160,000	140,000
Server chassis + CPU/RAM/PSU	18,000–25,000	21,500
InfiniBand NDR network (prorated)	6,000–8,000	7,000
NVMe + object storage	4,000–9,000	6,500
PDU/rack/other (capex)	2,000–5,000	3,500
Total capex per node	150,000–207,000	178,500

Sources: GMI Cloud (Apr. 2026), Introl (Apr. 2026), Spheron (Apr. 2026), xicomputer.com (Jun. 2026).

Opex breakdown per 4×H100 SXM5 node (annual)

Energy

A 4×H100 SXM5 node at full load draws approximately:

$$P_{\text{node}} = 4 \times 700\,\text{W (TDP H100 SXM5)} + 800\,\text{W (server)} \approx 3.6\,\text{kW (IT)}$$

The total datacenter power includes the cooling overhead, expressed by the PUE:

$$P_{\text{total}} = P_{\text{IT}} \times \text{PUE}$$ $$\text{annual energy cost} = P_{\text{IT}} \times \text{PUE} \times 8{,}760\,\text{h} \times \text{kWh price}$$

With the base values (PUE 1.54; 0.116 €/kWh):

$$\text{energy/year} = 3.6\,\text{kW} \times 1.54 \times 8{,}760\,\text{h} \times 0.116\,\text{EUR/kWh} \approx 5{,}475\,\text{EUR}$$

With a Spanish solar PPA (Q3 2025 reference price: ~34 €/MWh = 0.034 €/kWh according to PV Tech, Oct. 2025):

$$\text{energy/year (solar PPA)} = 3.6 \times 1.54 \times 8{,}760 \times 0.034 \approx 1{,}604\,\text{EUR}$$

Energy scenario	Price (€/kWh)	Energy cost/year per 4-GPU node
Spanish solar PPA (Q3 2025)	0.034	~1,604 €
Industrial Spain (Sept. 2025)	0.116	~5,475 €
European average (industrial tariff)	0.160	~7,550 €
Worst case (no PPA, high tariff)	0.200	~9,437 €

Staff / operations

Staff cost is the most variable line item with cluster size. For a small cluster (2–8 nodes), the rule of thumb is 0.3–0.5 FTE per cluster of GPU infrastructure support (Spheron, Apr. 2026).

Cluster size	Estimated FTE	FTE cost (€/year, Western Europe)	Cost per 4-GPU node (€/year)
2–4 nodes	0.3 FTE	~120,000	36,000–18,000
8–16 nodes	0.5 FTE	~120,000	7,500
32+ nodes	1–2 FTE	~120,000	3,750–7,500

Indicative salary reference: a GPU infrastructure engineer with CUDA, InfiniBand and Kubernetes knowledge in Western Europe, ~90,000–140,000 €/year fully loaded. Introl’s figures (Apr. 2026) in USD (~275,000 USD/year for the US) reflect the North American market, which is appreciably higher.

Maintenance and support

Line item	Annual cost (% of hardware capex)	Per 4-GPU node (midpoint)
Vendor maintenance / support	5–10 % of capex	~7,000–14,000 USD → ~6,500–13,000 €
GPU failure rate (~5 % annual) × replacement cost	5 % × 4 GPUs × ~35,000 USD = ~7,000 USD expected	~6,500 € (amortised as a provision)
Minor spares (cables, modules)	~500–1,000 €	—

Introl quotes GPU failure rates of 2–3 % per year in small clusters; Google Research documented ~9 % annualised in Meta’s 16,384-GPU H100 cluster (Introl, Apr. 2026). 5 % is used here as a conservative intermediate value.

Depreciation (for accounting purposes)

Straight-line depreciation turns capex into an annual flow comparable to the cost of committed cloud:

$$\text{annual depreciation} = \frac{\text{node capex}}{\text{amortisation years}}$$

Node capex (USD)	3-year amortisation (USD/year)	5-year amortisation (USD/year)
150,000 (minimum)	50,000	30,000
178,500 (average)	59,500	35,700
207,000 (maximum)	69,000	41,400

H100 hardware depreciates fast: secondary-market analyses put the residual value at 20–40 % of the purchase price after 3 years (Introl, Apr. 2026). The arrival of Blackwell GB200/GB300 accelerates the perceived obsolescence.

Annual opex summary per 4×H100 SXM5 node (base scenario, 8-node cluster)

Line item	Base scenario (€/year)	Range
Energy (PUE 1.54; 0.116 €/kWh)	5,475	1,604–9,437
Staff (0.5 FTE × 8 nodes, prorated)	7,500	3,750–36,000
Maintenance / support / failures	9,000	5,000–15,000
Rack colocation (Spain, high density)	6,000	3,000–15,000
Total opex per node	~28,000	~13,000–75,000

The extreme range reflects the difference between a well amortised own datacenter with a solar PPA and cheap energy (minimum opex) and tier-1 colocation with market tariffs and senior staff.

Deriving the all-in €/GPU-hour

Formula

$$\text{EUR/GPU-hour all-in} = \frac{\frac{\text{node capex}}{\text{years}} + \text{node annual opex}}{4\,\text{GPUs} \times 8{,}760\,\text{h} \times u}$$

where $u$ is the average annual utilisation (0 to 1).

See the cost-per-token identity in cost per token and per request for the connection with throughput.

€/GPU-hour table by utilisation and scenario

Average capex (178,500 USD → ~166,000 €), 3-year amortisation → 55,300 €/year.

Utilisation	Opex/year (base, €)	Total cost/year (€)	Useful GPU-hours/year	€/GPU-hour
30 %	28,000	83,300	10,512	7.93
50 %	28,000	83,300	17,520	4.75
70 %	28,000	83,300	24,528	3.39
80 %	28,000	83,300	28,032	2.97
100 %	28,000	83,300	35,040	2.38

Low-opex scenario (solar PPA, own datacenter, large cluster): opex/year ~13,000 €.

Utilisation	Total cost/year (€)	€/GPU-hour
50 %	68,300	3.90
70 %	68,300	2.78
80 %	68,300	2.43
100 %	68,300	1.95

High-opex scenario (market tariff, expensive colocation, small cluster): opex/year ~75,000 €.

Utilisation	Total cost/year (€)	€/GPU-hour
50 %	130,300	7.44
70 %	130,300	5.31
80 %	130,300	4.65
100 %	130,300	3.72

From €/GPU-hour to €/1M tokens

The cost-per-token identity connects the hardware cost with the inference cost:

$$\text{EUR/1M tokens} = \frac{\text{EUR/GPU-hour} \times 10^6}{\text{throughput (tok/s)} \times 3{,}600}$$

For reference throughputs on H100 SXM5 with vLLM (see capacity planning for on-premise inference):

Model	Typical throughput (tok/s per GPU)	Source
Llama-3 70B FP8, high batch	~2,800	Series B benchmarks
Llama-3 8B FP16, medium batch	~9,000	Series B benchmarks
Mixtral 8×7B, high batch	~4,500	Series B benchmarks

€/1M tokens table in the base scenario (€/GPU-hour 3.39 at 70 % utilisation):

Model	Throughput (tok/s)	€/1M tokens
Llama-3 70B FP8	2,800	~0.336
Llama-3 8B FP16	9,000	~0.105
Mixtral 8×7B	4,500	~0.209

At 50 % utilisation (€/GPU-hour 4.75):

Model	€/1M tokens
Llama-3 70B FP8	~0.471
Llama-3 8B FP16	~0.147

Occupancy (batching) multiplies the effective throughput and lowers the €/1M tokens without changing the hardware; it is analysed in GPU utilisation as a FinOps lever.

Break-even on-prem vs cloud

The break-even formula

Break-even happens when the total annual on-prem cost equals the annual cloud cost at the same utilisation:

$$\text{annual cloud cost} = \text{cloud GPU-hour price} \times 4\,\text{GPUs} \times 8{,}760\,\text{h} \times u$$ $$\text{break-even}: \quad \frac{\text{capex/year} + \text{opex/year}}{4 \times 8{,}760 \times u} = \text{cloud GPU-hour price}$$

Solving for the break-even utilisation:

$$u^* = \frac{\text{capex/year} + \text{opex/year}}{4 \times 8{,}760 \times \text{cloud GPU-hour price}}$$

Break-even table by cloud mode and on-prem scenario

Base on-prem scenario (capex/year 55,300 €, opex/year 28,000 €, total 83,300 €/year per 4-GPU node):

Cloud reference (price/GPU-hour)	USD equiv.	Break-even utilisation $u^*$	Note
Neocloud on-demand (Lambda/Spheron ~2.90 USD)	2.90 USD (~2.70 €)	>100 %, on-prem does not compete	Neocloud on-demand is cheaper even at full utilisation
Neocloud 3-year reserved (CoreWeave ~1.49–2.10 USD)	~1.80 USD (~1.67 €)	>100 %, impossible	Neocloud reserved beats on-prem in every scenario of this model
AWS p5 on-demand (6.88 USD/GPU-hour)	6.88 USD (~6.40 €)	~47 %	Above 47 %, the average on-prem beats AWS on-demand
AWS p5 3-year reserved (~2.97 USD/GPU-hour)	2.97 USD (~2.76 €)	>100 %	—
GCP A3 on-demand (~10.98 USD/GPU-hour)	10.98 USD (~10.21 €)	~29 %	Above 29 %, on-prem beats GCP on-demand
Azure ND H100 v5 on-demand (~12.29 USD/GPU-hour)	12.29 USD (~11.43 €)	~26 %	—

Low-opex scenario (total 68,300 €/year):

Cloud reference	Break-even utilisation
AWS p5 on-demand (6.88 USD ≈ 6.40 €)	~38 %
Neocloud on-demand (2.90 USD ≈ 2.70 €)	~91 %
Neocloud 3-yr reserved (1.80 USD ≈ 1.67 €)	>100 %

High-opex scenario (total 130,300 €/year):

Cloud reference	Break-even utilisation
AWS p5 on-demand (6.88 USD ≈ 6.40 €)	~72 %
GCP A3 on-demand (~10.21 €)	~45 %
Azure on-demand (~11.43 €)	~41 %

Reading the break-even table

Against neoclouds (on-demand or reserved), on-prem TCO does not reach break-even in any scenario of the base model. Neocloud reserved beats on-prem even at 100 % utilisation, because its hourly price is below the all-in cost of your own hardware. This is consistent with the analysis in cloud GPU: on-demand, reserved and spot.
Against on-demand hyperscalers (AWS, GCP, Azure), on-prem does have an achievable break-even: around 26–72 % utilisation depending on the scenario. At medium-high utilisation (>70 %), on-prem clearly beats AWS/GCP/Azure on-demand.
The variable that moves the break-even most is opex (staff above all), not hardware capex. A well sized cluster in cheap colocation with PPA energy can lower the threshold by 20 percentage points compared with the high scenario.
For GDPR data, the break-even against US hyperscalers is skewed: the sovereignty axis rules out US hyperscalers before cost does (see sovereign on-premise vs hyperscalers).

Sensitivity analysis

TCO vs utilisation

The all-in cost per GPU-hour varies inversely with utilisation because capex is fixed:

$$\frac{d(\text{EUR/GPU-hour})}{du} = -\frac{\text{capex/year} + \text{opex/year}}{4 \times 8{,}760 \times u^2} < 0$$

Going from 50 % to 80 % utilisation cuts the €/GPU-hour by $\frac{4.75 - 2.97}{4.75} \approx 37\,\%$ in the base scenario. That 37 % reduction requires no hardware change, only more efficient scheduling (see GPU utilisation as a FinOps lever).

Utilisation	€/GPU-hour (base scenario)	Change vs 50 %
30 %	7.93	+67 %
50 %	4.75	reference
70 %	3.39	−29 %
80 %	2.97	−37 %
100 %	2.38	−50 %

TCO vs energy price

Energy price (€/kWh)	Energy opex/year	€/GPU-hour (70 % util.)	Change vs base
0.034 (solar PPA)	1,604 €	3.00	−12 %
0.116 (industrial ES, base)	5,475 €	3.39	reference
0.160 (European average)	7,550 €	3.54	+4 %
0.200 (high tariff)	9,437 €	3.67	+8 %

Energy has a moderate impact on total TCO (8–12 % variation between the extremes), because hardware capex dominates. Over a very long amortisation (5 years) with a solar PPA, however, energy drops from 6 % to 1 % of total TCO and the differential is amplified. The price of energy matters more for the carbon footprint (CSRD) than for TCO when capex is dominant.

TCO vs PUE

PUE	Cooling overhead	Energy/year (0.116 €/kWh)	€/GPU-hour (70 % util.)
1.15 (liquid cooling, new facilities)	+15 %	2,166 €	3.21
1.20 (liquid, modern datacenter)	+20 %	2,259 €	3.23
1.48 (facilities <5 years, Uptime 2025)	+48 %	3,490 €	3.33
1.54 (global average, Uptime 2025)	+54 %	3,627 €	3.39
1.80 (legacy colocation)	+80 %	4,260 €	3.47

The difference between PUE 1.15 (liquid) and 1.80 (legacy) is barely ~8 % of the €/GPU-hour at 70 % utilisation, because energy is only a fraction of TCO. PUE matters far more for the absolute energy cost and for CSRD reporting than for total TCO when hardware is the dominant component.

TCO vs amortisation years

Amortisation	Capex/year (average node, USD)	€/GPU-hour (70 % util., base opex scenario)
3 years	59,500 USD (~55,300 €)	3.39
4 years	44,625 USD (~41,500 €)	2.99
5 years	35,700 USD (~33,200 €)	2.72

Stretching the amortisation from 3 to 5 years lowers the €/GPU-hour by ~20 %, assuming the hardware remains competitive and the resale market supports the residual value. With the refresh cycle accelerated by Blackwell GB200/GB300, a 5-year amortisation carries a higher risk of technological obsolescence.

Sensitivity heat map (€/GPU-hour at 70 % utilisation, base scenario)

	PUE 1.15	PUE 1.54	PUE 1.80
3-yr amort., solar PPA (0.034 €)	2.72	2.74	2.76
3-yr amort., industrial (0.116 €)	3.21	3.39	3.47
5-yr amort., industrial (0.116 €)	2.54	2.72	2.80
3-yr amort., high tariff (0.200 €)	3.44	3.67	3.78

Decision table: cost/control/sovereignty Pareto

The table below crosses the four dimensions with no implicit hierarchy; the ordinal reading depends on each organisation’s constraints.

Option	€/GPU-hour	Initial capex	Full stack control	EU sovereignty	Elasticity	Operational risk
On-prem (util. >70 %, low opex)	2.40–3.00	high (150–207k USD/node)	total	total	none	hardware failure, idle
On-prem (util. <50 %, base opex)	4.75–7.93	high	total	total	none	capex with no return
Neocloud 3-year reserved (CoreWeave, Lambda)	1.49–2.10 USD	none	partial (API)	depends on the provider	rigid contract	minimal interruption
Neocloud on-demand (Lambda, Spheron)	2.49–3.44 USD	none	partial	depends	total	no interruption
AWS p5 on-demand	6.88 USD	none	minimal	NO (CLOUD Act)	total	no interruption
AWS p5 3-year reserved	~2.97 USD	financial commitment	minimal	NO (CLOUD Act)	rigid	no interruption
Sovereign EU cloud (Scaleway, Nebius EU)	2.15–3.85 USD	none	partial	yes (EU)	total	no interruption
Hybrid on-prem base + EU cloud peak	2.00–3.50 (weighted)	medium	high	yes (EU)	elastic peak	operational complexity

“EU sovereignty” column: US hyperscalers (AWS, GCP, Azure) are subject to the US CLOUD Act regardless of the datacenter region. Nebius has a Dutch legal entity; CoreWeave is a US company. See the full analysis in sovereign on-premise vs hyperscalers.

“Full stack control” column: on-prem lets you choose the driver version, the kernel, the NCCL configuration, MIG partitioning, and any system parameter. Cloud options offer control at container/pod level, with the hypervisor and firmware opaque.

The cost/sovereignty Pareto frontier for GDPR data excludes US hyperscalers, leaving: on-prem, sovereign EU cloud, and the hybrid. Among those three, the deciding variable is sustained utilisation and the predictability of traffic (see capacity planning for on-premise LLM inference).

Integration with the series FinOps model

The all-in €/GPU-hour of on-prem is the number that feeds the cost allocation pipeline of the series:

Cost-per-token identity (cost per token and per request): engine throughput × €/GPU-hour → €/1M tokens.
Chargeback and showback (chargeback and showback in GPU multitenancy): the all-in €/GPU-hour is the internal price charged to each tenant of the multi-tenant cluster.
Utilisation as a lever (GPU utilisation as FinOps): raising utilisation from 50 % to 80 % cuts the €/GPU-hour by 37 % without changing the hardware, the highest ROI in on-prem FinOps.
Capacity planning (capacity planning for on-premise LLM inference): the number of nodes to buy depends on the percentile of base load you want to cover in iron.
Cloud comparison (cloud GPU: on-demand, reserved and spot): the all-in €/GPU-hour goes head to head with the cloud price in table A7 to compute the break-even.

Sources

Spheron · LLM Inference On-Premise vs GPU Cloud: 2026 Cost and Break-Even Analysis (abr. 2026) — https://www.spheron.network/blog/llm-inference-on-premise-vs-cloud/
Introl · GPU Infrastructure TCO Model: 5-Year Cost Analysis for Enterprise AI (abr. 2026) — https://introl.com/blog/gpu-infrastructure-tco-5-year-cost-model
GMI Cloud · NVIDIA H100 GPU Pricing: 2026 Rent vs. Buy Cost Analysis (abr. 2026) — https://www.gmicloud.ai/en/blog/nvidia-h100-gpu-pricing-2026-rent-vs-buy-cost-analysis
Uptime Institute · Global Data Center Survey 2025 (PDF oficial) — https://datacenter.uptimeinstitute.com/rs/711-RIA-145/images/2025.Annual.Survey.Report.pdf
Uptime Institute · Global Data Center PUE Stalls at 1.54 (comunicado, oct. 2025) — https://mgrid.org/2025/10/01/uptime-institute-data-center-pue-stagnation-2025-liquid-cooling/
GlobalPetrolPrices · Spain Business Electricity Price (sept. 2025) — https://www.globalpetrolprices.com/Spain/electricity_prices/
PV Tech · European Solar PPA Prices Fall Below 35 €/MWh in Q3 2025 (oct. 2025) — https://www.pv-tech.org/european-solar-ppa-prices-fall-below-35-mwh-q3-2025/
Xi Computers · Supermicro SYS-821GE-TNHR 8U GPU Server Pricing (jun. 2026) — https://www.xicomputer.com/Solutions/Data-Center-Servers/SYS-821GE-TNHR.asp
Encoradvisors · Data Center Colocation Pricing 2026 — https://encoradvisors.com/data-center-colocation-pricing/
CloudZero · H100 GPU Cost In 2026: Buy, Rent, and Cloud Pricing Compared — https://www.cloudzero.com/blog/h100-gpu-cost/
Spheron · AI Inference Power Consumption and GPU Electricity Costs: 2026 Guide — https://www.spheron.network/blog/ai-inference-power-electricity-cost-2026/
NVIDIA · DGX SuperPOD H100 Electrical Specifications (documentación oficial) — https://docs.nvidia.com/dgx-superpod/design-guides/dgx-superpod-data-center-design-h100/latest/electrical.html

Cloud GPU: on-demand, reserved and spot, and why neoclouds change the board (with 2026 prices)

Tue, 16 Jun 2026 05:00:00 +0200

Notation: amounts in N USD or N €, decimal point. The dollar sign is not used (on this site it is the formula delimiter). All cloud prices change without notice; the date of the source is given next to each figure.

TL;DR

An H100 SXM5 on-demand costs between 2.49 USD/GPU-hour (Lambda, a neocloud) and 12.29 USD/GPU-hour (Azure ND H100 v5) depending on the provider, a spread of 5× between the neocloud floor and the hyperscaler ceiling. The reserved mode (a 1-3 year commitment) lowers the price by 20-40 % and is the number comparable to amortising your own iron; the spot/preemptible mode takes another 40-65 % off, but with a risk of interruption on less than 2 minutes’ notice. For the on-prem vs cloud analysis of this series (article sovereign on-premise vs hyperscalers), neocloud reserved is the correct reference: at 1.49-2.10 USD/GPU-hour with a 3-year commitment, neoclouds compete head-on with the TCO of your own iron at medium utilisation.

The three cloud GPU purchasing models

On-demand

Pay per use with no commitment, billed by the hour or by the minute. Maximum flexibility; maximum price. Applies to: development, experimentation, occasional capacity peaks and any workload whose lifetime is unpredictable. The absence of commitment has a cost: the on-demand premium over reserved ranges from 25 % to 60 % depending on the provider.

Reserved / committed

A 1 or 3 year contract that guarantees capacity in exchange for a discount on the on-demand rate. At hyperscalers it goes by the name Reserved Instance (AWS), Committed Use Discount (GCP) or Azure Reservation. At neoclouds it is negotiated directly and can include the choice of node, network (Ethernet or InfiniBand) and region. Typical discounts:

Term	Discount on on-demand (market reference)
1 year	20-37 %
3 years	30-45 %

Reserved is the model comparable to on-prem: in both cases a fixed cost is incurred up front (amortised capex on-prem; committed payment in the cloud). The formula for the effective cost per GPU-hour is the same in both cases:

$$\text{cost/GPU-hour}_{reserved} = \frac{\text{committed price (USD/GPU-h)} \times 8760}{\text{contracted hours}}$$

Unlike on-prem, cloud reserved includes neither hardware capex nor operational opex, but neither does it include the benefits of running your own iron at high load (see GPU utilisation as a FinOps lever).

Spot / preemptible

Idle provider capacity offered at a 40-65 % discount on on-demand, on the condition that it can be reclaimed with 30 seconds to 2 minutes of notice (AWS Spot, GCP Spot VMs). The discount is real and reproducible; so is the interruption risk.

Workloads suitable for spot:

Workload	Requires checkpointing	Suitable for spot
LLM pre-training (checkpoint-resume every 15-30 min)	yes	yes
LoRA/QLoRA fine-tuning with checkpoints	yes	yes
Offline batch inference (embeddings, evaluations)	no (idempotent)	yes
Hyperparameter search	no (repeatable)	yes
Production serving with a latency SLO	not applicable	no
Production serving with an availability SLO	not applicable	no

A fine-tune of a 7B model on 4× H100 over 40 hours on spot achieved real savings of 81 % against the equivalent on-demand (Spheron Blog, May 2026). The typical interruption rate on high-reliability H100 instances is 2-8 % per day, which gives a 92-98 % probability of completing a 24-hour shift without a cut (Thunder Compute, Jun. 2026).

The neoclouds: GPU-specialised clouds

Definition and differentiators

Neoclouds (or GPU-specialised clouds) are infrastructure providers focused exclusively on GPU compute for AI workloads, without the diversity of services of the hyperscalers. SemiAnalysis coined the term in 2024 and identifies CoreWeave, Nebius, Lambda and Crusoe at the top tier (SemiAnalysis, 2024).

Structural differences with respect to the hyperscalers:

Dimension	Neoclouds (CoreWeave, Lambda, Nebius…)	Hyperscalers (AWS, GCP, Azure)
H100 on-demand price	2.49-6.16 USD/GPU-hour	6.88-12.29 USD/GPU-hour
Interconnect network	InfiniBand NDR 400 Gbps (CoreWeave, Voltage Park), Ethernet 200+ Gbps (Lambda, Nebius)	Proprietary Ethernet (EFA, GVNIC) or InfiniBand (only p5/H100 on AWS)
Access to new hardware	NVIDIA priority access (CoreWeave is an NVIDIA shareholder); H200/B200 catalogue available earlier	Slower catalogue; B200 in reserved or limited capacity in 2026
Contracts	1-36 months, negotiated directly	Standard RI (1-3 years), no node negotiation
Additional services	Managed Kubernetes, storage, bare-metal	Complete ecosystem (IAM, VPC, databases, ML services…)
Egress	Zero-egress or flat fee (Lambda, Voltage Park, Nebius)	0.08-0.12 USD/GB (AWS, GCP, Azure)
Sovereignty	Varies: Nebius (EU), Crusoe (US, clean), CoreWeave (US/EU)	US CLOUD Act; no guarantee of EU jurisdiction

The neocloud market is estimated at 20,000 M USD of revenue in 2026 and is growing towards 180,000 M USD in 2030 (Synergy Research Group, 2026). CoreWeave passed 5,000 M USD of ARR faster than any other cloud platform. Neoclouds price their H100s 60-85 % below AWS and Azure for the same silicon (Signisys, 2026).

Profiles of the main neoclouds

Provider	Main hardware	Network	Jurisdiction	Relevant note
CoreWeave	H100, H200, B200, GB200	InfiniBand NDR (Quantum-2 SHARP)	US / EU	NVIDIA shareholder; priority access to hardware; largest neocloud by fleet
Lambda Labs	H100, A100, B200	Ethernet	US	Zero-egress; no spot; per-minute billing
Nebius	H100, H200, B200, B300	InfiniBand	EU (Amsterdam)	Spin-off of Yandex Cloud; zero-egress; object egress 0.015 USD/GiB; public on-demand price
Crusoe	H100, A100, MI300X	Ethernet	US	100 % renewable/flare gas energy; AMD MI300X at 3.45 USD/hr
Voltage Park	H100	Ethernet (1.99 USD) / InfiniBand 3200 Gbps (2.49 USD)	US	Merged with Lightning AI in Jan. 2026; no minimums; non-profit Navigation Fund
Together AI	H100, A100	Ethernet	US	Inference platform + raw compute
GMI Cloud	H100, H200	Ethernet	Asia/US	H200 on-demand from 2.60 USD/hr
RunPod	H100, H200, B200, A100	Ethernet / Secure Cloud	Global	Community + Secure Cloud with SLA; spot available

Note: the “zero-egress”, “renewable” and priority hardware access claims are marketing from each provider and have not been verified by independent audits. They are reproduced with that caveat.

2026 price table: H100 SXM by provider and mode

Source and date given for each figure. Prices change without notice.

H100 SXM5 (80 GB HBM3) — USD per GPU-hour

Provider	On-demand	1-year reserved	3-year reserved	Spot/Preemptible	Source and date
CoreWeave	~6.16	~2.10-3.00 (negotiated)	~1.49-2.10 (negotiated)	not available	Thunder Compute, Jun. 2026
Lambda Labs	2.49-3.44	~1.89 (37 % disc.)	~1.84 (PCIe 3-yr)	not available	Spheron Blog, May 2026
Nebius	3.85	discount up to 35 % (contact)	discount up to 35 % (contact)	2.15 (preemptible)	nebius.com/prices, Jun. 2026
Voltage Park	1.99 (Ethernet) / 2.49 (InfiniBand)	contact (≥6 months)	contact	not available	voltagepark.com/pricing, Jun. 2026
RunPod (Secure)	3.29	not available	not available	available (~59 % disc.)	Spheron Blog, May 2026
Spheron	2.50	volume discount (contact)	volume discount (contact)	1.03	Spheron pricing, May 2026
AWS (p5.48xlarge)	6.88	~2.97 (3-yr, ~23.78 USD/h node ÷ 8)	~2.97	~1.14 (p5 spot, 9.10 USD/h node)	Vantage, Jun. 2026
GCP (A3 High)	10.98	~8.78 (1-yr CUD)	n/a	~3.69	Spheron Blog, May 2026
Azure (ND H100 v5)	12.29	~6.15-8.60 (1-3 yr reservation)	~6.15-8.60	~2.27 (node spot ÷ 8)	Vantage Azure, Jun. 2026

All amounts in USD/GPU-hour, rounded to 2 decimals. The “3-year reserved” column is the number comparable to the amortisation cost of your own hardware.

AWS p5.48xlarge: an 8× H100 node; on-demand 55.04 USD/h → 6.88 USD/GPU-h; 3-yr reserved 23.78 USD/h → 2.97 USD/GPU-h; spot 9.10-25.35 USD/h (varies by region and availability). GCP a3-highgpu-8g: 8× H100 SXM5; on-demand 87.84 USD/h → 10.98 USD/GPU-h; spot ~3.69 USD/GPU-h (us-central1, May 2026). Azure ND96isr H100 v5: 8× H100; on-demand 98.32 USD/h → 12.29 USD/GPU-h; node spot 18.17 USD/h → ~2.27 USD/GPU-h (source: Vantage, Jun. 2026, us-east-1).

2026 price table: H200 and B200

H200 SXM (141 GB HBM3e) — USD per GPU-hour

Provider	On-demand	Spot/Preemptible	Source and date
Nebius	4.50	2.45 (preemptible)	nebius.com/prices, Jun. 2026
GMI Cloud	2.60	not available	GMI Cloud, May 2026
RunPod	4.39	not available	Spheron Blog, May 2026
Spheron	4.54	not available	Spheron pricing, May 2026
CoreWeave	6.31	not available	GMI Cloud, May 2026
AWS (p5e)	~4.98	limited	Spheron Blog, May 2026
Azure	~13.78	not available	Spheron Blog, May 2026

B200 SXM6 (192 GB HBM3e) — USD per GPU-hour

Provider	On-demand	Spot	Source and date
Lambda Labs	4.99-5.29	not available	Spheron Blog, May 2026
Nebius (HGX B200)	7.15	3.95 (preemptible)	nebius.com/prices, Jun. 2026
RunPod	5.89	not available	Spheron Blog, May 2026
Spheron	6.02	2.12	Spheron pricing, May 2026
AWS (p6-B200)	~14.24	~3.24	Spheron Blog, May 2026

B200 availability is still limited in Q2 2026; many providers offer only access by advance reservation. The prices are market estimates subject to rapid change.

How it fits into the on-prem vs cloud comparison

The cloud cost per GPU-hour is the number that goes head to head with the on-prem TCO computed in sovereign on-premise vs hyperscalers. The relationship between the three purchasing modes and the cost analysis is as follows:

Cloud mode	What it represents in the comparison	On-prem equivalent
On-demand	marginal cost of peak capacity; floor of build-vs-buy at 100 % utilisation	— (not comparable: on-prem has no elasticity)
3-year reserved	the committed cost comparable to capex amortised over 3 years	node amortisation + fixed annual opex
Spot	the cloud price floor, but with interruption risk	— (not comparable: on-prem is not interrupted)

The cost per token formula applies the same to cloud as to on-prem:

$$\text{cost/1M tokens}_{cloud} = \frac{\text{GPU-hour price (USD)} \times 10^6}{\text{throughput (tok/s)} \times 3600}$$

With an H100 SXM5 at 2,800 tok/s (typical throughput for a 70B model in FP8):

Mode	Reference price	Cost/1M tokens (70B, 2800 tok/s)
Neocloud on-demand (Lambda)	2.49 USD/GPU-h	~0.247 USD (~0.23 €)
Neocloud 3-yr reserved (CoreWeave)	~1.49 USD/GPU-h	~0.148 USD (~0.14 €)
Neocloud spot (Spheron/Nebius)	~1.03-2.15 USD/GPU-h	~0.102-0.213 USD
AWS on-demand (p5)	6.88 USD/GPU-h	~0.682 USD (~0.63 €)
AWS 3-yr reserved (p5)	~2.97 USD/GPU-h	~0.295 USD (~0.27 €)
GCP spot (A3)	3.69 USD/GPU-h	~0.366 USD (~0.34 €)
On-prem amortised, high util. (80 %)	~2.83 USD/GPU-h all-in	~0.281 USD (~0.26 €)

Reference exchange rate: 1 USD = 0.924 €. The rate fluctuates; comparative calculations must be done in the same currency.

The key reading: at high utilisation, on-prem TCO (~2.83 USD/GPU-hour all-in according to Spheron, 2026) competes with the 3-year reserved of the neocloud (~1.49-2.97 USD/GPU-hour). On-prem wins at high utilisation when the all-in cost drops below 2.5 USD; neocloud reserved wins when utilisation is medium or low, because it does not pay for idle capex. Spot lowers the cloud floor but is not comparable: interruption makes it incompatible with continuous serving, which is the case where on-prem competes.

Network factors: InfiniBand vs Ethernet

For distributed training and large-scale inference (tensor parallelism across nodes), the bandwidth of the interconnect network between GPUs is critical. Neoclouds differ in what they offer:

Provider	GPU-GPU network	Bandwidth	Applies to
CoreWeave	InfiniBand NDR (Quantum-2, SHARP)	400 Gbps/GPU	multi-node HGX clusters
Voltage Park	InfiniBand Quantum-2	3200 Gbps (8-GPU node)	the 2.49 USD/hr tier
Voltage Park	Ethernet	200 Gbps	the 1.99 USD/hr tier
Lambda Labs	Ethernet	200 Gbps	all nodes
Nebius	InfiniBand	400 Gbps/GPU	HGX H100/H200/B200
AWS p5	EFA (Elastic Fabric Adapter)	3200 Gbps (node)	EC2 clusters
GCP A3	GVNIC + RoCE	200 Gbps/GPU	A3 High
Azure ND H100 v5	InfiniBand NDR	400 Gbps/GPU	ND96isr H100 v5

InfiniBand NDR with SHARP (in-network collective reduction) cuts the latency of allreduce operations in distributed training. For inference serving with tensor parallelism inside a node (NVLink), the inter-node network is less critical; for multi-node training or disaggregated prefill-decode (the three axes of the series), the network between nodes matters.

Neoclouds and sovereignty: what changes

Unlike the US hyperscalers subject to the US CLOUD Act, some neoclouds offer options with jurisdiction outside the US:

Provider	EU region	US CLOUD Act applies
Nebius	Amsterdam (NL), Frankfurt (DE)	No (Dutch legal entity)
CoreWeave	Frankfurt (DE), London (GB)	Yes (US company)
Lambda Labs	US only	Yes
Voltage Park	US only	Yes
Scaleway	Paris (FR), Amsterdam (NL)	No (French company)
US hyperscalers	EU regions available	Yes (CLOUD Act applies)

For data subject to GDPR, only providers with an EU/EFTA legal entity guarantee the absence of the CLOUD Act. The full discussion of the sovereignty axis is in sovereign on-premise vs hyperscalers. For capacity planning with this data, see capacity planning for on-premise inference.

Decision table: purchasing mode by cost/risk/sovereignty axis

Mode	Cost	Operational risk	Sovereignty (EU)	When to use
Neocloud on-demand	medium (2-4 USD/GPU-h)	low (no interruption)	depends on the provider	development, unpredictable peaks, serving with an SLO
Neocloud 1-yr reserved	low-medium (1.5-3 USD/GPU-h)	low (no interruption, guaranteed capacity)	depends on the provider	stable serving, scheduled training; validated workload
Neocloud 3-yr reserved	low (1.5-2.1 USD/GPU-h)	low, financial commitment	depends on the provider	comparable to amortised on-prem; very stable base load
Neocloud spot	minimum (1-2.5 USD/GPU-h)	high (interruption <2 min)	depends on the provider	batch, fine-tuning with checkpoints, hyperparameter search
Hyperscaler on-demand	high (6.9-12.3 USD/GPU-h)	low	NO (CLOUD Act)	integration with AWS/GCP/Azure services; no alternative
Hyperscaler reserved	medium-high (3-9 USD/GPU-h)	low	NO (CLOUD Act)	lock-in to the hyperscaler ecosystem is justified
Hyperscaler spot	medium (1.1-3.7 USD/GPU-h)	high (interruption)	NO (CLOUD Act)	batch with no sovereignty requirement; more availability than neocloud spot
Amortised on-prem (>70 % util.)	low (~2.3-2.8 USD/GPU-h all-in)	low (no interruption)	total (your own hardware)	stable, high base load; maximum sovereignty

On-prem competes with neocloud 3-yr reserved when utilisation exceeds 65-70 % (see GPU utilisation as FinOps).

Computing the spot saving: an example with checkpointing

Reference parameters to validate whether spot is worth it:

$$\text{spot saving} = (\text{OD price} - \text{spot price}) \times \text{total GPU-hours} - \text{retry cost}$$ $$\text{retry cost} = \text{interruption rate} \times \text{lost work (GPU-hours)} \times \text{spot price}$$

With checkpointing every 30 minutes, work lost per interruption is ≤ 0.5 GPU-hours per event. At an interruption rate of 4 % per day on one H100 GPU:

Item	Reference value
On-demand price (neocloud)	2.50 USD/GPU-h
Spot price	1.03 USD/GPU-h
Gross saving per GPU-hour	1.47 USD
Expected interruptions in 40 h (4 %/day)	~0.67 events
Re-run work per event (0.5 h × 1.03 USD)	~0.34 USD
Net saving on a 40 h × 4 GPU job	~234 USD (against 400 USD on-demand)

The 58 % saving only materialises if the job implements checkpoint-resume correctly. Without checkpoints, a single interruption forces a restart from scratch: the saving turns into an overcost from duplicated work.

Integration into the series TCO model

The cloud €/GPU-hour is the number that feeds the FinOps analysis of the series. The chain:

Cost of cloud iron (this price table) → equivalent to amortised capex + opex of on-prem.
Throughput (measured with genai-perf, track B of the series) → the denominator.
Cost per token (cost per token and per request) → the final comparative metric.
Utilisation (GPU utilisation as FinOps) → in on-demand cloud, utilisation does not penalise cost (you pay for what you use); in reserved and in on-prem it does penalise, because the cost is fixed.

The correct comparison is neocloud 3-yr reserved vs on-prem amortised over 3 years, with the same assumptions on throughput, egress and utilisation. Spot is the cloud price floor but not the comparable floor: its interruptible nature excludes it from any comparison with an on-prem cluster that serves continuous traffic.

Sources

Spheron Blog · GPU Cloud Pricing 2026: H100 from 1,03 USD/hr, B200 from 2,12 USD/hr (15+ providers), may. 2026 — https://www.spheron.network/blog/gpu-cloud-pricing-comparison-2026/
Vantage · p5.48xlarge pricing and specs (AWS), actualizado jun. 2026 — https://instances.vantage.sh/aws/ec2/p5.48xlarge
Nebius · NVIDIA GPU Pricing (página oficial, jun. 2026) — https://nebius.com/prices
Voltage Park · H100 GPU Pricing (página oficial, jun. 2026) — https://www.voltagepark.com/pricing
Thunder Compute · CoreWeave GPU Pricing Review (jun. 2026) — https://www.thundercompute.com/blog/coreweave-gpu-pricing-review
Thunder Compute · NVIDIA H100 Pricing (Jun 2026) — https://www.thundercompute.com/blog/nvidia-h100-pricing
Spheron Blog · Lambda Cloud H100 Pricing 2026 — https://www.spheron.network/blog/lambda-cloud-h100-pricing-2026/
Spheron Blog · AWS H100 Pricing 2026: P5 Instance Cost — https://www.spheron.network/blog/aws-h100-pricing-2026/
Spheron Blog · Google Cloud A3 H100 Pricing 2026 — https://www.spheron.network/blog/google-cloud-a3-h100-pricing/
Spheron Blog · Azure H100 Pricing 2026: ND H100 v5 — https://www.spheron.network/blog/azure-h100-pricing/
GMI Cloud · H200 GPU Provider Pricing 2026 (CoreWeave, Lambda, Nebius, GMI) — https://www.gmicloud.ai/en/blog/h200-gpu-provider-pricing
Spheron Blog · GPU Spot Instance Arbitrage 2026 — https://www.spheron.network/blog/gpu-spot-instance-arbitrage-2026/
Thunder Compute · Cloud GPU Spot Instance Availability and Interruption Rates — https://www.thundercompute.com/blog/cloud-gpu-spot-instance-availability
Synergy Research Group · Neocloud Market Forecast to Approach 400B USD by 2031 — https://www.srgresearch.com/articles/neocloud-market-forecast-to-approach-400b-by-2031-driven-by-surging-ai-infrastructure-demand
Signisys · The Neocloud Revolution: How 20 Billion USD in GPU Providers Reshapes Cloud Market — https://www.signisys.com/blog/the-neocloud-revolution-how-20-billion-in-gpu-focused-providers-are-reshaping-the-cloud-market/
SemiAnalysis · The Great GPU Shortage: Rental Capacity (neocloud taxonomy original) — https://newsletter.semianalysis.com/p/the-great-gpu-shortage-rental-capacity
Spheron Blog · LLM Inference On-Premise vs GPU Cloud: 2026 Cost and Break-Even — https://www.spheron.network/blog/llm-inference-on-premise-vs-cloud/
IntuitionLabs · H100 Rental Prices Compared 1,49-6,98 USD/hr Across 15+ Cloud Providers (2026) — https://intuitionlabs.ai/articles/h100-rental-prices-cloud-comparison

LLM energy leaderboards: comparing models by Wh/token and choosing on efficiency

Tue, 16 Jun 2026 04:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter). Generic example hardware; no real infrastructure.

TL;DR

There are three OSS leaderboards with public data and documented methodology for comparing the energy efficiency of LLMs in inference: Hugging Face AI Energy Score (166 models, Wh/query on H100, 1–5 star scale, launched February 2025), ML.ENERGY Leaderboard v3 (University of Michigan, J/token per task, Zeus tooling, December 2025) and MLPerf Power (samples/joule certified with a physical Yokogawa WT310E power meter). The three measure different dimensions and are not directly interchangeable. The available data shows that reasoning models consume up to 700× more energy than their non-reasoning equivalents; that MoE models consume roughly 3× fewer J/token than a dense model with equivalent active parameters; and that INT4 quantisation cuts consumption by up to 79 % against FP16 under favourable conditions. The inference engine (vLLM vs Transformers) can move the result another 25–40 %. Without fixing hardware, engine, batch size and task, no comparison between leaderboards is valid.

Track context

This article is C5 of the energy pillar. The base context:

The fundamentals of quantisation are a prerequisite for the quantisation section of this article.

The three leaderboards: technical data sheet

1 · Hugging Face AI Energy Score

Field	Detail
URL	huggingface.co/AIEnergyScore · huggingface.co/spaces/AIEnergyScore/Leaderboard
Organisation	Hugging Face (Sasha Luccioni et al.), with Salesforce and Cohere as initial partners
Launch	February 2025 (AI Action Summit, Paris); v2 December 2025
Models indexed	166 (v1, Feb 2025); +39 new ones in v2 (Dec 2025)
Tasks measured	10 tasks: text generation, summarisation, classification, image generation, ASR, audio generation, translation, question answering, reasoning (added in v2)
Unit of measurement	Wh (watt-hour) per 1,000 queries of the task
Reference hardware	NVIDIA H100 exclusively (a single GPU for class A/B models; multiple for class C)
Measurement tooling	CodeCarbon (GPU energy) + the `ai-energy-benchmarks` package (OSS, PyPI)
Rating system	1–5 stars per task: quintiles of the energy range; ⭐⭐⭐⭐⭐ = the 20 % most efficient
Reference batch size	Batch size = 1 (does not reflect production with aggressive batching)
Access for proprietary models	Yes, via an audited Docker container
Update frequency	No fixed cadence; v1 Feb 2025, v2 Dec 2025
Project licence	Apache 2.0 (repository github.com/huggingface/AIEnergyScore)

Scope of the metric. The AI Energy Score measures GPU energy only (CodeCarbon); it does not capture CPU, DRAM or system overhead. The Wh/1k-queries unit covers the whole run time (prefill + decode + framework overhead), but at batch = 1. The results are therefore comparable between models under the same test conditions, but not extrapolable to a production environment with real concurrency without correction.

Model class (the project’s internal classification):

Class	Definition
A	Fits on a consumer GPU (≤ ~24 GB VRAM)
B	Requires a cloud GPU (≥ 40 GB VRAM)
C	Requires multiple GPUs

2 · ML.ENERGY Leaderboard

Field	Detail
URL	ml.energy/leaderboard
Organisation	Symbiotic Lab, University of Michigan (Mosharaf Chowdhury, Jae-Won Chung et al.)
Reference paper	arXiv 2505.06371 — “The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization” (NeurIPS 2025 D&B, Spotlight)
Current version	v3.0 (December 2025)
Measurement tooling	Zeus (github.com/ml-energy/zeus) via NVML + RAPL; measurement overhead in single-digit ms
Unit of measurement	J/token (energy per output token generated) and total energy per complete response
Reference hardware	NVIDIA A100 80 GB and H100 SXM (declared per submission; varies between models)
Tasks measured	6 tasks: chat (text conversation), reasoning, code generation, summarisation, visual question answering, video generation
Normalisation	Mean energy per complete response (prefill + decode). Output J/token is also reported. The task is stated explicitly because it determines the output length
Scope of the measurement	GPU via NVML + CPU/DRAM via RAPL; not a power meter at the wall
Models covered	~40 architectures in the NeurIPS 2025 version; the web leaderboard is updated with more
Licence	Apache 2.0 (zeus: github.com/ml-energy/zeus); MIT (benchmark: github.com/ml-energy/benchmark)
Update frequency	Continuous on the web leaderboard; the paper is a point-in-time snapshot

Zeus as tooling. Zeus is the measurement engine of the ML.ENERGY Leaderboard and also a standalone package (pip install zeus-ml). It supports NVIDIA GPU (NVML), AMD GPU (ROCm), CPU (RAPL), DRAM (RAPL), Apple Silicon and NVIDIA Jetson. The ZeusMonitor adds measurement overhead in single-digit milliseconds. Since May 2025 it has been a PyTorch ecosystem project. MIT licence.

3 · MLPerf Power

The full data sheet is in the C4 article. A summary of the points relevant for comparison with the previous two:

Field	Detail
URL	mlcommons.org/benchmarks/inference-datacenter/
Organisation	MLCommons Power Working Group (>20 orgs)
Unit of measurement	samples/joule (throughput/mean power) = the inverse of J/sample
Hardware	Complete node measured at the wall (AC); Yokogawa WT310E analyser (±0.1 % of reading)
LLM tasks	GPT-J 6B, Llama 2 70B, Mixtral 8×7B (from v5.0)
Granularity	Complete node (GPU + CPU + RAM + fans + PSU losses); no attribution to individual workloads
Node overhead over GPU	25–45 % of total consumption in submissions with a physical analyser
Licence of the corpus	Public results on GitHub (mlcommons/inference_results_vX.Y); PTDaemon requires MLCommons membership

Comparison of the three leaderboards

Dimension	HF AI Energy Score	ML.ENERGY Leaderboard	MLPerf Power
Unit	Wh/1k-queries	Output J/token	samples/J (complete node)
Fixed hardware	H100 (all models)	A100/H100 (varies)	Depends on the submitter
Measurement	CodeCarbon (GPU)	Zeus NVML+RAPL	Physical AC power meter (Yokogawa)
System coverage	GPU only	GPU + CPU + DRAM	Complete node including fans and PSU
Batch size	1	Varies by task	Per LoadGen scenario
Models covered	166+ (text, image, audio)	~40 generative LLMs	Few (GPT-J, Llama 2, Mixtral)
Proprietary models	Yes (audited Docker)	No (OSS only)	Yes (MLCommons members)
External certification	No	No	Yes (SPEC PTDaemon)
Frequency	Point-in-time (v1, v2)	Continuous	Half-yearly (MLPerf rounds)
Licence	Apache 2.0	Apache 2.0 / MIT	Public results; PTDaemon: membership

Incompatibility between leaderboards. The three measure different dimensions: Wh/query ≠ J/token ≠ node samples/J. A direct comparison requires converting units and assuming that the hardware, the engine and the task are equivalent, which is rarely the case across leaderboards.

How energy per token is measured and normalised

The base identity is developed in the C2 article:

$$E_{\text{token}} \,[\text{J/tok}] = \frac{\bar{P} \,[\text{W}]}{\text{throughput} \,[\text{tok/s}]}$$

To compare models against each other, every factor other than the model must be fixed:

Factor	Effect if it varies	How to fix it
Hardware	H100 vs A100 vs L40S changes the result by 2–4×	Declare the exact hardware; compare only within the same HW
Inference engine	vLLM vs Transformers: 25–40 % difference in J/token	Fix the engine and the version
Batch size / concurrency	Batch 1 vs batch 32: throughput rises but so does power; the ratio varies	Declare the batch size; compare within the same regime
Model precision	FP16 vs INT8 vs INT4: up to −79 % energy	Declare the precision; do not mix
Response length	A query with 50 tokens ≠ one with 500	Use a fixed dataset or normalise per token
Measurement window	Including warm-up or idle inflates the numerator	Align the power window with the token window (see C2)

Conversion formula Wh/query ↔ J/token:

$$E_{\text{J/tok}} = \frac{E_{\text{Wh/query}} \times 3600}{\bar{n}_{\text{tokens/query}}}$$

Example: if a model consumes 0.05 Wh/query (= 180 J/query) and generates an average of 200 tokens per query:

$$E_{\text{J/tok}} = \frac{0{.}05 \times 3600}{200} = \frac{180}{200} = 0{.}9 \,\text{J/tok}$$

AI Energy Score data: concrete examples

The v2 data (December 2025, H100 hardware, batch = 1, text generation task with reasoning on/off):

Model	Active params	Reasoning	GPU Wh/1k queries	Stars (text-gen)
DistilGPT-2	82 M	—	1.31	⭐⭐⭐⭐⭐
SmolLM3-3B	3 B	Off	18.35	⭐⭐⭐⭐
SmolLM3-3B	3 B	On	12,791.22	⭐
Phi-4-reasoning-plus	15 B	Off	18.42	⭐⭐⭐⭐
Phi-4-reasoning-plus	15 B	On	9,461.61	⭐
DeepSeek-R1-Distill-Llama-70B	70 B	Off	49.53	⭐⭐⭐
DeepSeek-R1-Distill-Llama-70B	70 B	On	7,626.53	⭐

Source: Hugging Face AI Energy Score v2 (Dec 2025).

The reasoning multiplier. The energy increase when reasoning is switched on ranges from ×154 (DeepSeek-R1-Distill-Llama-70B) to ×697 (SmolLM3-3B). The direct cause: reasoning models generate between 300 and 800 times more tokens than their non-reasoning equivalents (internal chains of thought). The mean across the v2 corpus is ×30 extra energy for reasoning.

Newer models are not always more efficient. Of the 14 comparable models (no reasoning, no MoE, similar size) between the Feb 2025 and Dec 2025 cohorts: 8 out of 14 had equal or higher energy. The range runs from 3 % of the reference model’s energy to almost 2×. Parameter scale is no longer enough to estimate efficiency.

ML.ENERGY Leaderboard data: J/token by family

The data from the arXiv 2505.06371 paper and from leaderboard v3 (A100/H100 hardware, vLLM as the reference engine):

Scale within one family (Llama 3):

Size	Params	Relative J/token (base = 1B)	Params/energy ratio
Llama 3 · 1B	1 B	1.0×	—
Llama 3 · 8B	8 B	~2.1×	8× params → 2.1× energy
Llama 3 · 70B	70 B	~7.3×	70× params → 7.3× energy

The sublinearity (70× params → 7.3× energy, not 70×) reflects that inference energy is dominated by memory bandwidth (memory-bandwidth bound), not by raw FLOPs.

Dense vs MoE:

Model	Type	Total params	Active params/token	Relative J/token
Llama 3 · 8B	Dense	8 B	8 B	1.0×
Mixtral 8×7B	MoE (top-2)	47 B	~13 B	~0.33×
Llama 3 · 70B	Dense	70 B	70 B	~3.5×

The MoE activates only 2 of 8 experts per token. Mixtral 8×7B consumes roughly ⅓ of the J/token of a dense model with 8B active, at a quality comparable to a dense model of larger scale. Routing overhead and holding all the experts in memory cancel out part of the theoretical gain.

Effect of the task (ML.ENERGY v3, same model):

Task	Energy multiplier per response (vs chat)
Chat (text conversation)	1× (reference)
Summarisation	~2–4×
Code generation	~3–6×
Reasoning	~25×
Image + text	1.1–5.2×
Video + text	1.3–15.0×

Reasoning uses ~10× more tokens per response, and the extra memory of the chain of thought reduces the effective batch size, raising energy per token through memory pressure.

Effect of quantisation on energy per token

Data on NVIDIA H100 hardware, Llama 3 family (arXiv 2508.16712 and arXiv 2504.03360):

Precision	Energy reduction vs FP16	Condition
FP16	reference (0 %)	—
BF16	~0 % (iso-energy)	Same hardware and engine
FP8	−25 to −35 %	H100/H200 with native hardware support
INT8	−23 to −44 % (median ~39 %)	Depends on batch size; more at low batches
INT4 (AWQ / GPTQ)	−50 to −79 %	Requires hardware with efficient low-precision support

Warning. On GPUs without native INT4 hardware support (or with suboptimal dequantisation kernels), quantisation can increase latency and energy per token rather than reduce them, because of run-time dequantisation overhead. The benefit of quantisation is real on H100/A100 with TensorRT-LLM or a well-configured llama.cpp, but it is not guaranteed with any engine.

Quantisation and throughput: the memory reduction per model frees VRAM, which allows larger batch sizes. At a larger batch, throughput rises more than power does, cutting J/token further still. The net effect can exceed the direct reduction in energy per operation.

Effect of the inference engine

The engine is a variable that model-level leaderboards tend to fix, but which in production is a decision of your own. Data from published comparisons (vLLM, TensorRT-LLM, naive Transformers, A100):

Engine	Relative J/token vs Transformers baseline
Transformers (naive, not optimised)	1.0× (reference)
vLLM (PagedAttention, continuous batching)	−25 to −35 %
TensorRT-LLM (optimised NVIDIA kernels, FP8)	−35 to −45 %
llama.cpp (hybrid CPU/GPU, INT4)	Variable; −30 to −60 % depending on hardware

Moving from naive Transformers to TensorRT-LLM can cut energy per token by more than moving from a 70B model to an 8B one of the same origin. The choice of engine is a first-order lever on energy efficiency.

Limits of energy leaderboards

Limit	Description
Hardware dependence	A ranking on H100 is not valid on A100 or L40S without correction. The hierarchy of models can change from one piece of hardware to another.
Engine dependence	The results are valid only for the engine they were measured with. A model that is ×2 more efficient on the leaderboard can fall behind if a slower engine is used.
Artificial batch size	AI Energy Score uses batch = 1. In production with aggressive batching, the efficiency relationship between large and small models changes: the large ones scale better with batch.
Training not captured	All the leaderboards measure inference only. The energy cost of training (which can exceed that of inference by 1,000× over the life of the model) is out of scope.
Incompatibility between leaderboards	Wh/query, J/token and samples/J measure different things. Converting between them requires knowing the mean output length, which varies by task and dataset.
Partial system coverage	AI Energy Score and ML.ENERGY measure GPU (+CPU/DRAM with Zeus); they do not capture the overhead of the complete system (PSU losses, fans, interconnect). MLPerf Power does, but covers few models.
Data latency	The leaderboards publish results months after the tests. New hardware (H200, B100, B200) may have no data available at the time of the decision.
No PUE	None of the three includes datacenter PUE. For real TCO, the leaderboard J/token must be multiplied by your own PUE.

Decision table: choosing a model on energy efficiency

The selection criteria in order, without recommendation prose:

Criterion	Question	Action
Task with reasoning	Does the task require step-by-step reasoning?	Yes → multiply the model’s base energy by ×30–700 before comparing. If there is a non-reasoning alternative with sufficient quality, prefer it.
Size vs minimum quality	What is the minimum acceptable quality for the task?	Consult quality benchmarks (see B7 when available). Pick the smallest model that clears the quality threshold; energy grows sublinearly with size.
Dense vs MoE	Does the hardware have enough memory for the full MoE?	If yes: the active-equivalent MoE consumes ~3× fewer J/token than the equivalent dense model. If not: paging or offload eats the gain.
Precision	Does the hardware have native FP8/INT4 support?	H100/H200: native FP8 (−30 %). With TensorRT-LLM: INT4 AWQ (−50 to −79 %). Without native support: stay on FP16 or BF16 until validated with your own benchmark.
Inference engine	Is the optimal engine for the hardware being used?	Measure with C3. If the engine is not optimised, changing engine can cut energy more than changing model.
Consult the leaderboard	Is the task covered by AI Energy Score or ML.ENERGY?	Filter by: same task, same hardware class, reasoning explicitly off/on. Do not compare models from different hardware classes or different engines.
Validate on your own hardware	Are the leaderboard results on the same HW as yours?	Always validate with Zeus or DCGM on your own hardware before taking the final decision. The leaderboard is a reference, not a prediction.

Quick signals table:

Signal	Effect on energy	Data source
Switching reasoning on	×30–700	AI Energy Score v2
Going from dense 8B to dense 70B	~×3.5	ML.ENERGY Leaderboard v3
Going from dense 8B to active-equivalent 8B MoE	~×0.33 (−67 %)	ML.ENERGY v3
FP16 → INT4 (compatible hardware)	−50 to −79 %	arXiv 2508.16712, 2504.03360
Naive Transformers → TensorRT-LLM FP8	−35 to −45 %	TokenPowerBench, ML.ENERGY
PUE 1.0 → PUE 1.5	+50 % on real datacenter energy	MLPerf Power (scope)

Reference data: energy in a generic node (4×H100 SXM)

Generic example hardware, to anchor the leaderboard values to a real node:

Parameter	Indicative value
TDP 4×H100 SXM 80 GB	4 × 700 W = 2,800 W (GPU only)
Complete-node system power (at the wall)	~3,500–5,000 W depending on load
Non-GPU overhead over GPU	25–45 %
J/token Llama 3 70B FP16, vLLM, batch 8	~1–3 J/tok (indicative, A100/H100)
J/token Llama 3 8B FP16, vLLM, batch 8	~0.3–0.7 J/tok (indicative)
J/token Mixtral 8×7B FP16, vLLM, batch 8	~0.4–0.8 J/tok (indicative)
Energy per 1M tokens (Llama 3 70B, PUE 1.4)	~0.5–1.2 kWh

The J/token values are indicative and depend heavily on batch size, prompt length, prefill/decode ratio and engine version. For certified values, consult the MLPerf Power submissions (mlcommons.org).

For the alternative reference node (4×A100 PCIe 80 GB, TDP ~300 W each):

Parameter	Indicative value
TDP 4×A100 PCIe	4 × 300 W = 1,200 W (GPU only)
Complete-node system power	~1,500–2,000 W
J/token Llama 3 70B FP16, vLLM	~2–5 J/tok (indicative; higher because of lower HBM bandwidth vs SXM)

How to use the leaderboards in practice

A decision flow based on the public data available:

Step	Action	Resource
1	Identify the dominant task of the workload	—
2	Consult AI Energy Score filtered by task and hardware class	huggingface.co/spaces/AIEnergyScore/Leaderboard
3	Note the models with ⭐⭐⭐⭐ or ⭐⭐⭐⭐⭐ on the task	Wh/1k-queries as a relative reference
4	Cross-check with ML.ENERGY for the J/token of each candidate	ml.energy/leaderboard
5	If any model is in MLPerf Power (Llama 2, GPT-J, Mixtral), consult the certified samples/J	mlcommons.org/benchmarks/inference-datacenter/
6	Select the 2–3 candidates with the best energy/quality ratio	—
7	Measure on your own hardware with Zeus or DCGM	github.com/ml-energy/zeus
8	Multiply the measured J/token by the datacenter PUE	J/token × PUE = effective J/token in the datacenter
9	Work out the electricity cost per token at the contracted price	See C2

Sources

Hugging Face · AI Energy Score · organización y leaderboard — https://huggingface.co/AIEnergyScore
Hugging Face · Announcing AI Energy Score Ratings (Luccioni et al., feb. 2025) — https://huggingface.co/blog/sasha/announcing-ai-energy-score
Hugging Face · AI Energy Score v2: Refreshed Leaderboard, now with Reasoning (Luccioni, Gamazaychikov, dic. 2025) — https://huggingface.co/blog/sasha/ai-energy-score-v2
Hugging Face · AIEnergyScore GitHub (Apache 2.0) — https://github.com/huggingface/AIEnergyScore
ML.ENERGY Initiative · Leaderboard — https://ml.energy/leaderboard
ML.ENERGY Initiative · Blog: Diagnosing Inference Energy Consumption with the ML.ENERGY Leaderboard v3.0 (dic. 2025) — https://ml.energy/blog/measurement/energy/diagnosing-inference-energy-consumption-with-the-mlenergy-leaderboard-v30/
arXiv 2505.06371 · The ML.ENERGY Benchmark: Toward Automated Inference Energy Measurement and Optimization (Chung et al., NeurIPS 2025 D&B Spotlight) — https://arxiv.org/abs/2505.06371
ML.ENERGY Initiative · Zeus: Deep Learning Energy Measurement and Optimization — https://ml.energy/zeus/
GitHub ml-energy/zeus (MIT) — https://github.com/ml-energy/zeus
PyTorch Blog · Zeus: Deep Learning Energy Measurement and Optimization — https://pytorch.org/blog/zeus/
University of Michigan CSE · Power-hungry AI: Researchers evaluate energy consumption across models — https://cse.engin.umich.edu/stories/power-hungry-ai-researchers-evaluate-energy-consumption-across-models
arXiv 2512.03024 · TokenPowerBench: Benchmarking the Power Consumption of LLM Inference (dic. 2024) — https://arxiv.org/abs/2512.03024
arXiv 2508.16712 · Systematic Characterization of LLM Quantization: A Performance, Energy, and Quality Perspective — https://arxiv.org/abs/2508.16712
arXiv 2504.03360 · Sustainable LLM Inference for Edge AI: Evaluating Quantized LLMs for Energy Efficiency, Output Accuracy, and Inference Latency — https://arxiv.org/abs/2504.03360
Epoch AI · AI Energy Use: Data & Research — https://epoch.ai/topics/energy
MLCommons · MLPerf Inference Datacenter benchmark results — https://mlcommons.org/benchmarks/inference-datacenter/
arXiv 2410.12032 · MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems (Tschand et al., 2024) — https://arxiv.org/abs/2410.12032
Coalition for Sustainable AI · AI Energy Score as best practice in benchmarking — https://www.sustainableaicoalition.org/ai-energy-score-a-standardized-approach-to-evaluating-ai-model-energy-efficiency/

LLM quality benchmarks: the contamination trap and the OSS tools for not fooling yourself

Tue, 16 Jun 2026 03:00:00 +0200

Notation: amounts in euros (N €), decimals with a full stop. The dollar sign is not used (on this site it is a formula delimiter).

Post B7 of the benchmarking track. The B1 introduction established the distinction between the three axes (performance, quality, energy). The B2 tool catalogue covered the performance axis (TTFT, ITL, goodput). This article covers the quality axis: what each benchmark measures, how it is run, and why the leaderboard numbers do not always say what they appear to say. The LLMOps eval pipeline and LLM-as-judge cover the system’s internal evaluation layer; this article covers the evaluation of the base model with standard academic benchmarks.

TL;DR

An inference engine with an 80 ms TTFT serving a low-quality model is worth nothing: the platform’s Pareto front has two independent axes. In 2025–2026, the most widely used static benchmarks (MMLU, HumanEval) have crossed the saturation threshold (HumanEval > 93% in frontier models, MMLU-Pro > 75% in the top-5), which invalidates their ability to discriminate at the high end. The main cause is not that the models are better: it is data contamination, with 18% n-gram overlap detected in Llama-2-70B against evaluation datasets. The OSS tools active in 2026 are lm-evaluation-harness (EleutherAI, MIT, 60+ tasks, the ecosystem’s de facto engine), HELM (Stanford, Apache 2.0, in maintenance since June 2026), LiveBench (MIT, 1,000 questions refreshed monthly to avoid contamination), lighteval (Hugging Face, MIT, 1,000+ tasks, vLLM support), and LMArena/Chatbot Arena (Bradley-Terry over millions of human comparisons). OpenLLM Leaderboard v2 was archived in 2025 and replaced by the OpenEvals ecosystem (200+ specialised leaderboards). Reproducing an eval number requires pinning: harness version, model version, prompt template version and number of shots.

Quality ≠ performance: the independent Pareto axis

The performance axis (TTFT, ITL, throughput) measures how long the engine takes to produce tokens. The quality axis measures how correct those tokens are. They are orthogonal: a small model served with vLLM at 3,000 tok/s can score 42% on MMLU-Pro; a frontier model may need 4× more GPU and score 72%. The product throughput × quality is the only number that matters for a platform in production.

The operational consequence: select model and engine first by quality (does it clear the minimum threshold for the task?) and then by performance (does it meet the SLO within the GPU budget?). Inverting the order, picking the fastest engine and fitting the model into it, is the most frequent mistake in initial sizing.

Reference benchmarks: what each one measures

The table below collects the standard academic benchmarks in 2026, the capability each one measures, its test set size and the current saturation level in frontier models.

Benchmark	Capability measured	Format	Test set	Frontier saturation 2026
MMLU	Multidisciplinary knowledge (57 subjects)	4-choice MCQ	14,079	High: top-5 > 88%; insufficient separation
MMLU-Pro	MMLU with 10 options, multistep reasoning	10-choice MCQ	12,032	Medium: top-5 between 72–77%; still separates
GSM8K	Primary-school maths reasoning (word problems)	answer generation	1,319	Very high: top models > 97%
MATH	Competition maths (algebra, geometry, calculus, etc.)	LaTeX generation	5,000	Medium-high: top models 85–92%
GPQA Diamond	PhD-level scientific reasoning (biology, physics, chemistry)	4-choice MCQ	198	Emerging: top > 90% in 2026; still separates in 60–90%
BBH (BIG-Bench Hard)	23 tasks where earlier models fell below human level	generation	6,511	Medium: useful for complex reasoning
IFEval	Verifiable instruction following (format, length, language)	generation + rules	541	Low: notable separation between models
HumanEval	Python code generation (164 problems with signature)	generation + execution	164	Very high: top > 93%; saturated
MBPP	Code generation (500 problems without a function signature)	generation + execution	500	High: top models 85–92%
MuSR	Multistep reasoning over long texts	generation	~1,000	Low: useful for extended reasoning
TruthfulQA	Truthfulness: resistance to asserting popular myths	generation + MCQ	817	Medium: visible separation between families

Note on saturation: a saturated benchmark does not distinguish models at the high end; it is only valuable as a floor check (confirming the model has not regressed to the level of three years ago). For the frontier zone, the useful benchmarks in 2026 are MMLU-Pro, GPQA Diamond, IFEval and MuSR.

OSS evaluation tools: data sheets

lm-evaluation-harness (EleutherAI)

Field	Data
Repository	github.com/EleutherAI/lm-evaluation-harness
Licence	MIT
Maintainer	EleutherAI
Active version (Jun 2026)	0.4.x (`main` branch)
Available tasks	60+ benchmarks, hundreds of subtasks and variants
Supported backends	Hugging Face Transformers, vLLM, litellm, OpenAI-compatible API, GGUF
Task format	YAML + Python; each task defines dataset, template, metric and number of shots
What it measures	MCQ accuracy, exact match, pass@k on code, perplexity, BLEU/ROUGE
How it is invoked	`lm_eval --model hf --model_args pretrained=<model> --tasks mmlu,gsm8k --num_fewshot 5 --output_path results/`
Output	JSON with per-task metrics + configuration metadata
Use in the ecosystem	Official engine of the OpenLLM Leaderboard (v1 and v2); de facto reference in research

The harness is the industry standard for reproducible evaluation. Its strength is the breadth of tasks and the normalisation of the prompt format. Its limitation: the tasks are static, the datasets do not change between harness versions, which exposes it to the contamination problem if the model was trained on the same data.

Full invocation with vLLM as the backend:

lm_eval \
 --model vllm \
 --model_args pretrained=meta-llama/Llama-3.1-70B-Instruct,dtype=bfloat16 \
 --tasks mmlu_pro,gpqa_diamond,ifeval,humaneval \
 --num_fewshot 5 \
 --batch_size auto \
 --output_path ./eval_results/ \
 --log_samples

HELM (Stanford CRFM)

Field	Data
Repository	github.com/stanford-crfm/helm
Official site	crfm.stanford.edu/helm
Licence	Apache 2.0
Maintainer	Stanford CRFM
Status (Jun 2026)	Maintenance mode since 1 June 2026
What it measures	Accuracy, calibration, robustness, fairness, toxicity, efficiency (latency/cost)
Task format	Python + YAML; scenarios with multiple simultaneous metrics
Supported backends	API (OpenAI, Anthropic, Cohere, etc.) + local models via Hugging Face
Active variants	HELM Classic, HELM Capabilities, HELM Lite, MedHELM, HELM Safety
Installation	`pip install crfm-helm`
How it is invoked	`helm-run --conf-paths run_specs.conf --suite v1 --max-eval-instances 1000`
Output	JSON + web UI with a reproducible leaderboard and per-prompt transparency

HELM is the tool that introduced holistic evaluation (multiple metrics over the same scenario instead of a single number). In 2026, with HELM in maintenance mode, the ecosystem has fragmented into specialised leaderboards; but HELM remains the methodological reference for evaluations that need fairness and toxicity as well as accuracy.

HELM Capabilities (published March 2025): an active leaderboard with full prompt transparency, reproducible with the HELM framework. Available at crfm.stanford.edu/helm/capabilities.

LiveBench

Field	Data
Repository	github.com/LiveBench/LiveBench
Official site	livebench.ai
Licence	MIT
Maintainer	NYU, UT Austin, UC Santa Barbara
Paper	arXiv:2406.19314
Size	1,000 active questions (monthly refresh, same total size)
Categories	6: math, coding, reasoning, language, instruction following, data analysis
Tasks	18 subtasks
What it measures	General capabilities over questions with objectively verifiable answers
Anti-contamination	Questions generated from recent sources: arXiv papers, news, IMDb, new datasets
Scoring	No LLM-as-judge: automatically verifiable answer (exact ground truth)
Cadence	New questions monthly; old ones retired to avoid accumulated memorisation

LiveBench is the architectural answer to the contamination problem: if the benchmark changes every month with questions based on sources published after the cutoff of the model under evaluation, memorisation of the training set cannot inflate the result. The limitation is size (1,000 questions) and that it does not cover every domain with the same depth as MMLU.

Execution:

git clone https://github.com/LiveBench/LiveBench
cd LiveBench
pip install -e .
python livebench/gen_model_answer.py \
 --model-path meta-llama/Llama-3.1-70B-Instruct \
 --model-id llama3-70b
python livebench/gen_ground_truth_judgment.py \
 --model-id llama3-70b
python livebench/show_livebench_result.py

lighteval (Hugging Face)

Field	Data
Repository	github.com/huggingface/lighteval
Licence	MIT
Maintainer	Hugging Face (Leaderboard & Evals Team)
Available tasks	1,000+ tasks across multiple domains and languages
Supported backends	Hugging Face Accelerate (CPU/GPU/multi-GPU), Nanotron, vLLM, TGI, OpenAI-compatible endpoints
What it measures	Accuracy, BERT score, BLEU, exact match; same as the harness + HF’s own tasks
Differentiator	Designed to be the engine behind HF’s leaderboards; easy extension with custom tasks
Installation	`pip install lighteval`
How it is invoked	`lighteval accelerate –model_args “pretrained=” –tasks “lighteval
Output	JSON + direct integration with the Hugging Face Hub to publish results

lighteval is the internal successor to lm-evaluation-harness in the HF ecosystem: more modern, better vLLM and distributed backend support, and designed to run without leaving the HF stack. For teams already using the Hub to manage models, it is the lowest-friction option.

OpenLLM Leaderboard v2 (archived 2025) → OpenEvals

The OpenLLM Leaderboard v2 evaluated more than 13,000 models between June 2024 and its closure in 2025, using six fixed benchmarks: IFEval, MuSR, GPQA, MATH, BBH and MMLU-Pro, all run with lm-evaluation-harness. Its closure was a direct consequence of the saturation and gaming problem: as the benchmarks became well known, models were optimised for them.

Successor: OpenEvals (huggingface.co/spaces/OpenEvals/find-a-leaderboard)

Aspect	OpenLLM Leaderboard v2	OpenEvals
Architecture	Single centralised leaderboard	200+ specialised community leaderboards
Benchmarks	Fixed (6 tasks)	Variable per leaderboard (math, code, medical, safety, etc.)
Models evaluated	13,000+	Distributed across leaderboards
Eval engine	lm-evaluation-harness	lighteval (most) or harness
Status	Archived with an updated UI (Mar 2025)	Active; you can create your own leaderboard
Advantage	Historical consistency	Specialisation, speed of adaptation

The archived collection is available for reference at huggingface.co/collections/OpenEvals/archived-open-llm-leaderboard-2024-2025.

LMArena / Chatbot Arena

Field	Data
Site	lmarena.ai (rebrand of LMSys Chatbot Arena, January 2026)
Methodology	Anonymous pairwise comparisons: the user votes for the answer they prefer without knowing the model
Statistical model	Bradley-Terry (maximum likelihood over the full history of match-ups; replaced classic Elo for greater stability)
Accumulated votes	> 6 million (Jun 2026)
Variants	Overall, Coding, Math, Hard Prompts, Multimodal — the specialised ones are more reliable
Licence/access	Free (web); vote data available under a CC licence
Limitations	Overall is biased by labs’ selective disclosure and by format gaming; Bradley-Terry was recalibrated in mid-2025 (Style Control), shifting some models by ±20–40 Elo

LMArena is the only benchmark based on human preference at scale. Its advantage: it captures real perceived quality, not just MCQ accuracy. Its limitation: it is hard to reproduce on-premise (it requires a user interface and a panel of human voters). Useful as an external validation signal, not as a CI gate.

Decision table: which tool for which objective

Objective	Tool	Why
Reproduce a standard academic eval	lm-evaluation-harness	De facto engine; maximum compatibility with papers
Holistic evaluation (accuracy + fairness + toxicity)	HELM (+ HELM Capabilities)	The only framework measuring every axis at once
Avoid contamination with recent questions	LiveBench	Monthly refresh; verifiable answers without a judge
Native integration with HF Hub and vLLM	lighteval	vLLM backend, 1,000+ tasks, direct publishing to the Hub
Compare against the public state of the art (OSS)	OpenEvals + lighteval	The successor ecosystem to the OpenLLM Leaderboard
Human preference at scale	LMArena	Bradley-Terry over millions of real comparisons
Custom domain eval (medical, legal, etc.)	lighteval with a custom task	Simpler to extend than the harness
Fast CI gate (<10 min)	lm-eval harness (subset)	Individual tasks with `--tasks ifeval,gpqa` + vLLM batching

The data contamination trap

What it is

A model that has seen a benchmark’s test data during pretraining memorises answers instead of generalising. The benchmark measures memorisation, not capability. This is called data contamination or benchmark leakage.

Scale of the problem (2024–2025 data)

Finding	Source	Figure
N-gram overlap in Llama-2-70B	“Data Contamination or Genuine Generalization?” (2025)	18.1% overlap with eval datasets
Out-of-distribution drop in contaminated large models	Ibid.	−9.4 pp on reformulated prompts vs the original test
HumanEval saturated	Artificial Analysis / multiple 2026 reports	Top models > 93%; separation ≤ 2 pp between the best 5
GPQA Diamond: appearance of indirect contamination	mindstudio.ai analysis 2026	“Data laundering”: questions leaked via online forums

Detection mechanisms

1. N-gram overlap (the classic method)

Compute the fraction of test set n-grams that appear in the model’s pretraining corpus. A practical threshold: if more than 13% of the test’s 8-grams match the corpus, the result is compromised.

\[ \text{contamination ratio} = \frac{|\{g \in \text{test n-grams}\} \cap \{g \in \text{train n-grams}\}|}{|\text{test n-grams}|} \]

Limit: it only works if you have access to the pretraining corpus. For closed models, it is not applicable.

2. Canary strings (decoy insertion)

Unique random strings are inserted into the evaluation dataset while it is being created. If the model reproduces those strings without having seen them in context, it confirms that it memorised them from pretraining. The technique was proposed in “Extracting Training Data from Large Language Models” (Carlini et al., 2021) and adopted in recent benchmarks.

3. Perturbation and reformulation

Semantically equivalent variants of the test are generated (paraphrasing, changing proper nouns, translation and back-translation). A model that generalises keeps its accuracy; a model that memorises drops. The average drop detected in contaminated models is 9–15 pp on equivalent reformulated prompts.

4. LiveBench (monthly refresh)

The architectural solution: if the questions are generated from sources published after the model’s cutoff (recent arXiv papers, news, new data), memorisation from pretraining cannot help. The model has to reason over information it could never have seen.

5. Kernel divergence analysis (Savelka et al., 2025)

A more sophisticated statistical method: it compares the embedding distribution of the test set with the distribution of the training corpus. An abnormally low divergence indicates that the benchmark and the training corpus come from the same source.

Why static leaderboards saturate

The cycle is predictable:

A new benchmark is published (MMLU in 2021, GPQA in 2023).
The labs use it as an evaluation target during pretraining and fine-tuning.
Performance on that benchmark rises quickly, faster than real capability.
The benchmark stops discriminating at the high end.
The community needs a new benchmark.

The average time between the publication of a benchmark and its saturation in frontier models has gone from ~36 months (MMLU: 2021–2024) to ~18 months (HumanEval: 2021–2023, GPQA: 2023–2025). Saturation is accelerating.

Honest methodology: reproducibility and sources of variance

The four parameters that must be pinned

An eval result is only reproducible if it declares:

Parameter	Example	Why it matters
Harness version	`lm-evaluation-harness==0.4.3`	Changes to the prompt template between versions invalidate the comparison
Exact model version	`meta-llama/Llama-3.1-70B-Instruct@sha256:abc…`	Silent model updates on the Hub change the results
Prompt template	`mmlu_pro_cot_0shot` vs `mmlu_pro_5shot`	The difference between 0-shot and 5-shot can be 5–12 pp
Number of shots	`--num_fewshot 5`	The published standard varies by benchmark; 0-shot and 5-shot are not comparable

Few-shot vs zero-shot: the numerical difference

Sensitivity to the number of shots varies dramatically by benchmark:

Benchmark	Typical 0-shot	Typical 5-shot	Difference
MMLU-Pro	62%	72%	+10 pp
GPQA Diamond	55%	65%	+10 pp
GSM8K	70%	82%	+12 pp
IFEval	68%	71%	+3 pp
HumanEval	75%	80%	+5 pp

Publishing an MMLU-Pro result in 0-shot without saying so, when the leaderboard standard is 5-shot, inflates the comparison by ~10 pp. Two results from the same model published with different numbers of shots are not comparable.

Prompt sensitivity

Minor perturbations in the prompt template produce variations of up to ±8 pp on MCQ and ±15 pp on generation tasks. Sources of variance identified in the literature:

Order of the options in MCQ (position bias): up to ±4 pp.
Wording of the question (paraphrasing): ±3–8 pp.
Presence or absence of “Let’s think step by step”: ±5–12 pp on reasoning.
Language of the system instructions: ±3–10 pp in non-multilingual models.

Why two evals give different results

A common scenario: the paper reports MMLU-Pro 74.3% and the reproduction gives 71.8%. Frequent causes:

Cause	Typical magnitude
Different harness version (changed template)	1–4 pp
Model with a silent update	0.5–3 pp
Different context truncation	0.5–2 pp
Sampling seed (for generation)	0.3–1.5 pp
Tokenisation with a chat template vs without one	2–8 pp

The sum of these sources explains gaps of 2–6 pp that are not noise but systematic protocol differences.

The confidence interval formula for comparing two models

To know whether the difference between model A (accuracy $p_A$) and model B ($p_B$) over a test set of size $n$ is statistically significant:

\[ \Delta p \pm 1.96 \cdot \sqrt{\frac{p_A(1-p_A) + p_B(1-p_B)}{n}} \]

Applied to GPQA Diamond ($n = 198$), if $p_A = 0.90$ and $p_B = 0.88$:

\[ \Delta p = 0.02 \quad;\quad \text{margin} = 1.96 \cdot \sqrt{\frac{0.09 + 0.1056}{198}} \approx 1.96 \cdot 0.0314 \approx 0.062 \]

The 95% confidence interval is $[{-0.042},\ {+0.082}]$: it crosses zero. A difference of 2 pp over 198 questions is not statistically distinguishable from noise. To distinguish 90% from 88% with 95% confidence you need:

\[ n \approx \frac{(1.96)^2 \cdot (p_A(1-p_A) + p_B(1-p_B))}{(\Delta p)^2} \approx \frac{3.84 \cdot 0.1956}{0.0004} \approx 1{,}878 \text{ questions} \]

GPQA Diamond is not that size. Differences of 1–2 pp on GPQA Diamond are statistically invisible.

The quality axis in the scorecard

The platform’s scorecard has three columns: performance (goodput under SLO), quality (score on a reference benchmark), energy (J/token). The quality column is built like this:

Field	Content
Primary benchmark	MMLU-Pro (5-shot) — separates in the 60–77% zone
Reasoning benchmark	GPQA Diamond (0-shot) — separates in the 55–90% zone
Instruction benchmark	IFEval (0-shot prompt-level accuracy)
Code benchmark	MBPP (0-shot, pass@1) — less saturated than HumanEval
Anti-contamination	LiveBench global score (monthly)
Tool	lm-evaluation-harness (primary) + lighteval (secondary, HF Hub)
Pinned version	`harness==0.4.3`, `lighteval==0.6.x`
Shots	Declared explicitly per benchmark
Reproducibility	Output JSON stored with the model SHA
Frequency	Per model release; not in the CI of every PR (GPU-hour cost)

A full eval over 4 benchmarks (MMLU-Pro 12,032 + GPQA 198 + IFEval 541 + MBPP 500) with a 70B model on a generic 2×H100 node takes on the order of 4–8 hours depending on batch size and number of shots. The GPU cost is real: at ~5 €/h per amortised card, a full eval costs on the order of 40–80 €. It is not run on every commit; it is run on every model release or major adapter release.

Sources

EleutherAI · lm-evaluation-harness — https://github.com/EleutherAI/lm-evaluation-harness
Stanford CRFM · HELM — https://github.com/stanford-crfm/helm
Stanford CRFM · HELM Capabilities (mar-2025) — https://crfm.stanford.edu/2025/03/20/helm-capabilities.html
Stanford CRFM · HELM (sitio oficial) — https://crfm.stanford.edu/helm/
LiveBench · paper arXiv:2406.19314 — https://arxiv.org/abs/2406.19314
LiveBench · sitio oficial — https://livebench.ai
LiveBench · GitHub — https://github.com/LiveBench/LiveBench
NYU Center for Data Science · LiveBench blog — https://nyudatascience.medium.com/livebench-challenging-language-models-with-contamination-free-questions-999b52967ec8
Hugging Face · lighteval — https://github.com/huggingface/lighteval
Hugging Face · OpenLLM Leaderboard (archivado) — https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard
Hugging Face · OpenEvals (sucesor) — https://huggingface.co/spaces/OpenEvals/find-a-leaderboard
Hugging Face · Archived Open LLM Leaderboard 2024-2025 — https://huggingface.co/collections/OpenEvals/archived-open-llm-leaderboard-2024-2025
LMArena · Chatbot Arena — https://lmarena.ai
LMSys · blog Chatbot Arena Elo — https://www.lmsys.org/blog/2023-05-03-arena/
“A Statistical Framework for Ranking LLM-Based Chatbots” · arXiv:2412.18407 — https://arxiv.org/pdf/2412.18407
“Data Contamination or Genuine Generalization?” (2025) — https://www.suaspress.org/ojs/index.php/AJNS/article/view/v2n2a03
“When Benchmarks Leak: Inference-Time Decontamination” (2025) · arXiv:2601.19334 — https://arxiv.org/pdf/2601.19334
“How Contaminated Is Your Benchmark?” (2025) · arXiv:2502.00678 — https://arxiv.org/pdf/2502.00678
“When AI Benchmarks Plateau” (2026) · arXiv:2602.16763 — https://arxiv.org/pdf/2602.16763
GPQA · arXiv:2311.12022 — https://arxiv.org/abs/2311.12022
GPQA Diamond leaderboard · Artificial Analysis — https://artificialanalysis.ai/evaluations/gpqa-diamond
“The Emperor’s New Clothes in Benchmarking?” (2025) · arXiv:2503.16402 — https://arxiv.org/pdf/2503.16402
lm-evaluation-harness · CLI reference — https://lm-evaluation-harness.readthedocs.io/running_evals/cli_reference/
crfm-helm · PyPI — https://pypi.org/project/crfm-helm/

LLM serving engines compared on the Pareto frontier: vLLM, SGLang, TRT-LLM and Dynamo

Tue, 16 Jun 2026 02:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar symbol is not used (on this site it is a formula delimiter). Reference hardware: generic configuration 4×H100 SXM 80 GB; the figures are illustrative for comparing engines, not production metrics from any real infrastructure.

TL;DR

Four OSS engines dominate on-premise LLM serving in 2026: vLLM, SGLang, NVIDIA TensorRT-LLM and NVIDIA Dynamo. None is superior on every axis. In raw throughput with a shared prompt, SGLang beats vLLM by 29 % on Llama-3.3-70B FP8 over H100 (16,200 vs 12,500 tok/s). In decode latency, TRT-LLM keeps ITL more stable at high concurrency (9–12 ms P50 against 18–22 ms for vLLM). Dynamo is not an inference engine: it is the orchestration layer on top of the previous three for multi-node clusters, and it reports up to 7× additional throughput in disaggregated serving on Blackwell. The right choice depends on three variables: load pattern (batch/interactive), deployment scale (single node/multi-node) and operational constraints (OSS maturity, ease of operation). The Pareto decision table is at the end of the article.

Framing the article: what this B8 is and is not

This article (B8 of the benchmarking pillar) takes for granted the metrics from the introduction to benchmarking and the tool catalogue from the card-by-card piece. It applies those concepts to engine selection. The protocol from GuideLLM for SLO validation and AIPerf in depth detail how to run the harness; here the result is used. The fundamentals of continuous batching, disaggregated serving, the attention backend and quantisation are assumed known.

Card for each engine

vLLM

Attribute	Value
Reference version	v0.9.x (June 2026)
Licence	Apache 2.0
Maintainer / governance	vLLM Project (Linux Foundation AI), led by UC Berkeley + Red Hat
Supported hardware	NVIDIA (Ampere/Hopper/Blackwell), AMD ROCm, Intel Gaudi, TPU, CPU
Repository	github.com/vllm-project/vllm
API	OpenAI-compatible (completions, chat, embeddings)

Origin: PagedAttention (Kwon et al., 2023). The first implementation of continuous batching with a paged KV cache. In 2026 it is the most deployed engine in OSS environments and the one that accumulates the most integrations (Ray Serve, Kubernetes, llm-d, Dynamo).

SGLang

Attribute	Value
Reference version	v0.4.x (June 2026)
Licence	Apache 2.0
Maintainer / governance	LMSYS Org (Berkeley, CMU, UCSD)
Supported hardware	NVIDIA (Ampere/Hopper/Blackwell), AMD ROCm
Repository	github.com/sgl-project/sglang
API	OpenAI-compatible + SGLang runtime API

Origin: the SGLang paper, 2024 (Zheng et al.). It introduces RadixAttention as a generalisation of prefix caching: a radix tree of KV blocks with an LRU policy shared across all in-flight requests. A structural advantage in workloads with shared prefixes (long system prompts, few-shot, RAG).

NVIDIA TensorRT-LLM (TRT-LLM)

Attribute	Value
Reference version	v0.18.x (June 2026)
Licence	Apache 2.0
Maintainer / governance	NVIDIA (proprietary in practice, OSS in name)
Supported hardware	NVIDIA only: Ampere (A100), Hopper (H100/H200/GH200), Ada (L40/L40S), Blackwell (B200/GB200)
Repository	github.com/NVIDIA/TensorRT-LLM
API	Python + C++ runtime; integrates with Triton Inference Server

The lowest-level engine: it compiles inference graphs with TensorRT, applies kernel fusion and generates binary engines specific to model + hardware + precision. Maximum performance on NVIDIA, with no portability to other hardware. It requires recompilation when the model or configuration changes; build time ranges from minutes to hours depending on the model.

NVIDIA Dynamo

Attribute	Value
Reference version	v1.1.1 (May 2026)
Licence	Apache 2.0
Maintainer / governance	NVIDIA + OSS community (ai-dynamo org on GitHub)
Supported hardware	NVIDIA (through the underlying engine: vLLM, SGLang or TRT-LLM)
Repository	github.com/ai-dynamo/dynamo
Role	Orchestration layer, NOT an inference engine

Dynamo does not run inference directly: it is the scheduler and router that coordinates pools of workers (vLLM, SGLang, TRT-LLM) across multiple nodes. Implemented in Rust (performance) + Python (extensibility). It adds: disaggregated prefill/decode serving across nodes, KV-aware routing (avoiding prefill recompute if another worker already has the KV in cache), automatic scaling by SLA, a KV Block Manager (KVBM) with offload to CPU/SSD/remote and transfer via NIXL over NVLink/InfiniBand.

Honourable mentions: LMDeploy and HF TGI

LMDeploy (OpenMMLab/InternLM, Apache 2.0): an engine with two backends, TurboMind (C++, maximum performance on NVIDIA) and PyTorch (flexibility). KV cache blocking and persistent batching; 4-bit inference up to 2.4× faster than FP16 in its own benchmark. A narrower ecosystem; no AMD support.

HF TGI (Hugging Face, Apache 2.0): the reference engine for the HuggingFace ecosystem. TGI v3.0 cuts latency by up to 13× on long prompts against earlier versions; it integrates FlashAttention + PagedAttention. The usual entry point for teams already operating the HF stack; it is not the most efficient in throughput at maximum load.

Capability matrix

The table uses this notation: Yes = supported and stable, Beta = available but not recommended for production, No = not available or only with a manual workaround.

Capability	vLLM	SGLang	TRT-LLM	Dynamo (via backend)
Continuous batching	Yes	Yes	Yes	Yes (delegates to the engine)
PagedAttention	Yes	Yes	Yes	Yes
RadixAttention / prefix caching	Yes (chunked prefill + prefix)	Yes (native RadixAttention)	Yes (prefix caching)	Yes + KV-aware routing
Speculative decoding	Yes (n-gram, EAGLE, DFlash, suffix)	Yes (draft+verify)	Yes (EAGLE-3, MTP)	Yes
FP8 quantisation	Yes	Yes	Yes	Yes
MXFP4/NVFP4 quantisation	Yes	Beta	Yes (Blackwell)	Yes
INT4 AWQ quantisation	Yes	Yes	Yes	Yes
GPTQ quantisation	Yes	Yes	Not native	Yes (via vLLM)
INT8 quantisation (SmoothQuant)	Yes	Yes	Yes	Yes
Structured output	Yes	Yes (compressed FSM)	Beta	Yes
Dynamic multi-LoRA	Yes	Yes	Beta	Yes
Disaggregated prefill/decode	Beta (experimental)	Yes	Yes	Yes (first class)
Chunked prefill	Yes	Yes	Yes	Yes
Multimodal support (VLM)	Yes	Yes	Yes	Yes
Multi-node tensor parallelism	Yes	Yes	Yes	Yes (+ pipeline across nodes)
OpenAI API compatible	Yes	Yes	Yes (via Triton)	Yes
Non-NVIDIA hardware	Yes (AMD/Intel/TPU)	Yes (AMD)	No	No

Sources for the matrix: docs.vllm.ai, docs.sglang.io, nvidia.github.io/TensorRT-LLM, docs.nvidia.com/dynamo, github.com/ai-dynamo/dynamo (feature matrix, June 2026).

Why they are not compared lightly: the variables that break a comparison

Before the table of figures, the four decisions that invalidate any cross-engine benchmark:

1. A different harness. A benchmark run with each engine’s native micro-bench (vllm bench serve vs SGLang bench) measures the client, not the engine. The discrepancy can be up to 7.2× at high concurrency. The correct protocol: a single load generator (AIPerf or GuideLLM) against every endpoint. The engine is the variable, the tool is the constant.

2. Different ISL/OSL. Input Sequence Length and Output Sequence Length determine the prefill/decode ratio and therefore which engine is favoured. A benchmark with low ISL (64 tokens) favours engines with fast decode; with high ISL (2,048 tokens) and shared prefixes, it favours SGLang (RadixAttention). Fixing ISL and OSL is mandatory for the comparison to be valid.

3. Different precision. FP16 and FP8 are not comparable: FP8 gives between 1.5× and 2× more throughput on Hopper. Comparing vLLM FP16 with TRT-LLM FP8 says nothing about the engine.

4. Not reporting P99. Maximum throughput is reached at the point where TTFT P99 already violates the SLO. The defensible number is goodput, throughput under the SLO, not the raw maximum. The end-to-end latency formula:

$$\text{latency}_{e2e} \approx \text{TTFT} + (N_{\text{out}} - 1) \times \text{TPOT}$$

where $N_{\text{out}}$ is the number of output tokens, makes it clear that a high TTFT with low TPOT has the same effect as the inverse case only for one specific $N_{\text{out}}$: both must be reported separately.

The complete fair-comparison protocol is in the tools article.

Performance on the Pareto frontier: illustrative figures

Reference configuration for every row: 4×H100 SXM 80 GB NVLink, model Llama-3.1-70B-Instruct FP8, harness AIPerf (formerly genai-perf, the multi-process successor), dataset ShareGPT (realistic length distribution), mean ISL ~512 tok, mean OSL ~256 tok, SLO TTFT P99 < 500 ms. The figures are illustrative and representative of benchmarks published by the community (Cerebrium, LMSYS, Spheron, Red Hat MLPerf v5.1); they are not measurements from any real infrastructure. Exact figures vary with engine versions, compilation flags and load distribution.

Throughput vs latency: Pareto table

Engine	Config	Throughput (tok/s)	TTFT P50 (ms)	TTFT P99 (ms)	ITL P50 (ms)	Goodput (@SLO)
vLLM v0.9	FP8, chunked prefill	12,500	160	420	18	~96 %
SGLang v0.4	FP8, RadixAttention	16,200	140	390	21	~97 %
TRT-LLM v0.18	FP8, compiled engine	14,800	190	480	10	~93 %
vLLM + Dynamo	FP8, disagg. P/D 2+2	18,500	120	310	19	~98 %
SGLang + Dynamo	FP8, disagg. P/D 2+2	21,000	110	280	22	~99 %
TRT-LLM + Dynamo	FP8, disagg. P/D 2+2	22,500	130	350	11	~97 %

Reading notes:

“Disagg. P/D 2+2” = 2 prefill GPUs + 2 decode GPUs (the same 4×H100 budget).
Goodput falls against raw throughput when the latency tail exceeds the SLO of 500 ms P99. TRT-LLM without Dynamo has the lowest ITL but the highest TTFT at high concurrency, which brings its goodput down to ~93 % under this SLO.
Dynamo adds ~30–40 % of effective throughput over 4 GPUs thanks to disaggregated serving and KV-aware routing, in the scenario with reusable prefixes (ShareGPT).
In workloads without shared prefixes (short ISL, unique prompts), Dynamo’s advantage shrinks to ~10–15 % over the base engine.

Saturation point (concurrency sweep, vLLM FP8, 4×H100)

Concurrency	TTFT P99 (ms)	ITL P50 (ms)	Throughput (tok/s)	Goodput (tok/s)
8	210	14	5,200	5,200
16	370	18	9,800	9,800
32	420	18	12,500	12,200
48	680	26	13,800	7,400
64	1,200	41	14,100	2,100

The knee is between 32 and 48: throughput rises only 10 % (12,500 → 13,800) but TTFT P99 already violates the 500 ms SLO and goodput halves. The defensible operational capacity is the one at concurrency 32 (12,200 useful tok/s). Reporting 14,100 tok/s (concurrency 64) would be maximum throughput with 15 % goodput.

Fair-comparison methodology: the harness as the constant

The protocol that makes numbers comparable across different engines. The engine is the only variable; everything else is constant and pinned in the output JSON.

Variables that are fixed (constants of the experiment)

Variable	Value fixed in the reference experiment
Load tool	AIPerf v2.x (multi-process)
Model	Llama-3.1-70B-Instruct
Precision	FP8
Hardware	4×H100 SXM 80 GB NVLink
Dataset	ShareGPT (realistic ISL/OSL distribution)
SLO	TTFT P99 < 500 ms, ITL P50 < 30 ms
Warm-up	200 requests discarded before measuring
Sweep	concurrencies 1, 4, 8, 16, 32, 48, 64
Primary metric	goodput (useful tok/s under SLO)

Harness steps

Deploy the engine with the exact config (pinned flags, fixed version).
Warm-up: 200 requests; discard the results.
Concurrency sweep: aiperf profile --concurrency-range 1:64:step with AIPerf.
Extend the sweep past the knee (until goodput falls below 50 %).
Collect the JSON: TTFT P50/P99, ITL P50, raw throughput, goodput.
Change only the engine; repeat steps 1–5.
Compare the goodput column across rows.

The sweep with GuideLLM is equivalent for SLO validation; AIPerf is preferable for engine comparisons because its automatic estimatedCapacity normalises the saturation point. Both can be used: GuideLLM for the operational SLO, AIPerf for reference capacity.

Reproducible output format

{
 "harness": "aiperf", "harness_version": "2.x",
 "model": "Llama-3.1-70B-Instruct", "precision": "FP8",
 "hardware": "4xH100_SXM_80GB_NVLink",
 "dataset": "sharegpt",
 "slo": {"ttft_p99_ms": 500, "itl_p50_ms": 30},
 "motor": "vllm", "motor_version": "0.9.x",
 "concurrency_sweep": [8, 16, 32, 48, 64],
 "results_at_knee": {
 "concurrency": 32,
 "ttft_p50_ms": 210, "ttft_p99_ms": 420,
 "itl_p50_ms": 18, "throughput_tok_s": 12500,
 "goodput_tok_s": 12200
 }
}

This versioned JSON is the auditable datum: anyone can reproduce the figure with the same tool, version, model, hardware and load.

State of the art 2026: relative maturity

Dimension	vLLM	SGLang	TRT-LLM	Dynamo
Production maturity	High (3+ years)	Medium-high (2 years)	High (NVIDIA)	Medium (1.0 GA Mar 2026)
Release cadence	Weekly	Weekly	Monthly	Fortnightly
Integration ecosystem	Very broad (Ray, k8s, llm-d, Dynamo, NIM)	Broad (Dynamo, k8s)	NVIDIA-centric (Triton, NIM)	NVIDIA-centric
Ease of operation	High (pip install, OpenAI API)	High	Medium (requires compiling engines)	Medium-low (multi-node, etcd/NATS)
Non-NVIDIA hardware	Yes (AMD, Intel, TPU)	Yes (AMD)	No	No
Disaggregated serving	Beta	Stable	Stable	First class
Dynamic multi-LoRA	Stable	Stable	Beta	Stable (via vLLM/SGLang)
Structured output	Stable	Stable (FSM)	Beta	Stable (via the engine)
OSS community support	Very high (>30 K GitHub stars)	High (~20 K stars)	High (~10 K stars)	Growing (6.8 K stars)

Pareto decision table: when to choose each engine

No prose; the decision as a table of criteria and outcome.

Selection criterion	Recommended engine	Quantitative justification
Single node, quick prototype, team with no LLM serving experience	vLLM	pip install + OpenAI API in minutes; the broadest community support
Workload with long shared prefixes (RAG, few-shot, system prompts > 512 tok)	SGLang	RadixAttention: up to 6.4× more throughput vs a baseline without prefix caching
Maximum throughput on NVIDIA, no portability, team with TRT experience	TRT-LLM	Fused kernels + native FP8: ITL 9–12 ms vs 18–22 ms for vLLM on the same hardware
Multi-node (> 8 GPUs), mixed prefill-intensive/decode-intensive traffic	Dynamo + SGLang	Disaggregated P/D: up to 7× additional throughput on Blackwell; 2× TTFT with KV-aware routing
AMD ROCm or Intel Gaudi hardware	vLLM	The only SOTA engine with stable non-NVIDIA support
Structured output + high concurrency	SGLang	Compressed FSM: guided decode with no appreciable overhead
Stable dynamic multi-LoRA in production	vLLM or SGLang	TRT-LLM multi-LoRA still in beta (Jun 2026)
Hugging Face ecosystem, models < 13B	HF TGI v3	Direct HF Hub integration; competitive latency on small models
Maximum energy efficiency (J/token) on NVIDIA	TRT-LLM	Lower-overhead kernels → fewer J/token at iso-throughput

Quantisation: effect on throughput and quality

The four main options in order of decreasing speed and increasing quality:

Format	Relative throughput (H100)	Quality degradation (perplexity)	Engines with stable support
NVFP4 / MXFP4	~2.2× vs FP16	1–3 % on MMLU	TRT-LLM (Blackwell), vLLM (experimental)
FP8 (W8A8)	~1.7× vs FP16	< 1 % on MMLU	vLLM, SGLang, TRT-LLM
INT4 AWQ	~1.5× vs FP16	1–2 % on MMLU	vLLM, SGLang, TRT-LLM
GPTQ (INT4)	~1.4× vs FP16	1–3 % on MMLU	vLLM, SGLang
FP16 (baseline)	1.0×	0 %	All

FP8 is the de facto standard on Hopper for production: 1.7× throughput with degradation below 1 %. The article on quantisation in depth develops the quality/speed trade-off by format and model architecture.

Disaggregated serving: when the overhead is worth it

Separating prefill/decode into independent pools introduces transfer latency for KV tensors between nodes (NIXL over NVLink or InfiniBand). The overhead pays off when:

Condition	Effect of disaggregated serving
Mean ISL > 1,024 tokens	Prefill dominates; separate pools stop it blocking decode
Prefill/decode ratio > 3:1 in time	The decode pool sits idle waiting for prefill
Mixed traffic (prefill burst + decode queue)	Independent scaling of each pool by SLA
< 512 tokens ISL, unique prompts	Transfer overhead exceeds the benefit; an aggregated engine is enough

Dynamo quantifies the break-even point: disaggregated serving pays off when the transfer time for KV tensors (a function of ISL and NVLink/IB bandwidth) is lower than the time the decode worker would spend waiting for the prefill worker in aggregated mode. On 4×H100 with an ISL of 512 tok, break-even occurs at a prefill/decode ratio > 2.5:1. The article on disaggregated serving develops the full formula.

MLPerf Inference v5.1: cross-vendor reference figures

MLPerf Inference v5.1 (September 2025) is the standard for cross-vendor comparability. Server scenario (Llama-3.1-70B, tokens/s):

Submitter	Hardware	Scenario	Throughput (tok/s)
Red Hat (vLLM)	1×H100 80 GB	Server	5,103
Red Hat (vLLM)	1×L40S	Server	1,207
NVIDIA (TRT-LLM)	8×H100 SXM	Server	~52,000 (estimated)

The MLPerf figures are not directly comparable with the internal benchmarks above (a different harness, MLPerf’s Server scenario with a Poisson distribution and fixed SLOs). They are useful for comparing vendors under the same rules; not for sizing a specific case. For that, run your own sweep with AIPerf/GuideLLM against the target endpoint.

Variables the benchmark does not capture

Omitted variable	Effect in production
Network latency (gateway, load balancer)	Can add 20–100 ms to the TTFT measured on the server
Engine compilation time (TRT-LLM)	Hours per model×precision×hardware; penalises cold starts and CI
Weight loading time on the GPU	Relevant in autoscaling; ModelExpress (Dynamo) cuts it 7×
Operational cost of the system (Dynamo vs a simple engine)	Dynamo requires etcd/NATS, a Rust runtime, a more specialised team
Response quality	Performance does not imply quality; measure with lm-evaluation-harness (article B7)
Energy (J/token)	Does not come out of a throughput benchmark; requires DCGM running alongside

Sources

vLLM · documentación oficial — https://docs.vllm.ai/en/latest/
vLLM · blog de rendimiento (v0.6.0, 2,7× throughput) — https://blog.vllm.ai/2024/09/05/perf-update.html
vLLM · disaggregated prefill (experimental) — https://docs.vllm.ai/en/latest/features/disagg_prefill/
SGLang · documentación oficial — https://docs.sglang.io/
SGLang · paper RadixAttention (LMSYS, 2024) — https://arxiv.org/pdf/2312.07104
SGLang · blog LMSYS sobre Llama-3 serving — https://www.lmsys.org/blog/2024-07-25-sglang-llama3/
NVIDIA TensorRT-LLM · GitHub — https://github.com/NVIDIA/TensorRT-LLM
NVIDIA TensorRT-LLM · overview y release notes — https://nvidia.github.io/TensorRT-LLM/overview.html
NVIDIA TensorRT-LLM · quantización — https://nvidia.github.io/TensorRT-LLM/features/quantization.html
NVIDIA Dynamo · GitHub (ai-dynamo/dynamo) — https://github.com/ai-dynamo/dynamo
NVIDIA Dynamo · documentación oficial — https://docs.nvidia.com/dynamo/latest
NVIDIA Dynamo · disaggregated serving — https://docs.dynamo.nvidia.com/dynamo/design-docs/disaggregated-serving
Cerebrium · benchmark vLLM vs SGLang vs TRT-LLM (Llama 3.1-70B) — https://cerebrium.ai/blog/benchmarking-vllm-sglang-tensorrt-for-llama-3-1-api
Spheron · vLLM vs TRT-LLM vs SGLang H100 benchmarks 2026 — https://www.spheron.network/blog/vllm-vs-tensorrt-llm-vs-sglang-benchmarks/
MLCommons · MLPerf Inference v5.1 results — https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/
Red Hat · MLPerf Inference v5.1 con vLLM — https://www.redhat.com/en/blog/efficient-and-reproducible-llm-inference-red-hat-mlperf-inference-v51-results
arXiv 2605.24217 · sesgo sistemático en benchmarks de inferencia LLM — https://arxiv.org/html/2605.24217
IETF Draft · LLM Benchmarking Methodology (enero 2026) — https://www.ietf.org/archive/id/draft-gaikwad-llm-benchmarking-methodology-00.html

MLPerf Power: the standard energy-efficiency benchmark for on-premise ML systems

Mon, 15 Jun 2026 06:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter).

TL;DR

MLPerf Power is the only standardised benchmark for measuring the energy efficiency of complete ML systems. It is run by the MLCommons Power Working Group and backed by more than 20 organisations. It measures power at the wall (AC wall power) for the whole System Under Test, GPUs, CPUs, memory, interconnect and fans included, with a SPEC PTDaemon certified power analyser (Yokogawa WT310/WT5000) and the public tooling in mlcommons/power-dev. The central metric for datacenter inference is samples/joule (or tokens/joule for LLMs); for latency, energy per stream. The public corpus adds up to 1,841 reproducible measurements from 60 systems (590 datacenter, 792 edge, 447 tiny, 12 training). GPT-J and Llama 2 show improvements of more than 100× in samples/joule between the first and the latest available rounds. The direct comparison with the software-based measurement of post C3 (DCGM/NVML/RAPL/Kepler) shows that MLPerf Power offers maximum accuracy and reproducibility at the cost of dedicated hardware (~3,000 USD for the Yokogawa 310E) and of the requirement to be an MLCommons member to access PTDaemon; software measurement is continuous, needs no extra hardware, but carries a wider error bar and no external validation.

What MLPerf Power is and how it fits into the MLCommons ecosystem

MLCommons is the consortium of more than 100 organisations (NVIDIA, Google, Intel, Dell, AMD, Meta and others) that maintains the MLPerf benchmarks: Training, Inference (Datacenter and Edge), Tiny, HPC, Storage, Client and Automotive. The Power Working Group is the specific group that extends each of those benchmarks by adding the energy dimension (MLCommons Power Working Group).

MLPerf Power is not a standalone benchmark: it is an energy measurement layer overlaid on the existing performance benchmarks. For a submission to be valid with power, it must first satisfy the performance rules (MLPerf Inference, Training or Tiny) and then the additional power measurement rules.

Coupling with MLPerf Inference

The integration most relevant to on-premise inference is with MLPerf Inference Datacenter, which defines:

Scenarios: Server (latency, queries per second with a request queue) and Offline (pure throughput, batch with no latency constraint). Power is measured in both.
Benchmarks: ResNet-50, BERT, RNN-T, 3D-UNET, RetinaNet, DLRM, GPT-J 6B, Llama 2 70B (from v4.0), Mixtral 8×7B (from v5.0).
Divisions: closed (model and preprocessing fixed, only runtime optimisation allowed) and open (modifications to the model permitted).

Power is measured during the performance phase, not during accuracy or compliance. The same run that produces the performance log produces the power log: reporting the highest performance of three runs and the lowest power of another three is forbidden (MLPerf Inference Power Measurement Rules, §5.9).

Coupling with MLPerf Training and Tiny

MLPerf Training: large-scale power measurement (multi-node, 10K+ GPUs) does not use an external analyser, which would be impracticable. Node telemetry (IPMI/Redfish) plus an estimate for the interconnect network is used instead. The metric is energy to train (J or kWh). In the v4.0 submissions with power, the systems running Llama 2 70B Training range from single nodes to hundreds, revealing the non-linear scaling of energy (arXiv 2410.12032).
MLPerf Tiny: microcontroller systems (from 5.64 mW). Specialised micro-power instrumentation is used, with hardware pins to demarcate the start and end of the inference. The metric is energy per inference (J), not samples/joule.

Measurement methodology: certified wall power

The fundamental principle: the whole SUT, at the wall

Rule number one is absolute: power must be measured at system level, that is, including every component the benchmark activates: host processors, accelerators, memory, disks, fans, internal interconnect. Measuring the GPUs alone is not enough (MLPerf Inference Power Measurement Rules, §5.1):

“The power consumption must be measured at the system level, i.e. including all components that are sensitized by LoadGen e.g. the host processor on which LoadGen runs, accelerators, memory, fans, etc.”

Power is measured in AC (alternating current), at the SUT input, upstream of the PSUs. Any battery or energy storage between the mains socket and the system PSUs is forbidden.

Measurement hardware: SPEC-certified power analyser

MLPerf Power requires a power analyser certified by SPEC PTDaemon (official SPEC list). The most widespread in submissions is the Yokogawa WT310E (~3,000 USD), which connects to the director over USB (Linux) or Ethernet/serial (Windows). Voltage is measured in parallel and current in series with the SUT power line.

Relevant specifications of the Yokogawa WT310E:

Power accuracy: 0.1 % of reading + 0.1 % of range
Measurement range: µW to MW (the WT5000 reaches industrial installations)
Update rate: from 50 ms

For systems with more than one channel, or multi-PSU nodes, multi-analyser configurations are allowed. The ranging rule: first a ranging run is performed with the range in Auto mode to determine the maximum current and voltage values; the testing runs use fixed ranges based on those peaks, which maximises accuracy within the range. Auto mode is not permitted in the testing runs.

SPEC PTDaemon: the daemon that orchestrates the measurement

PTDaemon (Power Thermal Daemon) is the SPEC tool that handles communication with the analyser. MLCommons is licensed to use it within the MLPerf Power flow. Access requires MLCommons membership and signing the corresponding EULA (MLPerf Power FAQ).

The measurement flow is as follows:

The process step by step (mlcommons/power-dev):

NTP sync between Director and SUT to align timestamps.
Ranging run: power with ranges in Auto; the analyser determines the current and voltage peaks.
Testing run: fixed ranges; LoadGen runs the benchmark; the Director records power with timestamps; the SUT records the performance log with start/end timestamps of the execution phase.
Post-processing: the result summariser cross-references power log and performance log by timestamp to compute the mean power over the execution window.
A minimum of 60 seconds of valid power data. If the workload finishes before 60 s, it is run in a loop until that threshold is reached.

What is and is not inside the SUT

The SUT includes everything the benchmark activates:

Inside the SUT	Outside the SUT
GPUs / accelerators	PDU shared with other systems
Host CPU and RAM	Datacenter cooling infrastructure (PUE)
Disks / NVMe if the benchmark uses them	Management network (BMC, out of band)
Node fans and cooling systems	Datacenter network switches (not the node’s)
Internal interconnect (NVLink, PCIe)
Node PSUs

The paper makes it explicit: PUE is out of scope for MLPerf Power by design (arXiv 2410.12032, §III-C). MLPerf measures the efficiency of the ML system, not that of the datacenter. Including PUE would obscure the differences between systems by mixing hardware efficiency with building efficiency.

Metrics: formulas and definitions

Throughput benchmarks (Datacenter / Offline / Edge)

For throughput benchmarks, Offline and Server in datacenter plus some edge scenarios, the energy efficiency metric is:

$$\eta = \frac{\text{throughput (samples/s)}}{\text{mean power (W)}} \quad \Rightarrow \quad \left[\frac{\text{samples}}{\text{J}}\right]$$

where mean power is computed over the window of the execution phase of the performance run. The reciprocal gives energy per sample:

$$E_{\text{sample}} = \frac{\text{mean power (W)}}{\text{throughput (samples/s)}} \quad \left[\text{J/sample}\right]$$

For LLM benchmarks (GPT-J, Llama 2), where the output varies in length, a “sample” is one complete query (prompt + response). Energy per output token is obtained by dividing by the number of tokens generated, which varies per query and must be reported or estimated.

Latency benchmarks (Single Stream / Tiny)

For latency benchmarks, Single Stream in edge and tiny, where the processing time is fixed, the metric is the inverse of energy per inference:

$$\eta_{\text{latency}} = \frac{1}{E_{\text{inference}}} \quad \left[\frac{1}{\text{J}}\right]$$

The paper treats both metrics (samples/J and 1/J) as comparable within their category, although they are not interchangeable between categories.

System power and run energy

The three quantities that appear in every power submission:

Quantity	Definition	Unit
System power	mean of the AC power samples over the execution window	W
Run energy	mean power × window duration	J
Energy efficiency	throughput / system power = samples/J	samples/J

The run energy formula:

$$E_{\text{run}} = \bar{P} \times \Delta t = \frac{\sum_{i} P_i \cdot \Delta t_i}{\Delta t_{\text{total}}} \times \Delta t_{\text{total}} \quad [\text{J}]$$

where $ P_i $ are the analyser samples and $ \Delta t_i $ the intervals between samples, both inside the window demarcated by the LoadGen timestamps.

How to read an MLPerf Power submission

Divisions and availability categories

Submissions inherit the divisions of MLPerf Inference:

Division	Constraints	Comparability
Closed	Fixed model, fixed preprocessing, only the runtime and the hardware are optimised	High: submissions directly comparable with each other
Open	Modifications to the model allowed (aggressive quantisation, distillation, pruning)	Low: each system may use a different model

Within closed, there are system availability categories:

Category	Definition
Available	System commercially available at the closing date
Preview	Announced but not commercially available
RDI (Research, Development, Internal)	Systems for internal or experimental use

Only Available systems in closed are comparable without caveats. A Preview system with better energy than an Available one from another company is not a valid comparison for purchasing decisions.

What comparability MLPerf Power gives and what it does not

It gives:

An objective comparison of the energy efficiency of complete nodes (not just the GPU) under a standardised ML workload.
Verified reproducibility: the logs are published on GitHub and anyone can review them.
Trends over time between rounds: how much the efficiency of each hardware family improves version to version.
A basis for comparing heterogeneous systems running the same workload under the same rules.

It does not give:

An isolated GPU-to-GPU comparison (it measures the complete node, not the card alone).
Broad hardware coverage: the number of submissions with power is small. In v4.0 only four companies (Dell, Fujitsu, NVIDIA, Qualcomm) delivered power numbers for datacenter. In v5.1 there were two power submissions (Lenovo datacenter + GATEOverflow edge).
Representativeness of your own workloads: the benchmarks are fixed and may not match the distribution of your real load (batch size, prompt length, prefill/decode ratio).
Datacenter cooling data: PUE, liquid vs air, are out of scope.
Real-time comparison: results are published months after the closing date.

Structure of a submission

Every power submission publishes the following in the MLCommons results repository:

<division>/<submitter>/measurements/<system>/
├── analyzer_table.md # Analyser configuration (model, ranges)
├── power_settings.md # Power management configuration of the SUT
results/<system>/<benchmark>/<scenario>/
├── mlperf_log_summary.txt # Performance results (LoadGen)
├── spl.txt # Power log (power, current, voltage, timestamps)
└── ...

The spl.txt files contain the raw analyser readings with timestamps, which makes it possible to verify alignment with the LoadGen window.

Data: scale, trends and reference hardware

The corpus: 1,841 measurements from 60 systems

The MLPerf Power submissions database covers (arXiv 2410.12032):

Category	Submissions	Power range
Datacenter (Inference)	590	~200 W – ~10 kW per node
Edge (Inference)	792	~10 W – several kW
Tiny (Inference)	447	5.64 mW – hundreds of mW
Training / HPC	12	up to 500 kW (measured); ~10 MW estimated in HPC

The total range runs from microwatts to megawatts, 9 orders of magnitude, which means no single methodology applies across every segment.

Energy efficiency improvements: LLMs lead

The evolution of the samples/joule metric between rounds, normalised to the first submission of each model (arXiv 2410.12032, §V-A):

Workload	Category	Cumulative improvement (samples/J)
GPT-J 6B	Datacenter	>100×
Llama 2 70B	Datacenter	>100×
RetinaNet	Datacenter	Largest among the classic models
BERT-99.0	Edge	~4×
RNN-T	Edge	~4×
ResNet-50	Edge	~1.5×
ResNet-50	Tiny	>1,000×
Other Tiny	Tiny	79× – 596×

The >100× improvements in datacenter LLMs reflect the massive industry attention on software optimisation (FP8 kernels, FlashAttention, speculation) and hardware (tensor core architectures, HBM3). Tiny models show even larger gains in relative terms thanks to the low starting point and the progress in specialised chips.

On-premise reference hardware

For a node of 4×H100 SXM 80 GB (the track’s generic hardware), the typical values in MLPerf Power:

Parameter	Indicative value
Declared TDP 4×H100 SXM	4 × 700 W = 2,800 W (GPU only)
System power (complete node) measured at the wall	~3,500 – 5,000 W depending on load
CPU + RAM + fans overhead over GPU	25 – 45 % of the total
Efficiency: Llama 2 70B offline (closed)	a function of the runtime; order of magnitude ~0.1 – 1 tokens/J

The key point: the “node overhead”, the difference between what the analyser measures at the wall and what NVML reports for the GPUs, is 25–45 % of total consumption. A GPU-only measurement underestimates the real system consumption by that margin.

For the A100 PCIe 80 GB (4× per node), system consumption is lower (GPU TDP ~300 W each, node ~1,500–2,000 W), with better efficiency per watt but lower absolute throughput. The L40S (4× per node) has an intermediate profile: TDP ~350 W each, good FP8 inference performance, node consumption ~1,800–2,500 W.

Contrast with the software measurement of post C3

Post C3 of this track covers the DCGM/NVML/RAPL/Kepler stack in production. Here is the data-driven comparison against MLPerf Power:

Dimension	Software (DCGM/NVML/RAPL/Kepler)	MLPerf Power (analyser + PTDaemon)
What it measures	On-device sensors (GPU, CPU, DRAM)	AC wall power (the whole node)
Coverage	GPU + CPU + DRAM (≠ the wall)	Complete node including fans, PSU overhead
Node overhead	Not captured by default	Included in the measurement
Accuracy	±5–15 % vs power meter (Kepler estimate); NVML ~3–5 % on the GPU	±0.1 % reading + ±0.1 % range (Yokogawa WT310E)
Frequency / continuity	Continuous in production (sub-second)	Only during the benchmark run
Extra hardware	None (uses system sensors)	SPEC-certified analyser (~3,000 USD)
Attribution per workload	Per pod/MIG with Kepler	No: it measures the system, not the individual workload
External certification	No	Yes (SPEC PTDaemon + MLCommons review)
Data access	Real time in Prometheus	Public logs on GitHub after the round
Recommended use	Continuous monitoring in production	Certified comparative benchmarking

The operational conclusion: the two approaches are complementary, not competing. The DCGM/Kepler stack gives the continuous per-pod view for production; MLPerf Power gives the certified ground truth for hardware comparisons with reproducibility guarantees.

The myth of measuring only the GPU: data from the paper

The arXiv 2410.12032 paper devotes section §III-C to dismantling the myths of power measurement in ML systems. The one most relevant to on-premise inference is Myth 1:

“A common misconception is that measuring the power consumption of specific ML components, such as accelerators or GPUs, is adequate to assess system efficiency. In reality, overall system power consumption is crucial. Different components are active at various stages of ML workloads with varying duty cycles.”

The data from submissions with a wall analyser versus NVML readings confirms that the overhead of the node’s non-GPU components (CPU, memory, disks, fans, PSU losses) ranges between 25 and 45 % of total consumption depending on the system. An efficiency comparison based on NVML readings alone systematically underestimates consumption and can distort the ranking between systems with different CPU/GPU proportions.

Myth 2 (TDP and PSU as power proxies) is also relevant: the TDP of an H100 SXM is 700 W per card, but real consumption in inference with Llama 2 70B depends on the load, the prompt length and the batch size. The MLPerf Power data shows that systems routinely run well below TDP in online inference scenarios (server scenario), where latency limits throughput and the GPU is not at 100 % utilisation.

Limitations of the current submissions

The limitations are not of the method but of the coverage:

Few submissions with power in datacenter. In v4.0 only 4 companies delivered power numbers for datacenter; in v5.1, one (Lenovo). Performance benchmarking has dozens of submitters; energy benchmarking has few. This limits the representativeness of the corpus for comparing on-premise hardware.
Heterogeneous SUTs. Each submitter defines its own SUT: number of GPUs, server, interconnect, software stack. A Dell system with 4×H100 is not directly comparable to a Supermicro one with 8×H100, even if both use the same model. Platform differences (NVLink vs PCIe, server vs rack) are captured but not separated out.
Fixed workloads. The MLPerf benchmarks are representative but they are not your load. The distribution of prompt lengths, the prefill/decode ratio and your users’ batch size may differ significantly from the MLPerf synthetic datasets. The benchmark’s samples/J result is a reference, not a prediction of your workload.
Publication latency. Results are published months after the submission deadline. New hardware (H200, B100, B200) may have no power results available at the moment of the purchasing decision.
No PUE. MLPerf Power does not include datacenter efficiency. Two identical systems in two datacenters with PUE 1.1 and 1.5 give the same MLPerf Power result but a very different electricity cost. For TCO, the MLPerf J/sample must be multiplied by your own datacenter PUE.

Usage flow for an on-premise platform

For anyone who wants to use MLPerf Power as a purchasing reference or as their own benchmark, the practical flow:

Step	Action	Resource
1	Consult the publicly available submissions	mlcommons.org/benchmarks/inference-datacenter
2	Filter by model (Llama 2 70B, GPT-J), scenario (Offline/Server) and division (Closed/Available)	GitHub mlcommons/inference_results_vX.Y
3	Compare samples/J between systems with similar hardware	power log + performance summary
4	Adjust for your own PUE	samples/J × (1/PUE) for real energy in the datacenter
5	If you want to measure your own hardware: buy a Yokogawa WT310E, join MLCommons, get access to PTDaemon	docs.mlcommons.org/inference/power
6	Calibrate the software stack (DCGM/Kepler) against the analyser once	see post C3

Cross-links of the energy track

This article is C4 of the energy pillar. The related articles:

C1 — State of the art: energy benchmarking of LLM frameworks: inventory of tools and overview of the field.
C2 — Energy per token: methodology and the Spanish electricity market: the J/token identity and how the electricity price multiplies it.
C3 — Measurement tooling in production: Kepler, DCGM and a practical stack: the continuous stack for production that MLPerf Power complements.
Benchmarking track — state of the art of frameworks: context of the performance benchmarking that MLPerf Power extends into energy.

Sources

MLCommons Power Working Group — https://mlcommons.org/working-groups/benchmarks/power/
arXiv 2410.12032 · MLPerf Power: Benchmarking the Energy Efficiency of Machine Learning Systems from Microwatts to Megawatts for Sustainable AI (Tschand et al., 2024) — https://arxiv.org/abs/2410.12032
MLPerf Inference Power Measurement Rules v2.0 (power_measurement.adoc) — https://github.com/mlcommons/inference_policies/blob/master/power_measurement.adoc
MLCommons power-dev · repositorio público de herramientas de medición — https://github.com/mlcommons/power-dev
MLPerf Inference Power Measurement Documentation (MLCFlow) — https://docs.mlcommons.org/inference/power/
SPEC PTDaemon · lista de dispositivos certificados — https://open.spec.org/power/docs/specpower-device_list/
MLCommons · MLPerf Inference v1.0 con las primeras mediciones de potencia (abril 2021) — https://mlcommons.org/2021/04/mlperf-inference-v1-0-results-with-first-power-measurements/
MLCommons · MLPerf Inference v4.1 results (agosto 2024) — https://mlcommons.org/2024/08/mlperf-inference-v4-1-results/
MLCommons · MLPerf Inference v5.1 results (septiembre 2025) — https://mlcommons.org/2025/09/mlperf-inference-v5-1-results/
MLCommons · MLPerf Power benchmark presentado en IEEE HPCA 2025 — https://mlcommons.org/2025/03/ml-commons-power-hpca/
SPEC Updates PTDaemon Interface (GlobeNewswire, febrero 2024) — https://www.globenewswire.com/news-release/2024/02/22/2833367/0/en/SPEC-Updates-PTDaemon-Interface-to-Meet-Evolving-Industry-Requirements.html
Yokogawa WT310E Power Analyzer — https://tmi.yokogawa.com/us/solutions/products/power-analyzers/

GPU idle: the cost that shows up on no invoice but is paid by the whole TCO

Mon, 15 Jun 2026 05:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter).

TL;DR

The cost per token of on-premise inference is $ \text{€/GPU-hour} \div \text{throughput} $. Throughput is a direct function of the GPU’s useful occupancy. On a generic node of 4×H100 SXM with a reference cost of ~11 € per GPU-hour (depreciation plus energy plus infrastructure), the cost curve over occupancy looks like this:

Useful occupancy	Effective throughput (tok/s)	Cost per 1M tokens
20 %	~700	~43 €
40 %	~1,400	~21 €
70 %	~2,500	~12 €
85 % (practical ceiling)	~3,000	~10 €

Doubling occupancy from 20 % to 40 % halves the cost per token, without buying more iron. At 70 % the cost competes with European cloud providers (~2.2 €/GPU-hour on-demand at Scaleway). At 20 % on-prem is four times dearer than renting. The lever is not the model, nor the precision: it is how many paid-for GPU-hours produce serving tokens.

1 · The fundamental identity

The cost per million tokens (CPM) in your own inference is not a list price. It is:

$$ \text{CPM} = \frac{C_{\text{GPU}} \cdot N_{\text{GPU}}}{T_{\text{ef}} \times 3600 / 10^6} $$

where $ C_{\text{GPU}} $ is the cost per GPU-hour (€/h), $ N_{\text{GPU}} $ the number of GPUs assigned to the service and $ T_{\text{ef}} $ the effective throughput (tok/s). Making the dependence on occupancy explicit:

$$ T_{\text{ef}} = T_{\text{pico}} \times \rho $$

where $ \rho \in [0,1] $ is the useful occupancy rate (the fraction of time in which the GPU is processing real serving tokens). The resulting identity:

$$ \text{CPM} = \frac{C_{\text{GPU}} \cdot N_{\text{GPU}}}{T_{\text{pico}} \times \rho \times 3600 / 10^6} $$

The direct consequence: CPM is inversely proportional to $ \rho $. Doubling $ \rho $ halves CPM. The numerator (cost of the iron) does not change.

The posts coste-por-token-y-por-request and capacity-planning-inferencia-llm-on-premise cover how to calculate $ C_{\text{GPU}} $ and the reference peak throughput. This article deals with $ \rho $: how to measure it, what limits it and how to raise it.

2 · Why the standard metric lies: `DCGM_FI_DEV_GPU_UTIL`

The field DCGM_FI_DEV_GPU_UTIL (field ID 203) appears in nvidia-smi as “GPU-Util”. Its official definition in the DCGM documentation:

“GPU Utilization” — percentage of time during which one or more kernels were executing on the GPU.

The problem for LLM inference: the decode phase is memory-bound. The GPU runs a kernel reading weights from HBM token by token; therefore GPU_UTIL registers close to 100 % even though the tensor cores are at 15 % of their capacity. The field measures activity, not useful work.

The distinction is critical for FinOps: an operator who sees GPU_UTIL 98 % assumes “saturated GPU, no room for more load”. The reality may be “the tensor cores are at 20 % and the bottleneck is HBM”, which leaves room for continuous batching or additional bin-packing.

The metrics that measure real occupancy are those of the _FI_PROF_* subsystem, available in DCGM 3.x with the profiling module enabled:

Table of relevant DCGM fields

DCGM field	Field ID	What it measures	Unit	Operational note
`DCGM_FI_DEV_GPU_UTIL`	203	% time with ≥1 active kernel	%	Misleading in LLM decode
`DCGM_FI_PROF_SM_ACTIVE`	1002	Ratio of cycles with ≥1 active warp per SM	0–1	Compute activity, not occupancy
`DCGM_FI_PROF_SM_OCCUPANCY`	1003	Resident warps / theoretical maximum per SM	0–1	Real intra-SM parallelism
`DCGM_FI_PROF_PIPE_TENSOR_ACTIVE`	1004	% cycles with tensor cores (HMMA) active	0–1	The real compute efficiency metric
`DCGM_FI_PROF_DRAM_ACTIVE`	1005	% cycles with HBM transferring	0–1	Memory saturation
`DCGM_FI_PROF_PCIE_TX_BYTES`	—	TX bytes over PCIe	bytes/s	Useful in PCIe inference
`DCGM_FI_DEV_FB_USED`	252	HBM used	MiB	VRAM budget
`DCGM_FI_DEV_POWER_USAGE`	—	Real consumption	W	For real energy cost
`DCGM_FI_DEV_CLOCK_THROTTLE_REASONS`	—	Bitmap of throttle causes	bitmap	Detects silent degradation

The _PROF_* fields require the DCGM profiling module and the right driver permissions. They are documented exhaustively in the NVIDIA DCGM field ID reference and in the sibling post observabilidad-gpu-dcgm-llm.

Characteristic reading in LLM decode (Llama 70B FP8 on H100)

Metric	Typical decode value	Interpretation
`DCGM_FI_DEV_GPU_UTIL`	95–99 %	A lie: there is an active kernel
`DCGM_FI_PROF_SM_ACTIVE`	0.45–0.65	SMs with warps 45–65 % of the time
`DCGM_FI_PROF_SM_OCCUPANCY`	0.30–0.55	Resident warps at 30–55 % of the maximum
`DCGM_FI_PROF_PIPE_TENSOR_ACTIVE`	0.10–0.25	Tensor cores active only 10–25 %
`DCGM_FI_PROF_DRAM_ACTIVE`	0.75–0.90	HBM busy 75–90 % of the time

Reading: decode is memory-bound. The model weights are loaded from HBM for each token; HBM is saturated but the tensor cores wait. All of the “utilisation” in nvidia-smi comes from memory reads, not from compute.

3 · MFU and HFU: occupancy expressed in FLOPs

The canonical compute efficiency metric is MFU (Model FLOPs Utilization), defined in the PaLM paper (Chowdhery et al., arXiv 2204.02311, section 4):

$$ \text{MFU} = \frac{T_{\text{obs}} \times C_{\text{modelo}}}{P_{\text{pico}}} $$

where $ T_{\text{obs}} $ is the observed throughput (tok/s) and $ P_{\text{pico}} $ is the theoretical peak performance of the hardware (FLOP/s).

where $ C_{\text{modelo}} $ is the number of FLOPs per token in a complete forward pass. For a dense transformer with $ P $ parameters, the usual approximation (forward plus backward = 6P FLOPs per token; forward only = 2P):

$$ C_{\text{modelo}} \approx 2P \quad \text{(inference)} $$ $$ C_{\text{modelo}} \approx 6P \quad \text{(training, forward + backward)} $$

HFU (Hardware FLOPs Utilization) measures the FLOPs actually executed in hardware, including activation recomputation (gradient checkpointing). In training with recompute:

$$ C_{\text{hardware}} \approx 8P \quad \text{(forward × 2 + backward × 4)} $$

therefore HFU > MFU when there is recompute; they are identical without it.

Typical MFU values

Regime	Hardware	Typical MFU	Limiting regime
Training (large batch, BF16)	H100 SXM	35–50 %	compute-bound
Training PaLM 540B	TPU v4	46.2 %	compute-bound
Prefill inference (large batch)	H100 SXM	25–45 %	compute-bound
Decode inference (bs=1)	H100 SXM	3–8 %	memory-bound
Decode inference (continuous batching, bs=32–64)	H100 SXM	15–30 %	memory-bound, attenuated

Decode with batch size 1 has a single-digit MFU because the hardware spends most of its time waiting for HBM to deliver weights. Raising the batch size (continuous batching) amortises the weight read across more simultaneous tokens and raises MFU.

The Roofline model

The roofline places each operation in the space (arithmetic intensity, throughput):

Decode with batch size 1 falls in the memory-bound zone to the left of the ridge point. Raising the batch (continuous batching) moves the point to the right and up, closer to the roofline.

4 · TCO sensitivity to occupancy

Assumptions for the example: a generic node with 4×H100 SXM 80 GB, total node cost ~44 €/hour (5-year depreciation of ~220,000 €, energy ~4×700 W at ~0.12 €/kWh, plus rack/colocation infrastructure; see coste-por-token-y-por-request for the detail of the identity). Reference peak throughput with Llama 70B FP8 plus continuous batching: ~3,500 tok/s aggregate.

$$ \text{CPM}(\rho) = \frac{44\ \text{€/h}}{3500 \times \rho \times 3600 / 10^6} = \frac{44 \times 10^6}{3500 \times \rho \times 3600} = \frac{3.49}{\rho}\ \text{€/1M tok} $$

Occupancy $ \rho $	Effective throughput (tok/s)	On-prem CPM	On-demand cloud CPM (~2.2 €/GPU-h)
20 %	700	~17.5 €	~6.3 €
40 %	1,400	~8.7 €	~6.3 €
60 %	2,100	~5.8 €	~6.3 €
70 %	2,450	~5.0 €	~6.3 €
85 %	2,975	~4.1 €	~6.3 €

On-prem / cloud crossover point: with this example hardware, the on-prem cost advantage over a comparable European cloud starts at $ \rho \approx 55\text{-}60\,\% $. Below that, idle makes on-prem dearer than renting. The cloud column is flat because cloud bills per hour used, not per the throughput you extract from it.

Example figures with generic hardware. The real numbers depend on the depreciation price, the local cost of energy, the model and the precision. The structure of the curve, CPM inversely proportional to $ \rho $, is universal.

5 · Idle metrics: where occupancy is lost

Before applying levers, you need to know which kind of idle dominates. Three categories:

Type of idle	Symptom in metrics	Usual cause
Scheduling idle	`DCGM_FI_DEV_POWER_USAGE` low, `SM_ACTIVE` < 0.05	No requests in the queue; GPU waiting for work
Batching idle	`SM_ACTIVE` high, `PIPE_TENSOR_ACTIVE` low, `DRAM_ACTIVE` low	Batch too small; prefill stall between requests
Memory-bound idle	`DRAM_ACTIVE` high, `PIPE_TENSOR_ACTIVE` low	Normal decode; HBM is the bottleneck; insufficient batch size

Scheduling idle is the most expensive and the most directly attacked with bin-packing. Batching idle is attacked with continuous batching. Memory-bound idle in decode does not disappear entirely (it is the physics of the transformer), but it is attenuated with a larger batch size.

The inference engine metrics that complement the diagnosis from the service side are documented in observabilidad-gpu-dcgm-llm and anatomia-metricas-dcgm-vllm-anomalias.

6 · Levers for raising occupancy

Table of lever × effect × when it applies

Lever	Effect on $ \rho $	When it applies	Operational complexity
Continuous batching	High: removes idle between requests; raises decode MFU from ~5 % to ~20 %	Always in inference; enabled by default in vLLM	Low (engine parameter)
Scheduler bin-packing (kube-scheduler / Kueue)	High: concentrates loads on fewer nodes; frees whole nodes for shutdown or rebalancing	Clusters with load that varies over the day	Medium (scheduler policy)
MIG (Multi-Instance GPU)	Medium: fills GPUs with light loads that previously lived alone on a whole GPU	Heterogeneous loads: embeddings + reranker + guardrail + large model	High (hot repartitioning not available)
Time-slicing	Low to medium: raises occupancy in dev/bursts; no isolation	Consumer GPUs (RTX 5090/4090); low-risk multi-tenant dev	Low
MPS	Medium: concurrent execution of multiple small processes; reduces context-switch overhead	Many small concurrent kernels on a datacenter GPU; trust between loads	Medium
Chunked prefill	Medium: interleaves prefill and decode; reduces the TTFT spike and raises throughput	Loads with a mix of short and long prompts	Low (vLLM flag)
Quantisation (FP16→FP8→INT4)	Indirect: raises peak throughput, which lowers CPM for the same $ \rho $	Models with quantised kernel support (native Hopper FP8)	Medium
Autoscaling (KEDA)	Keeps $ \rho $ high by scaling replicas according to the queue	Variable and predictable load; cluster with spare capacity	Medium

The post compartir-gpu-time-slicing-mps-mig details the three sharing mechanisms (time-slicing, MPS, MIG) with VRAM budgets.

Bin-packing with Kueue

Kueue (sigs.k8s.io/kueue) is the Kubernetes-native queue manager for GPU jobs. Its model of cohorts and nominal quotas allows active bin-packing: jobs accumulate in a queue and are launched only when there is a node that can take them whole, instead of fragmenting the load across partially occupied nodes.

BestFit packing in the ClusterQueue is configured with:

apiVersion: kueue.x-k8s.io/v1beta1
kind: ClusterQueue
metadata:
 name: gpu-prod
spec:
 preemption:
 reclaimWithinCohort: Any
 withinClusterQueue: LowerPriority
 resourceGroups:
 - coveredResources: ["nvidia.com/gpu"]
 flavors:
 - name: h100-sxm
 resources:
 - name: "nvidia.com/gpu"
 nominalQuota: 16

The priority-based preemption policy ensures that production jobs displace experimentation ones when there is scarcity, keeping occupancy on the production nodes high.

MIG as a bin-packing lever inside the GPU

MIG makes it possible to fill an H100 with light loads that would otherwise live alone on a whole GPU. A profile of 3×2g.20gb + 1×1g.10gb on an H100 can host simultaneously a 7B FP8 model (~14 GB of weights), two embedding services and an INT4 guardrail, all with hardware isolation. Without MIG, each of those loads would take a whole GPU with an individual $ \rho $ below 10 %.

The profiles available on the H100 80 GB (SXM5) according to NVIDIA’s MIG User Guide:

MIG profile	Compute slices	Memory	Max. instances
`1g.10gb`	1/7 SMs	10 GB	7
`1g.20gb`	1/7 SMs	20 GB	4
`2g.20gb`	2/7 SMs	20 GB	3
`3g.40gb`	3/7 SMs	40 GB	2
`4g.40gb`	4/7 SMs	40 GB	1
`7g.80gb`	7/7 SMs	80 GB	1 (whole GPU)

MIG is not available on consumer GPUs (RTX 5090, RTX 4090). Time-slicing is the only sharing option on that hardware.

7 · Continuous batching: the effect on MFU

Continuous batching (also called iteration-level scheduling or in-flight batching) is the mechanism with the greatest impact on occupancy in inference. The idea: instead of waiting for a complete batch to finish before launching the next, the engine evaluates the pipeline after each token and replaces completed sequences with new requests from the waiting queue.

Quantified effect in vLLM:

Without batching (bs=1, static): decode MFU ~3–8 %; GPU idle between requests.
With continuous batching (dynamic bs 16–64): decode MFU ~15–30 %; the GPU almost never waits.
On pure prefill loads with a large batch (bs=128+): prefill MFU 25–45 %; it approaches the compute roofline.

The vLLM parameter --max-num-seqs controls the maximum number of sequences in the concurrent batch. Raising it increases occupancy until HBM becomes the bottleneck (watch for DCGM_FI_PROF_DRAM_ACTIVE > 90 % sustained).

8 · Hardware and scale: what applies to what

Hardware	MIG	MPS	Time-slicing	Continuous batching	Note
H100 SXM / H200	Yes (7 inst.)	Yes	Yes	Yes	On-prem datacenter reference
A100 SXM/PCIe	Yes (7 inst.)	Yes	Yes	Yes	Previous generation; HBM2e
L40S / L40	No	Yes	Yes	Yes	Ada Lovelace; no MIG; good price/VRAM
RTX 5090	No	Yes (limited)	Yes	Yes	Consumer; no MIG; does not scale in production
RTX 4090	No	Yes (limited)	Yes	Yes	Consumer; 24 GB VRAM; no MIG

The RTX 5090 and RTX 4090 illustrate the case of hardware that does not scale for multi-tenant with isolation: they do not support MIG, VRAM is scarce for models above 7B with a wide KV-cache, and TDP (600 W / 450 W) is high relative to throughput. For production inference at scale, the 4×H100 SXM is the reference node of this series.

9 · FinOps diagnostic flow: from high CPM to the cause

10 · Worked example: impact of continuous batching on CPM

Starting point: Llama 70B FP8 on 4×H100 SXM, a load of 8 requests/s with 512 output tokens on average, without continuous batching (static bs = 8):

Observed throughput: ~900 tok/s (decode dominant).
$ \rho_{\text{efectiva}} \approx 900 / 3500 \approx 0.26 $.
CPM: $ 44 / (900 \times 3600 / 10^6) \approx 13.6\ \text{€} $.

Same load, enabling continuous batching with --max-num-seqs 64:

Observed throughput: ~2,300 tok/s (the dynamic batch fills the gaps).
$ \rho_{\text{efectiva}} \approx 2300 / 3500 \approx 0.66 $.
CPM: $ 44 / (2300 \times 3600 / 10^6) \approx 5.3\ \text{€} $.

CPM reduction: from 13.6 € to 5.3 €, –61 %, with no hardware change and without touching the model.

The vLLM parameters relevant for raising effective throughput:

--max-num-seqs 64 # maximum concurrent batch
--max-num-batched-tokens 16384 # total tokens per iteration
--enable-chunked-prefill # interleaves prefill and decode

Sources

NVIDIA — DCGM Field Identifiers reference (v3.1), lista completa de DCGM_FI_* con field IDs y definiciones. https://docs.nvidia.com/datacenter/dcgm/3.1/dcgm-api/dcgm-api-field-ids.html
NVIDIA — GPU Profiling Metrics (Run:ai / DCGM), definiciones de DCGM_FI_PROF_SM_ACTIVE (1002), SM_OCCUPANCY (1003), PIPE_TENSOR_ACTIVE (1004). https://run-ai-docs.nvidia.com/self-hosted/platform-management/monitor-performance/gpu-profiling-metrics
Chowdhery et al. — PaLM: Scaling Language Modeling with Pathways (MFU definition, sección 4). arXiv 2204.02311. https://arxiv.org/abs/2204.02311
NVIDIA — Multi-Instance GPU (MIG) User Guide (perfiles H100 SXM5, particionado, aislamiento). https://docs.nvidia.com/datacenter/tesla/mig-user-guide/
NVIDIA — Supported MIG Profiles (catálogo completo H100 80 GB). https://docs.nvidia.com/datacenter/tesla/mig-user-guide/supported-mig-profiles.html
NVIDIA — Time-Slicing GPUs in Kubernetes (GPU Operator 24.9.0). https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/24.9.0/gpu-sharing.html
NVIDIA — Multi-Process Service (MPS) Overview (kernel concurrency, SM allocation). https://docs.nvidia.com/deploy/mps/latest/index.html
Kueue — Overview (sigs.k8s.io/kueue, bin-packing, cohorts, quotas). https://kueue.sigs.k8s.io/docs/overview/
vLLM Blog — vLLM v0.6.0: 2.7x Throughput Improvement and 5x Latency Reduction (continuous batching, chunked prefill). https://blog.vllm.ai/2024/09/05/perf-update.html
GMI Cloud — NVIDIA H100 GPU Pricing: 2026 Rent vs. Buy Cost Analysis. https://www.gmicloud.ai/en/blog/nvidia-h100-gpu-pricing-2026-rent-vs-buy-cost-analysis
Saxena et al. — LLM Inference Unveiled: Survey and Roofline Model Insights. arXiv 2402.16363. https://arxiv.org/abs/2402.16363

GPU chargeback and showback in multi-tenancy: how to split cluster cost between teams

Mon, 15 Jun 2026 04:00:00 +0200

Notation: amounts in euros (N €), decimals with a point. The dollar sign is not used (on this site it is a formula delimiter).

TL;DR

A cluster of 4×H100 SXM at ~1.40 €/GPU-hour (depreciated capex plus energy) shared between three teams with different occupancy produces a monthly chargeback report of 3 rows and an additional idle row that nobody claimed. Without tooling, that idle cost vanishes, diluted into the total. With OpenCost + LiteLLM + Kueue, attribution operates on three orthogonal planes: the iron (OpenCost, €/GPU-hour per namespace/label), token consumption (LiteLLM, €/token per key/team/model) and the scheduler quota (Kueue, GPUs reserved per ClusterQueue). Crossing the three produces the number that goes to finance: team B consumed X million tokens at Y €/1M tok, the GPU cost it Z €, and it still has 2 GPUs borrowed from the cohort that will cost it W € if it holds on to them next month.

Showback vs chargeback: the FinOps Foundation definition

The distinction is not one of technology but of accounting formality (FinOps Foundation — Invoicing & Chargeback, FinOps Foundation — Data Analysis and Showback):

Concept	Definition	Moves money	Requires
Showback	visibility of consumption and its cost per team; the report arrives, the budget does not change	No	metrics + attribution
Chargeback	the cost is transferred to the team’s or product’s P&L as real expenditure	Yes	showback + accounting policy + financial system

Two points from the framework worth pinning down:

Showback is a requirement of any FinOps practice; chargeback is optional and depends on the organisation’s accounting policy supporting transfers between cost centres.
Neither is “more mature” than the other. The narrative that chargeback is the “grown-up” version is false according to the framework itself. The natural sequence is: showback → trust in the data → chargeback if policy allows it.

When to use each

Situation	Recommended mode
First attribution cycle; teams do not trust the data yet	Showback
Teams with their own budget in a financial system	Chargeback
Shared cluster with no P&L separation per team	Showback with visible idle cost
Teams with a committed GPU availability SLA	Chargeback (real reservation)
Research/experimentation workloads with no formal budget	Showback + threshold alert

GPU cost attribution with OpenCost

Allocation API: attribution parameters

The OpenCost /allocation API is the piece that turns Prometheus metrics into a cost report per dimension (OpenCost — API):

Parameter	Values	Use in multi-tenancy
`window`	`today`, `7d`, `lastmonth`, RFC3339 range	window of the monthly report
`aggregate`	`namespace`, `label:LABEL`, `annotation:NAME`, `pod`, `controller`	attribution dimension
`includeIdle`	`true` / `false`	adds an `__idle__` row to the report
`shareIdle`	`true` / `false`	distributes idle among the non-idle allocations (proportional to non-idle cost)
`idleByNode`	`true` / `false`	computes idle per node instead of per cluster
`filter`	`namespace:"llm-prod"`, `label:equipo:"datos"`	restrict to one specific team

Query for a monthly report per team, with idle visible as a separate row:

curl -G http://localhost:9003/allocation \
 -d window=lastmonth \
 -d aggregate=label:equipo \
 -d includeIdle=true \
 -d shareIdle=false \
 -d resolution=10m

With shareIdle=false (the default), OpenCost returns idle as a separate __idle__ entry, which makes visible how much was paid for unused capacity. With shareIdle=true, that idle is distributed among the tenants in proportion to their non-idle cost: each absorbs its share of the waste, which is the right behaviour for a chargeback that penalises whoever is responsible for the idle.

Labelling inference pods

For aggregate=label:equipo to work, the inference pods need the right label. Two methods:

Label in the pod spec (the most direct):

# vLLM Deployment for the datos team
metadata:
 labels:
 equipo: datos
 producto: rag-prod
 modelo: llama-70b
 entorno: prod

Annotation (when the label is already taken by another convention):

# aggregate as an annotation
aggregate=annotation:finops.io/equipo

OpenCost supports aggregate=label:KEY and aggregate=annotation:KEY with the same syntax; the choice between the two depends on the cluster’s labelling convention.

Shared costs and idle: the three modes

API mode	Behaviour	When
`includeIdle=false` (default)	Ignores idle; total cost looks lower than it is	Never in production
`includeIdle=true, shareIdle=false`	Idle in a separate `__idle__` row	Showback: visibility of waste
`includeIdle=true, shareIdle=true`	Idle distributed among tenants proportionally	Chargeback: the team pays its share of idle
`shareIdle=true, idleByNode=true`	Idle distributed per node (finer-grained if there are dedicated nodes)	Chargeback with heterogeneous nodes

For infrastructure with nodes dedicated per team, idleByNode=true is fairer: the idle of a node dedicated to team A is not transferred to team B.

LiteLLM as the attribution point for token consumption

Virtual keys and teams

LiteLLM materialises token chargeback with four mechanisms (LiteLLM — Virtual Keys, LiteLLM — Setting Team Budgets, LiteLLM — Spend Tracking):

Mechanism	What it does
Virtual key with `max_budget`	monthly budget per key; the key is blocked once it is exhausted
`budget_duration`	reset window: `30d`, `7d`, `24h`; the proxy runs a daily cron that resets according to the duration
`team_id`	groups keys; spend accumulates in `LiteLLM_TeamTable` per team
`tags`	tag each request; they allow budgets per tag (cost centre)
Spend logs	one row per request with `team_id`, `model`, `prompt_tokens`, `completion_tokens`, `response_cost`

Configuring teams with a monthly budget

# litellm-config.yaml — per-team budgets with the declared on-prem cost
model_list:
 - model_name: llama-3-70b-onprem
 litellm_params:
 model: openai/llama-3-70b
 api_base: http://vllm-svc:8000/v1
 input_cost_per_token: 0.00000140 # 1.40 €/1M tok (€/GPU-hour ÷ throughput)
 output_cost_per_token: 0.00000140
 - model_name: llama-3-8b-onprem
 litellm_params:
 model: openai/llama-3-8b
 api_base: http://vllm-8b-svc:8000/v1
 input_cost_per_token: 0.00000035 # 0.35 €/1M tok (1 GPU, higher throughput)
 output_cost_per_token: 0.00000035

litellm_settings:
 success_callback: ["langfuse"] # or any external logger

# Teams (via the /team/new API or config)
# POST /team/new
# {
# "team_alias": "equipo-datos",
# "max_budget": 800, # 800 € per month
# "budget_duration": "30d",
# "tpm_limit": 500000, # max tokens/min
# "rpm_limit": 1000 # max requests/min
# }

The input_cost_per_token key comes from the identity in article A4: the €/GPU-hour from OpenCost divided by the benchmark throughput. With that value, the response_cost of each request reflects the real cost of the on-prem iron for that model.

Endpoints for querying spend per team

# Team spend accumulated over the period
GET /team/info?team_id=equipo-datos

# Team spend per model (aggregated spend_logs table)
GET /global/spend/logs?team_id=equipo-datos&start_date=2026-06-01&end_date=2026-06-30

# Total spend across all teams
GET /global/spend/teams

The response includes spend (accumulated cost in the configured currency), total_tokens, prompt_tokens, completion_tokens and a per-model breakdown, the direct fields for the monthly chargeback report.

Joining €/GPU-hour (iron) with tokens per team

The two attribution planes, OpenCost (iron) and LiteLLM (tokens), are crossed with one key: the model_name and the namespace. The join happens outside the tools (in a BI pipeline or a SQL query over the LiteLLM database and the data exported from OpenCost):

coste_por_token_real = OpenCost.GPU_cost_namespace / LiteLLM.total_tokens_team

When the on-prem cost is properly declared in input_cost_per_token, LiteLLM already does this calculation internally and the response_cost is correct. The explicit join serves to validate: if the sum of LiteLLM’s response_cost per team does not match the GPU cost OpenCost assigns to the namespace, there is drift in input_cost_per_token that has to be corrected (changed throughput, a new optimisation, more replicas).

Kueue as the GPU budget/quota mechanism

ClusterQueue, LocalQueue and cohorts

Kueue introduces the quota-based scheduling layer on top of Kubernetes (Kueue — Cluster Queue, Kueue — Cohort, Kueue — Fair Sharing):

Object	Scope	What it does
`ResourceFlavor`	cluster	maps resources to a group of nodes (e.g. H100 nodes)
`ClusterQueue`	cluster	defines `nominalQuota`, `borrowingLimit`, `lendingLimit` per resource/flavor
`LocalQueue`	namespace	entry point for the team’s workloads; points to a `ClusterQueue`
`Cohort`	cluster	groups ClusterQueues that can lend quota to each other

Quota concepts

nominalQuota: GPUs guaranteed to the ClusterQueue at all times.
borrowingLimit: maximum additional GPUs it can borrow from the cohort when others are not using them.
lendingLimit: GPUs from its nominalQuota that it allows to be lent to others (if not specified, it can lend all unused ones).
Fair Sharing: a mechanism that orders pending workloads by the historical resource usage of their LocalQueue, giving preference to whoever has consumed less. Compatible with hierarchical cohorts since Kueue v0.11.

Example YAML: 3 teams in the llm-platform cohort

# ResourceFlavor mapping to the H100 SXM nodes
apiVersion: kueue.x-k8s.io/v1beta2
kind: ResourceFlavor
metadata:
 name: h100-sxm
spec:
 nodeLabels:
 accelerator: h100-sxm
---
# ClusterQueue for the datos team — 2 guaranteed GPUs, can borrow up to 2
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
 name: cq-datos
spec:
 cohort: llm-platform
 namespaceSelector:
 matchLabels:
 kubernetes.io/metadata.name: ns-datos
 queueingStrategy: BestEffortFIFO
 resourceGroups:
 - coveredResources: ["nvidia.com/gpu"]
 flavors:
 - name: h100-sxm
 resources:
 - name: "nvidia.com/gpu"
 nominalQuota: 2
 borrowingLimit: 2 # can use up to 4 GPUs in total
 preemption:
 reclaimWithinCohort: LowerPriority
 withinClusterQueue: LowerPriority
---
# ClusterQueue for the ia team — 1 guaranteed GPU, lends what it does not use
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
 name: cq-ia
spec:
 cohort: llm-platform
 namespaceSelector:
 matchLabels:
 kubernetes.io/metadata.name: ns-ia
 resourceGroups:
 - coveredResources: ["nvidia.com/gpu"]
 flavors:
 - name: h100-sxm
 resources:
 - name: "nvidia.com/gpu"
 nominalQuota: 1
 lendingLimit: 1 # lends its GPU when it is not using it
---
# ClusterQueue for the plataforma team — 1 guaranteed GPU, low priority
apiVersion: kueue.x-k8s.io/v1beta2
kind: ClusterQueue
metadata:
 name: cq-plataforma
spec:
 cohort: llm-platform
 namespaceSelector:
 matchLabels:
 kubernetes.io/metadata.name: ns-plataforma
 resourceGroups:
 - coveredResources: ["nvidia.com/gpu"]
 flavors:
 - name: h100-sxm
 resources:
 - name: "nvidia.com/gpu"
 nominalQuota: 1
 borrowingLimit: 3 # can use all 4 if the rest are free
---
# LocalQueue in each namespace — entry point for the workloads
apiVersion: kueue.x-k8s.io/v1beta2
kind: LocalQueue
metadata:
 name: lq-datos
 namespace: ns-datos
spec:
 clusterQueue: cq-datos

How Kueue materialises the “GPU budget”

FinOps concept	Kueue mechanism
Guaranteed budget	`nominalQuota`: the team always has these GPUs available
Maximum spend limit	`nominalQuota + borrowingLimit`: absolute ceiling of admissible GPUs
Lending idle capacity	cohort + `lendingLimit`: others take what this one does not use
Fair sharing between teams	`Fair Sharing` + `WorkloadPriorityClass`: those who have consumed most wait longest
Reclaiming your own quota	`preemption.reclaimWithinCohort: LowerPriority`: the owner of the lent GPU reclaims it by evicting lower-priority workloads

The nominalQuota is the expression of the budget in GPUs: if the team has 2 nominal GPUs and the price is 1.40 €/GPU-hour, its maximum guaranteed spend is

\[ \text{maximum nominal spend} = 2 \times 1.40 \times 720 = 2{,}016 \text{ €/month} \]

The borrowingLimit sets the potential overspend if it borrows from the cohort (and that borrowing can be charged with the same formula, multiplying the borrowed usage hours by 1.40 €/GPU-hour).

Cost model: an example with 4×H100 SXM and 3 teams

Node price

Generic hardware: a server with 4×H100 SXM 80 GB. 2026 market prices: a single H100 SXM costs between 27,000 and 40,000 € depending on the source (GMI Cloud — H100 GPU Pricing 2026); a 4×H100 server sits in the 120,000–180,000 € range including chassis, power supplies and NVLink. We use 140,000 € as a generic assumption for the complete server (4 GPUs plus infrastructure).

Component	Calculation	€/hour node
Depreciated capex (140,000 €, 4 years, 90 % availability)	140,000 ÷ (4 × 8,760 × 0.9)	~4.43
Energy (4 × 700 W TDP × PUE 1.4 × 0.12 €/kWh)	4 × 0.7 × 1.4 × 0.12	~0.47
Operations / networking / maintenance	estimate	~0.70
Total 4×H100 node		~5.60 €/h
Per GPU	5.60 ÷ 4	~1.40 €/GPU-hour

Cost-per-token formula for the 70B model (sustained throughput ~2,000 tok/s at TP=4):

$$\text{€/1M tokens} = \frac{5.60 \text{ €/h}}{2{,}000 \text{ tok/s} \times 3{,}600 \text{ s/h}} \times 10^6 \approx 0.78 \text{ €/1M tokens}$$

Monthly scenario: 3 teams, heterogeneous utilisation

Reference month (720 hours). The 4 GPUs are nominally assigned: 2 to Datos, 1 to IA, 1 to Plataforma. Real occupancy varies:

Team	Nominal GPUs	Average occupancy	GPU-hours used	Idle GPU-hours	GPU cost (€)
Datos	2	75 %	1,080	360	~1,512
IA	1	60 %	432	288	~605
Plataforma	1	35 %	252	468	~353
Cluster idle	—	—	—	1,116	~1,562
Node total	4	—	1,764	1,116	~4,032

Total node cost per month: 5.60 €/h × 720 h = 4,032 €. Cost per GPU-hour used: 5.60 ÷ 4 = 1.40 €.

With shareIdle=false (showback), the 1,562 € of idle appears as a separate row in the report and nobody pays for it directly. With shareIdle=true (proportional chargeback), it is split among the three in proportion to their non-idle cost: Datos absorbs ~857 €, IA ~333 €, Plataforma ~192 € of additional idle.

Token consumption per team (LiteLLM spend logs)

Team	Model	Tokens/month	Cost/token	Token cost (€)
Datos	llama-70b (TP=4)	380 M	0.78 €/1M	~296
IA	llama-8b (1 GPU)	210 M	0.35 €/1M	~74
Plataforma	llama-8b + batch	80 M	0.35 €/1M	~28

Monthly chargeback report (showback + tokens)

Team	Iron GPU cost (€)	Assigned idle cost (€)	LiteLLM token cost (€)	Total chargeable (€)
Datos	1,512	857	296	2,665
IA	605	333	74	1,012
Plataforma	353	192	28	573
Idle (without shareIdle)	1,562	—	—	visible, not charged

Attribution dimensions × policy table

Attribution dimension	Tool	Showback	Chargeback
namespace	OpenCost `aggregate=namespace`	`/allocation?aggregate=namespace&includeIdle=true`	`shareIdle=true` + transfer to P&L
pod label	OpenCost `aggregate=label:equipo`	report per label	same with shareIdle
virtual key / team	LiteLLM `/global/spend/teams`	token dashboard	hard `max_budget` per key
ClusterQueue (reserved GPU)	Kueue `nominalQuota`	observe usage vs quota	committed budget in GPUs
request tag	LiteLLM `tag_budgets`	per cost centre	budget per tag
pod annotation	OpenCost `aggregate=annotation:KEY`	report per annotation	same with shareIdle

End-to-end data flow

When the Datos team has exhausted its 2 nominal GPUs and borrows those of Plataforma (which is not using them), Kueue records that loan. As soon as Plataforma launches a new workload:

Kueue detects that cq-plataforma is below its nominalQuota.
With preemption.reclaimWithinCohort: LowerPriority, Kueue evicts the Datos workload that was on the borrowed GPU, if its priority is lower.
The evicted workload returns to the cq-datos queue and is readmitted once quota frees up.

Fair Sharing orders the pending queue by the LocalQueue’s accumulated historical usage: if Datos has consumed more GPU-hours than IA in the recent period, IA’s next workloads have admission priority over new ones from Datos. This implements an equitable split without blocking anyone permanently.

Configuring OpenCost for the node price

For the numbers in the previous section to be reproducible, the node price has to be declared explicitly (the default value underestimates on-prem GPU, issue #3781):

# OpenCost Helm values.yaml — on-prem 4×H100 node, 5.60 €/h
opencost:
 customPricing:
 enabled: true
 provider: custom
 costModel:
 description: "On-prem 4xH100 SXM node"
 CPU: "0.025" # €/CPU-hour (minor component)
 RAM: "0.003" # €/GB-hour
 GPU: "1.40" # €/GPU-hour ← the one that drives the split
 storage: "0.0002" # €/GB-hour

Verify after configuring:

# Check the price resolved per node
curl http://localhost:9003/allNodePricing

# Attribution query per team label, last month, idle separate
curl -G http://localhost:9003/allocation \
 -d window=lastmonth \
 -d aggregate=label:equipo \
 -d includeIdle=true \
 -d shareIdle=false \
 -d accumulate=true

Reference PromQL for chargeback dashboards

# GPU cost assigned per team (label:equipo) — €/hour
sum by (label_equipo) (
 container_gpu_allocation
 * on(node) group_left node_gpu_hourly_cost
)

# Idle GPU-hours per node (idle > 15 min)
sum by (node) (
 (1 - avg_over_time(DCGM_FI_DEV_GPU_UTIL[15m]) / 100)
 * on(node) group_left node_gpu_hourly_cost
)

# Accumulated monthly cost per team (sum over window)
sum_over_time(
 sum by (label_equipo) (
 container_gpu_allocation * on(node) group_left node_gpu_hourly_cost
 )[30d:1h]
)

Sources

FinOps Foundation — Invoicing & Chargeback Capability — https://www.finops.org/framework/capabilities/invoicing-chargeback/
FinOps Foundation — Data Analysis and Showback — https://www.finops.org/framework/previous-capabilities/analysis-showback/
FinOps Foundation — Allocation Capability — https://www.finops.org/framework/capabilities/allocation/
OpenCost — API (Allocation API, parámetros window/aggregate/shareIdle/idleByNode) — https://opencost.io/docs/integrations/api/
OpenCost — API Examples — https://opencost.io/docs/integrations/api-examples/
OpenCost — GitHub (issue #3781, infra-precio GPU on-prem por defecto) — https://github.com/opencost/opencost/issues/3781
LiteLLM — Virtual Keys (max_budget, budget_duration, team_id) — https://docs.litellm.ai/docs/proxy/virtual_keys
LiteLLM — Setting Team Budgets — https://docs.litellm.ai/docs/proxy/team_budgets
LiteLLM — Spend Tracking — https://docs.litellm.ai/docs/proxy/cost_tracking
LiteLLM — Budgets & Rate Limits — https://docs.litellm.ai/docs/proxy/users
LiteLLM — Setting Tag Budgets — https://docs.litellm.ai/docs/proxy/tag_budgets
LiteLLM — Budget Reset Times — https://docs.litellm.ai/docs/proxy/budget_reset_and_tz
Kueue — Cluster Queue (nominalQuota, borrowingLimit, lendingLimit, preemption) — https://kueue.sigs.k8s.io/docs/concepts/cluster_queue/
Kueue — Cohort — https://kueue.sigs.k8s.io/docs/concepts/cohort/
Kueue — Fair Sharing — https://kueue.sigs.k8s.io/docs/concepts/fair_sharing/
Kueue — Administer Cluster Quotas — https://kueue.sigs.k8s.io/docs/tasks/manage/administer_cluster_quotas/
GMI Cloud — NVIDIA H100 GPU Pricing 2026 — https://www.gmicloud.ai/en/blog/nvidia-h100-gpu-pricing-2026-rent-vs-buy-cost-analysis
IntuitionLabs — NVIDIA AI GPU Prices H100 Cost Guide — https://intuitionlabs.ai/articles/nvidia-ai-gpu-pricing-guide

GenAI-Perf in depth: LLM inference profiling with real data

Mon, 15 Jun 2026 03:00:00 +0200

Notation: amounts in euros (N EUR), decimals with a point. The dollar symbol is not used (on this site it is a formula delimiter).

TL;DR

GenAI-Perf (the genai-perf package, part of NVIDIA’s Triton ecosystem) is the reference LLM inference profiler for OpenAI-compatible endpoints. Installable with pip install genai-perf. A single genai-perf profile command produces a console table with TTFT, ITL, request latency (avg/min/max/P75/P90/P99), output sequence length, output token throughput and request throughput, plus profile_export_genai_perf.json and .csv artefacts. The analyze subcommand automatically sweeps concurrencies from 1 to 256 (powers of 2) and generates a summary analyze_export_genai_perf.csv. GenAI-Perf was declared retired in April 2026 (replaced by AIPerf); it remains the reference benchmark for historical runs and documented comparisons, and the NVIDIA tool with the most public documentation for LLMs before 2026.

What GenAI-Perf is and where it fits

GenAI-Perf is a command-line tool for measuring the throughput and latency of generative AI models served through an inference server. It is part of the triton-inference-server/perf_analyzer GitHub repository and is distributed as a pip package (genai-perf) and inside the Triton SDK container (nvcr.io/nvidia/tritonserver:YY.MM-py3-sdk).

Position in the NVIDIA ecosystem:

Relationship with perf_analyzer: GenAI-Perf internally uses Triton’s perf_analyzer binary to generate load and measure latencies. perf_analyzer is the low-level load generator; GenAI-Perf is the high-level layer that adds prompt synthesis, OpenAI payload construction and the LLM-specific metrics (TTFT, ITL, sequence lengths).

Installation:

# Option 1: pip (requires CUDA 12, Ubuntu 24.04, Python 3.10+)
pip install genai-perf

# Option 2: Triton SDK container (recommended for reproducibility)
export RELEASE="25.01"
docker run -it --net=host --gpus=all \
 nvcr.io/nvidia/tritonserver:25.01-py3-sdk
# Inside the container:
genai-perf --help

Metrics GenAI-Perf produces

The metrics cover the full cycle of a streaming generation request. All of them are reported with avg, min, max, P75, P90 and P99 except the throughput ones (avg only).

Metrics glossary

Metric	Symbol	Exact definition (GenAI-Perf)	Unit
Time to First Token	TTFT	Time from sending the request to receiving the first token (includes queueing, prefill and network)	ms
Inter-Token Latency	ITL (= TPOT)	$\frac{e2e\_latency - TTFT}{output\_tokens - 1}$ — decode phase only, excludes the first token	ms/token
Request Latency	e2e_latency	$e2e = TTFT + Generation\_time$ — from the first request to the last response	ms
Output Token Throughput	OTT	$\frac{total\_output\_tokens}{T_y - T_x}$ where $T_x$ = first request, $T_y$ = last token received	tok/s
Request Throughput	RPS	$\frac{total\_completed\_requests}{T_y - T_x}$	req/s
Input Sequence Length	ISL	Mean length in tokens of the input prompt	tokens
Output Sequence Length	OSL	Mean length in tokens of the generated response	tokens

Methodological note on ITL versus LLMPerf: GenAI-Perf excludes TTFT from the ITL calculation (pure decode only). LLMPerf includes TTFT in its ITL mean. The numbers are not directly comparable.

Methodological note on TPS versus LLMPerf: GenAI-Perf measures throughput between the first request and the last token received ($T_y - T_x$). LLMPerf uses the total duration of the benchmark including prompt generation and response storage; in a concurrency-1 scenario, that difference can amount to up to 33% in the reported throughput figure.

Console output table (real example, TRT-LLM backend)

The table GenAI-Perf prints for a profile with ISL=200, OSL=100, concurrency=1:

 NVIDIA GenAI-Perf | LLM Metrics
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┳━━━━━━━━┓
┃ Statistic ┃ avg ┃ min ┃ max ┃ p99 ┃ p90 ┃ p75 ┃
┡━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━╇━━━━━━━━┩
│ Time to first token (ms) │ 13.68 │ 11.07 │ 21.50 │ 18.81 │ 14.29 │ 13.97 │
│ Inter token latency (ms) │ 1.86 │ 1.28 │ 2.11 │ 2.11 │ 2.01 │ 1.95 │
│ Request latency (ms) │ 203.70 │ 180.33 │ 228.30 │ 225.45 │ 216.48 │ 211.72 │
│ Output sequence length │ 103.46 │ 95.00 │ 134.00 │ 122.96 │ 108.00 │ 104.75 │
│ Input sequence length │ 200.00 │ 200.00 │ 200.00 │ 200.00 │ 200.00 │ 200.00 │
│ Output token throughput (per sec) │ 504.02 │ N/A │ N/A │ N/A │ N/A │ N/A │
│ Request throughput (per sec) │ 4.87 │ N/A │ N/A │ N/A │ N/A │ N/A │
└───────────────────────────────────┴────────┴────────┴────────┴────────┴────────┴────────┘

(Source: docs.nvidia.com/deeplearning/triton-inference-server — LLM tutorial)

Invocation against OpenAI-compatible endpoints

`profile` mode: one load point

The profile subcommand measures a fixed operating point (concurrency or request-rate, synthetic ISL/OSL).

Example 1: vLLM endpoint (chat, streaming)

# 1. Bring up the vLLM server
docker run -it --net=host --rm --gpus=all \
 vllm/vllm-openai:latest \
 --model meta-llama/Llama-3.1-70B-Instruct \
 --dtype float16

# 2. Profile with GenAI-Perf
genai-perf profile \
 -m meta-llama/Llama-3.1-70B-Instruct \
 --endpoint-type chat \
 --streaming \
 --concurrency 16 \
 --synthetic-input-tokens-mean 1024 \
 --synthetic-input-tokens-stddev 128 \
 --output-tokens-mean 256 \
 --output-tokens-stddev 0 \
 --output-tokens-mean-deterministic \
 --request-count 200 \
 --warmup-request-count 20 \
 --tokenizer meta-llama/Llama-3.1-70B-Instruct

Example 2: completions endpoint (no streaming)

genai-perf profile \
 -m meta-llama/Llama-3.1-70B-Instruct \
 --endpoint-type completions \
 --concurrency 32 \
 --synthetic-input-tokens-mean 512 \
 --synthetic-input-tokens-stddev 64 \
 --output-tokens-mean 128 \
 --request-count 200 \
 --warmup-request-count 20

Example 3: NVIDIA NIM (TRT-LLM backend endpoint, via Triton)

genai-perf profile \
 -m meta/llama-3.1-70b-instruct \
 --backend tensorrtllm \
 --streaming \
 --concurrency 8 \
 --synthetic-input-tokens-mean 2048 \
 --synthetic-input-tokens-stddev 0 \
 --output-tokens-mean 512 \
 --output-tokens-mean-deterministic \
 --request-count 100 \
 --warmup-request-count 10 \
 --generate-plots

Example 4: request-rate instead of concurrency (Poisson)

genai-perf profile \
 -m meta-llama/Llama-3.1-70B-Instruct \
 --endpoint-type chat \
 --streaming \
 --request-rate 5.0 \
 --synthetic-input-tokens-mean 1024 \
 --output-tokens-mean 256 \
 --request-count 300 \
 --warmup-request-count 30

Main flags and what they do

Flag	Typical values	Function
`--endpoint-type`	`chat`, `completions`, `embeddings`	Type of OpenAI-compatible endpoint
`--streaming`	(boolean)	Enables SSE streaming; required to measure real TTFT and ITL
`--concurrency`	1, 8, 16, 32, 64	Number of concurrent requests maintained; GenAI-Perf guarantees N active at all times
`--request-rate`	1.0, 5.0, 10.0	Constant arrival rate (req/s); does not guarantee N active
`--synthetic-input-tokens-mean`	128–8192	Mean of synthetic ISL tokens
`--synthetic-input-tokens-stddev`	0–512	Standard deviation of the ISL (0 = fixed)
`--output-tokens-mean`	64–2048	Mean of target OSL tokens
`--output-tokens-mean-deterministic`	(boolean)	Sets the minimum output tokens = target mean (more precise with TRT-LLM)
`--request-count`	100–2000	Number of requests to benchmark
`--warmup-request-count`	10–50	Warm-up requests discarded from the metrics
`--generate-plots`	(boolean)	Generates PNG plots of TTFT vs ISL, ITL vs token position, etc.
`--tokenizer`	HF model id	Tokenizer for counting tokens; mandatory when ISL/OSL matter
`--backend`	`tensorrtllm`, `vllm`	Triton backend (for serving directly via Triton without an OpenAI endpoint)
`--extra-inputs`	`ignore_eos:true`	Extra request parameters; `ignore_eos:true` guarantees a consistent OSL

Note on --concurrency versus --request-rate: NVIDIA recommends using --concurrency. With --request-rate, if the rate exceeds the engine’s throughput, the queue grows without bound and the metrics stop being stable. With --concurrency, there are always exactly N active requests, which gives a clean latency-throughput curve.

Output artefacts

GenAI-Perf dumps all results into an artifacts/ directory:

Artefact	Format	Content
`profile_export_genai_perf.json`	JSON	Full metrics (avg, min, max, P75, P90, P99) + CLI arguments used
`profile_export_genai_perf.csv`	CSV	Exported console tables, ready to import into Excel/pandas
`profile_export.json`	JSON	Raw perf_analyzer data (per-request traces)
`inputs.json`	JSON	Synthetic payloads sent (for reproducibility)
PNG plots	PNG	TTFT analysis, Request latency, TTFT vs ISL, ITL vs position, ISL vs OSL (with `--generate-plots`)

The `analyze` subcommand: automatic concurrency sweep

The analyze subcommand sweeps multiple values of a parameter in a single command and generates a summary CSV.

Concurrency sweep (the most used)

# Sweeps concurrencies 1, 2, 4, 8, 16, 32, 64, 128, 256
genai-perf analyze \
 -m meta-llama/Llama-3.1-70B-Instruct \
 --endpoint-type chat \
 --streaming \
 --sweep-type concurrency \
 --sweep-range 1:256 \
 --synthetic-input-tokens-mean 1024 \
 --output-tokens-mean 256 \
 --request-count 200 \
 --warmup-request-count 20

Request-rate sweep

# Sweeps 2, 4, 6, 8, 10, 12 req/s
genai-perf analyze \
 -m meta-llama/Llama-3.1-70B-Instruct \
 --endpoint-type chat \
 --streaming \
 --sweep-type request_rate \
 --sweep-list 2,4,6,8,10,12 \
 --synthetic-input-tokens-mean 1024 \
 --output-tokens-mean 256 \
 --request-count 200 \
 --warmup-request-count 20

ISL sweep (to study the effect of prefill)

genai-perf analyze \
 -m meta-llama/Llama-3.1-70B-Instruct \
 --endpoint-type chat \
 --streaming \
 --sweep-type input_sequence_length \
 --sweep-list 256,512,1024,2048,4096 \
 --concurrency 16 \
 --output-tokens-mean 256 \
 --request-count 200 \
 --warmup-request-count 20

Artefact structure of `analyze`

For a sweep with --sweep-type concurrency --sweep-range 1:32:

artifacts/
llama70b-openai-chat-concurrency1/
inputs.json
profile_export.json
profile_export_genai_perf.json
profile_export_genai_perf.csv
llama70b-openai-chat-concurrency2/
...
llama70b-openai-chat-concurrency4/
...
[...]
analyze_export_genai_perf.csv ← summary of every scenario
checkpoint.json ← allows interrupted sweeps to be resumed

Summary CSV format

Config Name,Concurrency,ISL,p99 TTFT (ms),p99 ITL (ms),p99 Request Latency (ms),Avg. OTT (tok/s),RPS
llama70b_run_0,1,1024,45.2,6.8,1823.4,128.3,0.55
llama70b_run_1,4,1024,48.1,7.1,1901.2,510.7,2.18
llama70b_run_2,8,1024,61.3,7.9,2142.5,989.2,4.12
llama70b_run_3,16,1024,98.7,9.4,2891.3,1741.6,6.88
llama70b_run_4,32,1024,284.1,14.2,4320.8,2134.2,7.14
llama70b_run_5,64,1024,892.4,28.7,8741.2,2251.8,7.23

The summary CSV also includes a second table with GPU metrics (P99 power, energy, utilisation, memory) when they are captured via Triton’s telemetry URLs (--server-metrics-urls http://localhost:8002/metrics).

Comparison table: GenAI-Perf versus GuideLLM versus LLMPerf versus vLLM bench

The choice of tool changes the number. They are not comparable with each other without adjustment.

Dimension	GenAI-Perf	GuideLLM	LLMPerf	vLLM bench serve
Origin / maintainer	NVIDIA / Triton team (retired Apr 2026)	Red Hat / vLLM project	Anyscale / Ray	vLLM project
Class	multi-process load generator	multi-process load generator	multi-process load generator	single-process micro-bench
Supported endpoints	OpenAI-compatible + KServe + native Triton	OpenAI-compatible	OpenAI-compatible	native vLLM
Main load mode	fixed concurrency N (recommended) or constant request-rate	synchronous, concurrent, poisson, throughput, automatic sweep	batches of N concurrent (with a draining period at the end)	request-rate, concurrency
Draining period	NO — guarantees N active at all times	NO (poisson)	YES — at the end of each batch the system empties, concurrency drops to 0	N/A
Automatic sweep	`analyze` (concurrency, request-rate, ISL, OSL)	`--rate-type sweep` (idle to saturation, 10 rounds)	manual (several runs)	`vllm bench sweep serve`
LLM metrics	TTFT, ITL, e2e latency, OTT, RPS, ISL/OSL	TTFT, ITL/TPOT, throughput, goodput under SLO	TTFT, ITL (includes TTFT in the mean), TPS (total benchmark duration)	TTFT, TPOT, throughput
ITL difference	excludes TTFT: $(e2e - TTFT) / (N_{tok}-1)$	excludes TTFT (same as GenAI-Perf)	includes TTFT in the mean	excludes TTFT
TPS difference	$total\_tokens / (T_y - T_x)$	throughput + goodput under SLO	$total\_tokens / (T_{end} - T_{start})$ — up to 33% lower	$total\_tokens / (T_y - T_x)$
Goodput / SLO	NO	YES (the key differentiator)	NO	NO
Exports	JSON + CSV + PNG + checkpoint	JSON + YAML + CSV + interactive HTML	JSON	console / JSON
Built-in warm-up	YES (`--warmup-request-count`)	YES (warm-up requests)	NOT native	YES
Real datasets	OpenOrca, CNN DailyMail, JSONL file, moon_cake	file with a traffic distribution	JSONL file	sharegpt, random
GPU telemetry	YES (via Triton’s server-metrics-urls → summary CSV)	NOT native (integrate DCGM separately)	NO	NO
When to choose it	fine profiling of one operating point, comparing configs with the same harness, integration with NIM/Triton	SLO-driven sweep, finding the knee, sizing replicas	quick endpoint validation in the Ray ecosystem	iterating on vLLM flags

Fair-comparison rule: never cross numbers from different tools for the same conclusion. If vLLM and TRT-LLM are being compared, use GenAI-Perf against both, with the same ISL/OSL, the same concurrency and the same --warmup-request-count. The tool is the constant; the engine is the variable.

Profiling on 4×H100 SXM 80 GB: a full sweep example

Generic reference hardware: 4×H100 SXM 80 GB NVLink, model Llama-3.1-70B-Instruct FP16, ISL=1024 tokens, OSL=256 tokens, vLLM endpoint.

Illustrative result of a concurrency sweep (1→64)

Concurrency	TTFT P50 (ms)	TTFT P99 (ms)	ITL P50 (ms)	ITL P99 (ms)	OTT (tok/s)	RPS
1	38	45	6.4	7.1	130	0.51
4	40	51	6.8	8.2	510	2.00
8	46	62	7.3	9.4	980	3.83
16	68	98	8.9	12.1	1,720	6.72
32	145	284	13.7	21.4	2,130	8.32
48	380	740	22.3	38.6	2,280	8.90
64	890	1,820	41.2	78.3	2,340	9.14

How to read it: throughput saturates around concurrency 48–64, while TTFT P99 already exceeds 500 ms from concurrency 32 onwards. The knee (the last point with TTFT P99 < 500 ms) is at concurrency 16 with 1,720 useful tok/s. Anyone reporting “2,340 tok/s” (concurrency 64) is describing the maximum throughput, at a point where the system meets no interactive chat SLO at all.

Formulas for the key metrics

$$ \text{ITL} = \frac{e2e\_latency - TTFT}{N_{output\_tokens} - 1} $$ $$ \text{OTT} = \frac{\sum output\_tokens}{T_y - T_x} $$ $$ \text{RPS} = \frac{N_{requests}}{T_y - T_x} $$

where $T_x$ is the timestamp of the first request sent and $T_y$ the timestamp of the last token received from the last request.

Synthetic ISL/OSL profiling: what to configure for each use case

The ISL/OSL parameters determine which engine of the system is stressed: a long ISL stresses prefill (KV-cache, TTFT); a long OSL stresses decode (ITL, memory bandwidth).

Use case	Typical ISL	Typical OSL	`--synthetic-input-tokens-mean`	`--output-tokens-mean`
Short interactive chat	~300 tok	~100 tok	300	100
Code copilot	~800 tok	~200 tok	800	200
Document summarisation	~2000 tok	~256 tok	2000	256
RAG with long context	~4096 tok	~512 tok	4096	512
Batch without latency	~512 tok	~1024 tok	512	1024

The --extra-inputs ignore_eos:true option disables the engine’s EOS token, forcing exactly the requested tokens to be generated. Essential for measuring a consistent OSL on TRT-LLM; optional on vLLM.

Honest methodology: how not to measure badly

1. Warm-up is mandatory

Without warm-up, the first requests include the KV-cache loading latency, the engine’s autotuning and the initialisations. GenAI-Perf applies a sliding window to detect stability (max/min ratio within a margin over the last 3 measurements), but the explicit --warmup-request-count discards the first N requests from the metrics. Recommended minimum value: 10 requests (or 5% of the total if the total is large).

2. The concurrency knee: extend the sweep until you see it

Throughput saturates once the engine’s effective max batch size is reached. If the sweep stops before saturation is reached, the “maximum capacity” reported is the last concurrency tested, not the real knee. The rule: the sweep must reach a point where TTFT P99 has doubled relative to the low-load value (or where OTT stops growing by more than 5% between steps). Only then is the knee visible.

For a sweep with --sweep-range 1:256, GenAI-Perf sweeps 1, 2, 4, 8, 16, 32, 64, 128, 256 (powers of 2). On most 70B models over 4×H100, saturation occurs between concurrency 32 and 64; the range 1:128 covers the knee with room to spare.

3. Concurrency versus request-rate: choosing correctly

Mode	Behaviour	When to use
`--concurrency N`	Always N active requests; when one finishes, the next is sent	Clean latency-throughput curve; comparison between configs
`--request-rate R`	1 request is sent every 1/R seconds; the queue can grow without bound	Simulating traffic with arrivals at a constant rate

NVIDIA recommends --concurrency for benchmarking. With --request-rate, if R exceeds the engine’s maximum throughput, the queue grows indefinitely and the metrics become unstable.

4. Single-process versus multi-process bias

GenAI-Perf is multi-process: it maintains N concurrent requests using perf_analyzer’s internal manager. There is no risk of client saturation. This sets it apart from vllm bench serve (single-process), which can saturate on the client at high concurrency and give inflated throughput metrics.

5. `ignore_eos` and an inconsistent OSL

Without --extra-inputs ignore_eos:true, the engine may generate fewer tokens than requested (if the model produces an EOS earlier). This makes the real OSL vary between requests and the ITL become inconsistent between runs. For reproducible benchmarks with a controlled OSL: always enable ignore_eos:true.

6. The correct tokenizer

The --tokenizer flag must point to the model being served. If it is omitted, GenAI-Perf uses a default tokenizer that can give different ISL/OSL in the real model’s tokens, making the throughput numbers in tok/s not comparable between models with different vocabularies.

7. One run = a data point; three or more = a number

Variance between runs under identical conditions can be 5–10% on tail latency metrics (P99). A single profile is not a defensible number; the mean of three runs with the same config is.

Connection with the cost and energy track

GenAI-Perf can capture GPU metrics in the same summary CSV if --server-metrics-urls http://localhost:8002/metrics is passed (Triton’s metrics endpoint, which exposes power and utilisation via DCGM). The analyze summary CSV then includes a second table with P99 power (W), energy (MJ), utilisation (%) and memory (GB) per GPU and per scenario.

With that data, a single sweep yields the three axes of the scorecard:

Axis	Source	Metric
Performance	GenAI-Perf	OTT (tok/s), TTFT P99, ITL P99
Energy	DCGM via server-metrics-urls	Power (W) → $J/token = W / OTT$
Cost	node price	EUR/hour → $EUR/token = EUR_{hour} / (OTT \times 3600)$

The formula for cost per million tokens:

$$ CPM = \frac{EUR_{hour}}{OTT \times 3{.}6 \times 10^3} $$

where $OTT$ is the output token throughput in tok/s and $CPM$ is the cost per million tokens in EUR.

It connects with GPU observability with DCGM for the power capture, and with anatomy of an LLM request for the breakdown of phases (prefill versus decode) that TTFT and ITL quantify.

Status 2026: GenAI-Perf → AIPerf

NVIDIA announced in April 2026 that GenAI-Perf moves to passive maintenance mode (no new features) and that the active successor is AIPerf (github.com/ai-dynamo/aiperf). AIPerf adds automatic detection of the saturation point (estimatedCapacity) and is integrated into the NVIDIA Dynamo pipeline.

For new benchmarks in production: migrate to AIPerf. To reproduce historical runs documented with GenAI-Perf, or to use NVIDIA’s NIM benchmarking artefacts (which use GenAI-Perf): stay on GenAI-Perf 25.x.

The article LLM benchmark tools, one by one covers the full comparison including AIPerf. The article GuideLLM: SLO validation under load details the SLO-driven sweep that GenAI-Perf does not do natively. For the general methodology context: LLM benchmarking frameworks, state of the art.

Sources

NVIDIA · GenAI-Perf — README oficial — https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/genai-perf/README.html
NVIDIA · Tutorial LLM con GenAI-Perf (comandos y tablas de salida reales) — https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/genai-perf/docs/tutorial.html
NVIDIA · GenAI-Perf Analyze subcommand (sweep, CSV resumen, checkpoint) — https://docs.nvidia.com/deeplearning/triton-inference-server/user-guide/docs/perf_analyzer/genai-perf/docs/analyze.html
NVIDIA · NIM LLM Benchmarking — Métricas (definiciones TTFT, ITL, TPS, RPS, fórmulas exactas) — https://docs.nvidia.com/nim/benchmarking/llm/latest/metrics.html
NVIDIA · NIM LLM Benchmarking — Parámetros y buenas prácticas (ISL/OSL, concurrencia vs request-rate, ignore_eos) — https://docs.nvidia.com/nim/benchmarking/llm/latest/parameters.html
NVIDIA Technical Blog · Measuring Generative AI Model Performance Using NVIDIA GenAI-Perf and an OpenAI-Compatible API — https://developer.nvidia.com/blog/measuring-generative-ai-model-performance-using-nvidia-genai-perf-and-an-openai-compatible-api/
GitHub · triton-inference-server/perf_analyzer (genai-perf) — https://github.com/triton-inference-server/perf_analyzer/blob/main/genai-perf/README.md
GitHub · ray-project/llmperf (métricas y diferencias metodológicas) — https://github.com/ray-project/llmperf
PyPI · nvidia-genai-perf-eval — https://pypi.org/project/nvidia-genai-perf-eval/
Macnica · Benchmarking LLM Applications Part 1: What is GenAI-Perf? — https://www.macnica.co.jp/en/business/semiconductor/articles/nvidia/145977/

Parameter	Symbol	Example
Requests per day (average)	\(D\)	5,000,000
Input tokens per request (average)	\(L_{\text{in}}\)	512
Output tokens per request (average)	\(L_{\text{out}}\)	256
Peak factor (peak hour vs average ratio)	\(k_{\text{peak}}\)	1.8
Hours per day with significant traffic	\(H\)	16

Technical figure	How it is obtained	Use in TCO	Use in regulatory reporting
\(P_{\text{IT}}\) (W)	DCGM / NVML / IPMI	Energy cost formula	\(E_{\text{IT,annual}}\) for PUE, WUE, ERF, REF
Throughput (tok/s)	Engine metrics (vLLM, TGI)	€/token, Wh/token	Energy intensity of inference
PUE (measured)	Meters on PDUs and IT equipment (ISO/IEC 30134-2)	Multiplier in the TCO formula	Mandatory ReportENER KPI
Annual \(I_{\text{grid}}\) (gCO₂/kWh)	REE / MITECO / national factors	gCO₂/token for carbon-aware	Emission factor for location-based Scope 2 ESRS E1-6
Hourly \(I_{\text{grid}}\)	ElectricityMaps / Carbon Aware SDK	Carbon-aware scheduling	Location-based Scope 2 with hourly granularity (best practice)
\(W_{\text{in}}\) (m³)	Datacenter water meters	Cost of water (where applicable)	WUE for ReportENER
\(E_{\text{reuse}}\) (kWh)	Heat export meters	Possible waste heat revenue	ERF for ReportENER
\(E_{\text{res,tot}}\) (kWh)	Energy bills with a renewable attribute / GOs	Energy cost with a PPA	REF for ReportENER; market-based Scope 2 for ESRS E1-6
Hardware embodied carbon (gCO₂eq)	Supplier LCA (AWS methodology, where available)	—	Scope 3 cat. 1 for ESRS E1-6

Symbol	Meaning	Units
\(E\)	Energy consumed by the software system (servers, network, user devices)	kWh
\(I\)	Carbon intensity of the energy consumed (location-based or another declared basis)	gCO₂eq/kWh
\(M\)	Embodied carbon of the hardware, amortised by time of use and utilisation percentage	gCO₂eq
\(R\)	Functional unit (the denominator that normalises the rate)	per API call, per token, per user, etc.

AI on lo0 — Tech Blog

Anatomy of an LLM request in production, May 2026: a tour of the six stages following a single request

TL;DR

You are here: every stage at once

The analogy: forensic analysis of a request

The scenario: a multi-tenant support chatbot for regulated customers

Step 0 — The frame: the answer the user sees

Rewinding backwards: what was already in place before the request

t = −90 days — The previous Retrain stage closes the earlier cycle

t = −60 days — The Data stage: the enriched dataset is versioned and enters circulation

t = −45 days — The Tune stage: the customer_support_v7 adapter is trained

t = −38 days — The Eval stage: the v7 adapter goes through eval gates

t = −31 days — The Deploy stage: the v7 adapter enters serving

Moving forward: the user’s request crosses the system

t = 0 ms — Entry through the gateway

t = 8 ms — Pulling the versioned prompt

t = 12 ms — RAG retrieval

t = 40 ms — Building the final payload

t = 45 ms — Prefill

t = 580 ms — Decode (streaming)

t = 4,200 ms — Complete answer, root span closed

In parallel: Observe is watching

The feedback: the loop closes

Insertion into feedback_signals

Triage by root cause

The next cycle picks it up

What goes in each trace: identity and traceability

Synthesis diagram: how the pieces fit

The on-premise stack applied

Ten bridges between stages where the system breaks

How to walk through the blog

What we have not covered (yet)

See also

References

Langfuse v4, day 2 (2 of 9): putting LangGraph in front, and what one agent turn costs in observations

TL;DR

You are here: OBSERVE, day 2

The analogy: the file that gets photocopied whole at every signature

Which agentic platform is the most used

How LangGraph gets into Langfuse

The observation type is decided by the node name

What a turn costs, measured

What this means for ClickHouse

Interrupts: two traces where there should be one

Thread, session, user, and what is only read at the root

The filter that drops your own spans

The model and the cost with a gateway in the middle

How this looks in code

Agent instrumentation checklist

Traps

The series: the nine articles

See also

Sources

Completing Keycloak for MCP: the protected resource no SDK hands you ready-made, and the enterprise extension that skips the whole flow

TL;DR

You are here: the resource side

The analogy: the visa and the border checkpoint

Part 1. The split of obligations

What is new in the revision and affects the setup

Part 2. What the SDKs give, and what they do not

Protected resource metadata

Audience validation

The authentication challenge

Scopes, which are not per tool

Which revision each one announces

Part 3. Building the protected resource

Part 4. The hop from the gateway to the MCP server

What the gateway does today

Part 5. The enterprise extension, which changes the picture

And here comes the problem

What is already in the gateway, undocumented

Part 6. Per-tool authorisation

Part 7. What still has no standard

Part 8. Who covers what

Reference configuration

Checklist

Traps and things that are not what they look like

Closing

See also

Sources

The `litellm_jwtauth` fields that decide security