Fundamentals on lo0 — Tech Blog

Anatomy of an LLM request in production, May 2026: a tour of the six stages following a single request

Fri, 22 May 2026 16:00:00 +0200

TL;DR

Across several series the blog has laid out the pieces that hold up an LLM system in production: the Data stage (dataset versioning, ingestion and vector stores, RAG over Kafka), the Tune stage (continuous fine-tuning), the Eval stage (evals as the layer after tracing, guardrails and safety), the Deploy stage (KV cache, PagedAttention, disaggregated serving, multi-tenant GPU cluster, vLLM on Kubernetes, LLM operators on K8s), the Observe stage (tracing with AgentSight, MCP observability, eBPF + drift), the Retrain stage (closing the feedback → dataset → adapter loop), and the cross-cutting components (prompt versioning and data versioning). What is missing is joining them up: seeing a single request cross every piece in order, in one coherent story. That is what this post does. We take a specific request from a multi-tenant support chatbot, rewind it back to the data that trained the adapter serving it today, follow it forward through serving, watch it reach the feedback store when the user marks a thumbs-down, and leave it as the seed of the next quarterly retrain cycle. The route works as a mental map and as an integrator’s guide: the system does not hold up if a single one of the seven pieces (six stages + two cross-cutting) is broken or absent. The practical lesson of the tour is not a new one. It is that everything is connected, that local measurements lie when taken in isolation, and that the real cost of not operating one stage well is paid by another stage further down the line.

You are here: every stage at once

Unlike previous posts, where the mini-map marked a single active box, this one covers the whole pipeline. It is the only post on the blog that activates the six stages and the two cross-cutting components simultaneously, because we follow a real request that crosses all of them.

The analogy: forensic analysis of a request

When an air accident happens, the forensic analysis does not limit itself to the last seconds of the flight. The investigation team rewinds to the maintenance of the previous six months, the manufacturer’s protocols, the pilot’s record, the weather briefing, the controller’s decisions, the incident history on the same model. The conclusion is rarely “the wing broke”; it is “the wing broke because an inspection protocol written a certain way did not detect microcracks that the 2014 calculation model did not consider critical and that were critical beyond a certain fatigue cycle”.

When an LLM request in production fails or succeeds, there is also a long causal chain behind it. The answer the user sees is the last frame; what determined it starts months earlier and branches across six operational stages. If you only look at the last frame, you attribute the result to the model. If you look at the whole chain, you see that the model is one of twelve factors and rarely the most important.

This post does that forensic analysis, but the other way round: instead of starting from a failure and rewinding, we start from a specific request that works and break down what had to happen for it to work, and what will happen to it afterwards. It is a guided tour, not an incident diagnosis. But the mental discipline is the same: no stage is autonomous, and understanding the system means understanding the bridges between stages, not just the boxes.

The scenario: a multi-tenant support chatbot for regulated customers

For the tour we use a concrete realistic scenario, representative enough that the observations carry over to most serious deployments in May 2026. It is an LLM customer support SaaS product, sold to several corporate customers (multi-tenant) in regulated sectors (banking, insurance, health). The product:

Accepts natural-language questions through a chat embedded in the customer’s website.
Retrieves relevant fragments from the customer’s internal knowledge base (product documents, contractual terms, FAQs).
Answers citing the fragment used, in the user’s language, with a tone adapted to the brand.
If the question requires an action (cancel, change plan, open a case), it invokes a tool from the catalogue exposed via MCP, with user confirmation.
It is contractually obliged not to invent customer data, to respect the tenant’s style guide, and to meet ENS / NIS2 compliance on data in the EU.

The infrastructure is on-premise over Kubernetes (RKE2 with the Cilium CNI, in the pattern covered in Cilium BGP and CNI networking). The GPU plane is a generic 4×H100 SXM cluster (80 GB each, NVLink) for serving and production training, plus an RTX 4090 (24 GB) for development, offline evals and ad-hoc re-embeddings.

The request we are going to follow:

User (end customer, Spanish, premium plan with an insurer): “How do I cancel my premium subscription? I have been paying for three months and I have not used half of the cover.”

At first sight, a simple question. Let us see what is behind it.

Step 0 — The frame: the answer the user sees

At 11:47:32 on a Tuesday, the user sends the message. 4.2 seconds later, this appears in the UI:

To cancel your premium subscription, follow these steps: (1) Sign in to the customer portal with your ID number and password. (2) Go to My policies → Premium Life 2024. (3) Press Request cancellation. (4) Select the reason (in your case, “not using the product”) and confirm. You will receive a confirmation email within 24-48 h.

Before you cancel, let me remind you that your plan includes a free annual medical check-up and 24/7 travel assistance, cover that is lost on cancellation. If you like, I can help you review your use of the cover over the past three months to see whether switching to a more tailored plan suits you better than cancelling.

Source: [Cancellation policy, section 4.2 of the 2024 General Terms]

After reading it, the user marks a thumbs-down and writes in the form: “the answer is correct but the tone is too salesy; I only wanted to know how to cancel”.

That sequence, question, answer, thumbs-down with structured feedback, is the last visible frame. Let us go backwards to understand what had to happen for the answer to come out that way.

Rewinding backwards: what was already in place before the request

Before the user typed, the system already had a model loaded in serving, an active prompt labelled production, an up-to-date vector index, a versioned dataset from the last fine-tuning, and a golden eval set that validated the promotion. Each of those artefacts got there through a process. We go through four jumps backwards.

t = −90 days — The previous Retrain stage closes the earlier cycle

Three months ago, during a quarterly Retrain cycle, two things happened. The first: the support team reviewed the feedback accumulated over the previous six months and saw a pattern, the model answering premium users in an excessively formal tone, with users reporting that it “feels robotic”. The second: a one-off incident (a customer cancelling over an answer perceived as curt) triggered an incident-driven mini-cycle.

The process, covered in detail in the Retrain post, followed five sub-processes:

Feedback capture — explicit thumbs-down plus implicit feedback (abandonments, retries) accumulated in a feedback_signals table in Postgres, all of them with a trace_id that allows rewinding to the exact context.
Triage by root cause — the “curt tone” incident cluster was categorised as a prompt issue (it was not the model answering badly, it was the system prompt asking for an overly formal register). One sub-cluster was a model issue (in some cases the model dug its heels in even with a warmer prompt).
Dataset enrichment — the team manually annotated 280 cases where the model was too curt, labelled with the reference answer (“how it should have answered”). Double annotation on the critical 20%; cases with a quality score below 4 were left out.
Cadence decision — the incident was treated as incident-driven; the rest of the quarterly Retrain followed the calendar.
Promotion — the new customer_support_v7 adapter went through eval gates against customer_support_v6, a 5% canary for a week, and was promoted when the golden set metrics showed a stable improvement in the “tone / clarity” segment with no regressions elsewhere.

The result: the adapter active in production when the user sent the Step 0 request is customer_support_v7, trained on the enriched dataset enriched_retrain_2026_q1 version 3, with double lineage back to the original incident.

t = −60 days — The Data stage: the enriched dataset is versioned and enters circulation

Immediately after Retrain, the Data stage of the LLMOps pipeline does its work. Three critical operations, covered in detail in the data versioning post:

Immutable versioning of the enriched dataset with DVC, sha256 hash propagated to the registry. The identifier (enriched_retrain_2026_q1, v3, sha256:9af...) becomes the luggage tag that will travel through the coming stages.
Schema contract validated by CI: every row meets the JSON Schema of the entry the trainer expects (example_id, input.user_query, input.retrieved_context, expected_output, rubric, segment, difficulty). A validation fails in CI if any row breaks the contract.
Holdout segregation check: the normalised sha256 hash of each input is compared against every hash in the active golden eval set (customer_support_golden_v12). Zero overlaps = the dataset does not contaminate the eval. Had there been a single one, CI would have blocked the merge.

In parallel, the RAG corpus (product manuals, FAQs, general terms of the insurer tenant) is kept alive. The ingestion pipeline keeps capturing changes from the customer’s CMS: a new section of the cancellation policy was modified in February and reindexed in Qdrant. As the post about RAG over Kafka explains, the corpus is not retrained with every change: only the delta is re-embedded, and lakeFS keeps a branch of the embeddings bucket with the new version. The branch is merged into main when recall@10 over a representative set of queries stays above the threshold (0.78 in this system).

t = −45 days — The Tune stage: the customer_support_v7 adapter is trained

Three weeks after closing the dataset, training of the new LoRA adapter starts. As the continuous fine-tuning post details, the production pattern in 2026 avoids retraining the base model, which is expensive, slow and irreversible, and favours a LoRA adapter over a stable base model (in this system, Llama 3 70B-instruct quantised to INT8 for serving). The training:

Runs over 4 of the H100s (NVLink, tensor parallel) for ~6 hours.
Uses transformers + PEFT + bitsandbytes, monitored by MLflow.
Every step records the dataset_id, dataset_version and dataset_hash as an input artifact in MLflow.
The output, a customer_support_v7.safetensors file of ~280 MB with the LoRA weights, is uploaded to MinIO with its own hash, and MLflow records model_id, model_version, parent_dataset.

At this point the lineage chain is closed for this leg:

enriched_retrain_2026_q1, v3, sha256:9af...
│
▼
mlflow run train, run_id: 0xa721...
│
▼
customer_support_v7, sha256:5c1...

t = −38 days — The Eval stage: the v7 adapter goes through eval gates

The freshly trained adapter is not promoted. It goes through an eval suite covered in detail in the post about evals. The golden eval set, customer_support_golden_v12, 850 human-curated examples with an inter-annotator kappa of 0.81, is run against two models: the candidate v7 adapter and the v6 currently in production. The metrics:

Metric	v6 (prod)	v7 (cand.)	Threshold
Faithfulness to the RAG fragment	0.87	0.89	≥ 0.82
Toxicity (low is good)	0.012	0.011	≤ 0.02
“Warm but professional” tone (judge LLM)	0.71	0.84	≥ 0.78
Format compliance (structured markdown)	0.94	0.93	≥ 0.90
Helpful-but-not-pushy (judge LLM)	0.66	0.79	≥ 0.75
Latency p95 (ms)	2,840	2,910	≤ 3,500

To this is added the guardrails and safety suite covered in the guardrails post: jailbreak resistance, PII leakage detection, prompt injection over MCP tools. The v7 improves on safety in two metrics and ties on the rest.

The v7 enters a 5% traffic canary for 7 days, under close monitoring. At the end of the canary, the online metrics confirm what the offline ones anticipated: an improvement in tone and helpfulness, equivalent latency, no new failure modes. Promotion approved. The v7 moves to the production label.

t = −31 days — The Deploy stage: the v7 adapter enters serving

The customer_support_v7 adapter is promoted to the serving cluster. Three pieces covered in separate posts come into play.

vLLM as the inference engine. The engine lives on Kubernetes, deployed via a dedicated Operator, as the post about LLM operators and the post about vLLM on K8s explain. The operator is responsible for detecting the new adapter in the registry, hot-loading it without restarting the engine (a native vLLM capability with --enable-lora), and directing traffic based on the label.

Disaggregated serving. As the post about disaggregated serving details, the system separates prefill (compute-intensive, throughput-bound) and decode (memory-intensive, latency-bound) into different GPU pools. The user’s request, when it arrives, prefills in a specialised pod and decodes in another, communicating over NVLink plus a shared KV cache fabric.

Multi-tenant GPU cluster. The H100 cluster serves several tenants, not just the insurer from Step 0. As the post about the multi-tenant cluster explains, isolation is realised on four planes: the Kubernetes namespace, ACLs over adapters (only the tenant’s namespace loads its LoRAs), partitioning of the KV cache per tenant (one tenant cannot read another’s cached prefixes), and a tokens-per-minute quota enforced at the gateway.

Synchronised prompt registry. The product’s system_prompt lives in Langfuse with the production label. The active version is customer_support_system_prompt, version 12. The gateway reads the prompt from Langfuse on the request path (with a few seconds of cache so it does not hammer the registry). Detailed in the prompt versioning post.

The result at t = −31 days: the combination (adapter v7, prompt v12, golden v12) is active and served. The system is ready for the request that will arrive 31 days later.

Moving forward: the user’s request crosses the system

Back to Step 0: 11:47:32 on a Tuesday. The user presses Enter. We go in real time, in milliseconds.

t = 0 ms — Entry through the gateway

The user’s browser POSTs to chat.aseguradora-ejemplo.com/api/chat. The traffic crosses the edge load balancer and enters the SaaS product’s API gateway. The gateway:

Authenticates the user’s JWT (an end customer of the insurer tenant).
Extracts the tenant_id and checks that its tokens-per-minute quota is not exhausted.
Resolves which model_id, adapter_id and prompt_id correspond to this tenant and product. In this case: llama-3-70b-int8 + customer_support_v7 + prompt label production.
Builds a unique trace_id (W3C TraceContext, propagable to OTel) and starts a root span.

At 8 ms, the gateway passes the request to the prefill pool.

t = 8 ms — Pulling the versioned prompt

Before serving, the OpenAI-compatible client the engine uses internally pulls the active system prompt. As the post about prompt versioning details, the pattern is:

prompt = prompt_registry.pull(
 name="customer_support_system_prompt",
 label="production", # now pointing at v12
)
# A local 30 s cache reduces the round-trip to 0.1 % of requests

The OTel span of the prompt pull carries the attributes gen_ai.prompt.id = customer_support_system_prompt, gen_ai.prompt.version = 12, gen_ai.prompt.label = production. They are propagated to all children.

t = 12 ms — RAG retrieval

The system needs context from the tenant’s knowledge base. It runs:

query_embedding = encoder.encode(user_query)
chunks = qdrant.search(
 collection=f"tenant_{tenant_id}_kb_v3",
 vector=query_embedding,
 limit=4,
 score_threshold=0.72,
)
reranked = reranker.rerank(user_query, chunks, top_k=2)

At 38 ms, the reranker returns two fragments: one from the Cancellation policy, section 4.2 and another from Premium plan benefits, section 2.1. As the post about PostgreSQL + Qdrant details, the tenant’s corpus is kept isolated by collection and ACL: no tenant can read another’s chunks.

t = 40 ms — Building the final payload

The engine composes:

[system_prompt v12]
+ [retrieved context: 2 chunks]
+ [short session history: 1 previous turn]
+ [user query]

Total: ~1,850 tokens of context. The OTel span records gen_ai.request.input_tokens = 1850, gen_ai.request.model = llama-3-70b-int8, gen_ai.request.adapter = customer_support_v7.

t = 45 ms — Prefill

The payload enters the prefill pool. The GPU processes the 1,850 tokens in a single parallel pass, computing for each token its K and V vectors (attention key and value). Those vectors are materialised as the KV cache, covered in detail in the KV cache fundamentals post. The resulting cache takes up ~120 MB of VRAM in INT8.

Here a key optimisation appears: the v12 system prompt is cached in the prefill pool (prefix caching, covered in the post about PagedAttention). Since the system prompt is the same for this tenant, the first ~500 tokens of the context are not recomputed: they are read from the prefix cache. That reduces the effective prefill from 1,850 tokens to ~1,350 tokens, saving ~270 ms of compute.

At 580 ms (effective prefill), the TTFT (time to first token) is ready. The first token leaves for the decode pool.

t = 580 ms — Decode (streaming)

The decode pool receives the prefilled KV cache and starts generation token by token. As the post about disaggregated serving details, the prefill/decode separation is what lets a multi-tenant system keep TPS stable: the decode pool is sized to sustain thousands of sessions decoding in parallel at a low cost per token, while the prefill pool is sized for bursts of short TTFT.

Generation at ~80 tokens per second. The answer will be ~290 tokens. Total decode time: ~3.6 s. Streaming: the user starts seeing words from t = 580 ms.

While decode advances, the engine emits child spans on every iteration with gen_ai.response.tokens_generated, gen_ai.response.cache_hit_ratio, gen_ai.response.cumulative_latency. The post about AgentSight and the post about MCP observability with OTel cover the detailed instrumentation of this layer.

t = 4,200 ms — Complete answer, root span closed

Generation finishes. The engine closes the root span with gen_ai.response.completion_tokens = 290, gen_ai.response.finish_reason = stop, gen_ai.response.total_latency_ms = 4200. The user sees the final answer. The session is ready for a next turn or for the user to click thumbs-up/thumbs-down.

By this point, every active stage has taken part:

Data (pre-existing): the indexed RAG corpus, the dataset that trained the adapter, the golden set that validated it.
Tune (pre-existing): the v7 adapter trained 45 days ago.
Eval (pre-existing): the gates that approved the promotion.
Deploy (at this very moment): vLLM + disaggregated + KV cache + multi-tenant.
Observe (at this very moment): the OTel spans emitted to Langfuse + Tempo, the metrics to Prometheus.
Retrain (about to activate): the feedback the user will leave in 15 seconds.

In parallel: Observe is watching

While the request happens, several pieces of Observe run in parallel and leave a structured trail.

OTel tracing. Every span (gateway, prompt pull, retrieval, prefill, decode) travels to Langfuse and to an OTel collector that forwards them to a backend (Tempo / Jaeger). The unique trace_id links all the spans. As the post about tracing with AgentSight details, end-to-end propagation is the main enabler of post-incident debugging: without it, you cannot reconstruct what happened three weeks later.

Runtime metrics. The engine emits Prometheus metrics per interval: gpu_utilization, kv_cache_usage, tokens_per_second, queue_depth, prefill_latency_p95, decode_latency_p95. The metrics are not tied to a trace; they are aggregated by tenant and service.

Online LLM-as-judge. A configurable percentage of answers (2% in this system) is also run through a judge LLM in the background, which scores the answer against a simple rubric (correct / partial / incorrect + a tone score). The judge does not block the answer to the user; it feeds the dashboard.

Statistical drift. In parallel, a slower pipeline computes drift over the distribution of inputs and outputs. As the post about eBPF + drift explains, low-level monitoring (latency, error rate per endpoint) is complemented with statistical drift detection (KS test, embedding distance) that spots when “something is wrong” before a thumbs-down confirms it.

Safety and guardrails monitor. The guardrails post describes the layer that watches for jailbreak attempts, PII leakage and prompt injection via MCP tools. In this case, none of them fires.

All these pieces operate continuously, not per request. But this particular request left its trail in every one of them.

The feedback: the loop closes

Fifteen seconds after reading the answer, the user marks a thumbs-down and writes in the form: “the answer is correct but the tone is too salesy; I only wanted to know how to cancel”. That apparently trivial gesture triggers an important sequence.

Insertion into feedback_signals

As the post about Retrain details, the thumbs-down is persisted as a structured row in a Postgres table:

INSERT INTO feedback_signals (
 signal_id, trace_id, request_id, signal_type, signal_value,
 prompt_id, prompt_version, model, user_segment, occurred_at
) VALUES (
 gen_random_uuid(),
 '4f5...', -- the trace_id from Step 0
 'r-22a...', -- request_id
 'thumbs',
 '{"vote":"down","reason":"too pushy","text":"I only wanted to know how to cancel"}',
 'customer_support_system_prompt',
 12,
 'llama-3-70b-int8+customer_support_v7',
 'premium-es',
 '2026-05-19T11:47:51+02:00'
);

With this, the row is linked by trace_id to everything that happened: prompt v12, retrieved context, complete output, latency metrics, judge score (0.82 in this case, considered good by the judge, though the human disagrees).

Triage by root cause

The MLE team runs triage the next morning. Combining heuristic rules, LLM-as-classifier and human review:

The signal is not a model issue: the model answered correctly to the prompt it received.
It is not a retrieval issue: the retrieved chunks were the right ones.
It is not an infra issue: latency was normal.
It is a prompt issue: the v12 system prompt instructs the model to “offer alternatives before processing destructive actions”. That instruction generates the “salesy tone” in some contexts.

The incident accumulates with others from the month in the “salesy tone” cluster. When the cluster crosses a threshold (typically 30-50 incidents of the same type or a percentage of the total), it will enter an incident-driven mini-cycle or wait for the quarterly Retrain, depending on the size.

The next cycle picks it up

Three months later, in the next quarterly Retrain, this feedback is one of many that will motivate two changes:

A new prompt version v13 with an adjusted instruction: “offer alternatives only if the user does not express a clear intention to cancel”.
A possible reinforcement of the adapter with cases of a more direct tone for premium-es. If the cluster justifies it.

The v13 will go into its own eval gate. The golden set will grow with cases where the correct tone is “direct, not salesy”. The v8 of the adapter (if it arrives) will retrain over the enriched dataset enriched_retrain_2026_q2 that already contains this annotated case.

The loop closes. The Step 0 request has contributed to the version of the system that will serve another user three months later.

What goes in each trace: identity and traceability

If the reader looks at the seven identifiers that are everywhere in this route, they see the network of identities that makes all of the above possible. It is the identity infrastructure of an LLM system in production:

trace_id 4f5... (unique per request)
request_id r-22a... (idem)
prompt_id customer_support_system_prompt
prompt_version 12
prompt_label production
dataset_id enriched_retrain_2026_q1
dataset_version v3 (sha256:9af...)
model_id llama-3-70b-int8
adapter_id customer_support_v7 (sha256:5c1...)
deployment_id d-prod-7b
schema_version 3.2
tenant_id aseguradora-ejemplo
user_segment premium-es
golden_set_id customer_support_golden_v12

If a single piece of that set is missing or does not propagate, the chain breaks. The next incident investigated will land in “we cannot rewind to the origin because the system did not record it”. That is why the cross-cutting components, prompt versioning and data versioning, are not luxuries: they are the connection without which the other six stages operate blind.

Synthesis diagram: how the pieces fit

 ┌─────────────────────────────────────────┐
│ User (end customer, B2C) │
└─────────────────┬───────────────────────┘
│ chat msg + JWT
▼
┌─────────────────────────────────────────┐
│ Edge LB + WAF + Cilium CNI │
└─────────────────┬───────────────────────┘
│ HTTPS, internal mTLS
▼
┌─────────────────────────────────────────────────┐
│ API Gateway (auth, quota, model routing) │
│ - Resolves tenant → model + adapter + prompt │
│ - Starts trace_id (W3C) │
└──────┬─────────────────────┬────────────────────┘
│ │
(pull prompt) │ │ (pull config)
▼ ▼
┌────────────────────┐ ┌──────────────────────┐
│ Langfuse Prompt │ │ Model registry │
│ Registry (v12) │ │ (adapter v7) │
└─────────┬──────────┘ └──────────┬───────────┘
│ │
└──────────┬───────────────┘
│ payload ready
▼
┌──────────────────────────────────────────┐
│ vLLM engine (K8s Operator) │
│ ┌──────────────┐ ┌──────────────┐ │
│ │ Pool prefill │ → │ Pool decode │ │
│ │ (H100×N) │ │ (H100×M) │ │
│ └──────┬───────┘ └──────┬───────┘ │
│ │ KV cache fabric │ │
│ └──────────────────┘ │
│ - prefix caching of the system prompt │
│ - PagedAttention │
└──────┬───────────────────────────────────┘
│ tokens stream
▼
┌─────────────────────────────────────────┐
│ User sees the answer + thumbs/UX UI │
└─────────────────┬───────────────────────┘
│ feedback (15 s later)
▼
┌─────────────────────────────────────────┐
│ feedback_signals (Postgres) │
│ + Langfuse scores │
└─────────────────┬───────────────────────┘
│
┌────────────────────────┼────────────────────────┐
│ │ │
▼ ▼ ▼
triage quarterly Retrain cycle dataset_id
root cause or incident-driven enriched (DVC)
│
▼
Tune of v8
(next cycle)
In parallel throughout the request, OTel instrumentation:
spans → Tempo / Jaeger ; events → Langfuse ; metrics → Prometheus

The on-premise stack applied

Taking the above to a generic on-premise consultant-profile infrastructure (RTX 4090 + 4×H100 SXM cluster):

Layer	Typical resources
Network plane	Edge LB (HAProxy / nginx ingress) + Cilium CNI with BGP, covered in Cilium BGP
K8s compute plane	RKE2 with two manager nodes + a GPU node pool
Production GPU plane	4× H100 SXM (NVLink, 80 GB each), partitioned via MIG into prefill/decode pools
Development GPU plane	1× RTX 4090 (24 GB) for offline evals, drift-check embeddings, smoke tests
Storage plane	MinIO or Ceph object store; DVC remote + lakeFS backend
OLTP data plane	Postgres 18 with replication; pgvector 0.8 for small cases
Vector plane	Qdrant or Milvus for large RAG corpora
Stream plane	Kafka (Redpanda / pure Apache) + Schema Registry; CDC with Debezium or Flink CDC
Observability plane	OTel Collector + Tempo (traces) + Prometheus (metrics) + Loki (logs); Langfuse for the LLM-specific part
Runtime security plane	Tetragon, covered in the runtime security post

The real density is not the sum of the boxes: it is the operations that tie the boxes together. A cluster with every piece but no versioning discipline, no end-to-end trace_id propagation, no schema contracts and no cadenced retraining is a cluster that serves LLM once and then ages. The difference between a project and a platform is exactly that.

Ten bridges between stages where the system breaks

The route reveals something important: failures are rarely inside a stage; they are on the bridges between stages. Ten common bridges:

Data → Tune: the dataset does not propagate its (dataset_id, dataset_version) to the trainer. The same dataset trained twice produces two model_id values that cannot be told apart.
Tune → Eval: the trained model does not propagate its lineage to the eval run. The eval passes, but there is no record of which dataset it was trained on. Three months later, irreproducible.
Eval → Deploy: promotion happens without the serving system recording which version of the adapter it is serving at each moment. The day the model gives a dangerous answer, nobody knows which adapter answered.
Deploy → Observe: the engine does not emit gen_ai.request.adapter, gen_ai.prompt.version, gen_ai.dataset.version as span attributes. The traces exist but cannot be crossed with the lineage.
Observe → Retrain: the feedback is captured in a tool (Langfuse, Phoenix) but nobody reads it. The Retrain stage “exists”, but feedback piles up untriaged.
Retrain → Data: the enriched dataset goes into the next Tune without passing through versioning discipline, schema contract and holdout check. Silent contamination of the golden set.
Prompt versioning ↔ everything: the prompt_id, prompt_version does not propagate to the spans. The day the team discovers that a prompt change regressed the system, it cannot isolate which one or when.
Data versioning ↔ everything: the dataset_id, dataset_version does not show up in experiment tracking. “v8 is retrained” but nobody can prove it was over the enriched dataset and not the old one.
MCP ↔ tools: the system invokes tools (cancellation, policy changes) but does not record a gen_ai.tool.invocation_id linked to the trace. The actions end up dissociated from the answer that generated them.
Schema Registry ↔ data: datasets version content but not schema. A breaking change in expected_output breaks the eval silently; nobody notices anything until a human reviews the results.

The bridges are covered throughout the blog. Operations enforce them. The team’s culture keeps them.

How to walk through the blog

If you arrive at this post from outside and want a reading route:

The map: The six-stage LLMOps pipeline — the master map of everything else.
The context: MLOps specific to LLMs in 2026 — the landscape and why LLMOps is not classic MLOps.
Inference from the inside out: KV cache → PagedAttention deep dive → Disaggregated serving → Multi-tenant GPU cluster → vLLM on K8s → LLM operators on K8s.
Data: Data versioning with DVC and lakeFS → PostgreSQL + Qdrant ingestion → RAG over Kafka.
Tune: Continuous fine-tuning in production.
Eval: Evals: the layer after tracing → Guardrails and safety.
Observe: AgentSight LLM tracing → MCP observability with OTel → On-device eBPF + drift.
Retrain: Closing the feedback → dataset → adapter loop.
Cross-cutting: Prompt versioning with Langfuse and MLflow.
Supporting infrastructure (the base everything is built on): RKE2 with Cilium BGP, Hubble + eBPF observability, Tetragon runtime security.

What we have not covered (yet)

At the first level the main things are there. The following posts on the blog, when the topics justify it, could go deeper into:

Schema Registry for LLM data and prompts: the other half of the data contract.
A dedicated AI Gateway: LiteLLM, Portkey, Kong AI Gateway as a control plane.
OTel gen_ai semantic conventions: the emerging standard that ties the seven identifiers of the “identity” block into well-formed spans.
Federated learning over regulated customer data: how to train without centralising the corpus.
Capacity planning for shared multi-tenant clusters.
Disaster recovery of an LLM service: how to reproduce the state of the system 30 days back.
Cost accounting per tenant: tokens × weights × adapter × infrastructure → invoice.

References

W3C Trace Context — end-to-end propagation of traceparent and tracestate.
OpenTelemetry GenAI Semantic Conventions — gen_ai.* attributes for LLM spans.
Langfuse documentation — observability and prompt registry.
vLLM documentation — production inference engine with PagedAttention and LoRA hot-swap.
Kubernetes Operators — the declarative management pattern.
MLflow Tracking and Model Registry — lineage of runs and input artifacts.
DVC and lakeFS — dataset versioning, unified in Nov 2025.
OpenLineage — an open standard for lineage events between systems.
ENS / NIS2: compliance frameworks that apply to operators in the EU; recommended reading for the context in which the scenario operates.

Knative and scale-to-zero for LLM inference: when switching off the GPU saves money and when it costs you the SLO

Mon, 31 Aug 2026 10:00:00 +0200

Closing the platform and self-service run, after Backstage and Kubeflow. The question here is purely economic with a technical trap inside: switching off the GPU when nobody is using it looks like free money, until the first user of the day waits two minutes.

TL;DR

A GPU costs the same serving as it does idle. An H100 on dedicated cloud runs at 2 to 4 euros an hour whether it is used or not, so switching the service off when there is no traffic looks like money given away. Knative is the piece that does it: its autoscaler scales down to zero replicas and brings them back when a request arrives, measuring concurrency instead of CPU, which is what the Kubernetes autoscaler cannot do.

The trap is in the cold start. Knative brings up a pod in seconds, but that is only the beginning: an LLM has to load its weights from disk into HBM and compile its graphs, and there you go from tens of seconds to minutes depending on model size and storage. One reference study measures vLLM startup at around 20 seconds even for a 3 billion parameter model, and weight loading grows linearly with size. The first user of every traffic trough pays that whole wait.

Hence the rule. Switching off the GPU pays off with sporadic traffic, many lightly used models or development environments, and it is a trap with a strict latency SLO, sustained traffic or huge models whose startup breaks any interactive budget. Between all and nothing there are three middle grounds that almost always win: keep one replica warm, share the GPU with model-swap, or speed up weight loading so that startup fits inside the autoscaler’s cycle.

The analogy: the restaurant that switches off the kitchen

A restaurant with the kitchen running at four in the afternoon, without a single customer, burns gas for nothing. The temptation is to switch it off and light it again when somebody walks in. The problem is the time it takes to heat up. If the kitchen is an induction hob, it heats in seconds and switching it off between services is obvious. If it is a wood-fired oven that takes two hours to come up to temperature, switching it off means the first customer of the night eats cold or leaves.

Serving a small model is an induction hob. Serving a 70 billion parameter model is the wood-fired oven: the weights that have to go up into HBM are the wood you have to burn until you reach temperature, and wishing does not speed it up. The decision to switch off the kitchen, that is, to scale to zero, does not depend on whether you want to save gas, but on how long your kitchen takes to heat and on how long your first customer is willing to wait. This whole article is that calculation.

How Knative scales

Knative graduated in the CNCF in October 2025 and has three components; the one that concerns us is Serving, the serverless application component with autoscaling. Its object model has four pieces: the high-level Service, the Configuration that describes the desired state, the immutable Revision which is what actually scales, and the Route that splits traffic between revisions.

What makes scaling to zero possible are two data plane components. The queue-proxy is a sidecar in every pod that measures concurrency and applies the simultaneous request limit. The activator is the key piece: when the service is at zero replicas, requests are routed to it, and it parks them in a queue, tells the autoscaler that capacity is needed, and forwards the parked request once a pod is ready. It is the waiter who takes your order and tells you the kitchen is warming up, instead of shutting the door on you.

KPA versus HPA

Knative’s own autoscaler, the KPA, measures concurrency and requests per second, not CPU or memory. That is the difference that matters: the Kubernetes autoscaler, the HPA, scales on CPU and does not know how to go down to zero, its operational minimum is one. To switch off completely you need the KPA.

The KPA works with two windows. The stable one, 60 seconds by default, averages normal traffic. The panic one, much shorter, reacts to sharp spikes: when demand exceeds twice the capacity of the current replicas, it enters panic mode and scales in one go. The default concurrency target per replica is 100, but for LLM inference the useful value is usually 1 to 4, because each request occupies the GPU heavily; the official KServe example uses a target of 1.

Two timers govern the shutdown. The scale-to-zero-grace-period, 30 seconds by default, is how long the system waits to have the restart machinery ready before withdrawing the last replica. The scale-to-zero-pod-retention-period, zero by default, is the minimum time the last pod survives after the decision to switch off. Raising this second value is one of the levers for not switching off as soon as there is a micro-trough in traffic.

Why an LLM cold start is not starting a pod

Here is the core of the article, and there is a nuance that most discussions skip. Knative bringing up a pod in seconds does not mean the model is ready in seconds. Starting an inference service has several phases, and the two costs that dominate are not the ones you would expect.

A recent study measuring vLLM cold start breaks it down step by step. For a 3 billion parameter model, total startup is around 20 seconds, and the counterintuitive finding is that most of it is CPU-bound, not GPU-bound: process initialisation, library imports, engine construction. Inside that there are two costs that grow, and they are the ones the acceleration techniques attack.

The first is weight loading, which scales linearly with model size. In that same study it goes from half a second for a small model to almost five for a 16 billion one. Projected onto real sizes, the on-disk weight rules: an 8 billion model takes around 16 GB in FP16 and 8 in FP8; a 70 billion one, 140 and 70; a 405 billion one, 810 and 405. Loading 140 GB from disk into HBM does not happen in a blink, and if storage is slow or sits on the network, the minute arrives on its own.

The second is graph compilation. Graph transformation and capture with torch.compile costs 11 to 21 seconds if there is no compilation cache, and drops to 3 to 6 with one. It is half a battle that has nothing to do with the weights, and it is won by keeping the compilation cache between startups.

The operational conclusion is that the enemy is not only HBM: it is weight loading plus compilation. For small models the weights are only a fraction of startup, and switching off the GPU is almost painless. For large models the weights dominate, and the cold start becomes the problem. Loading a 70 billion model with the naive loader of the usual library can run to several minutes, and that is exactly the time your first user waits.

The techniques that shorten the wait

The good news is that the cold start can be attacked on both fronts. On the weights side, streaming loaders such as NVIDIA’s transfer the weights to the GPU concurrently instead of sequentially: the published data takes an 8 billion model from around 48 seconds with the standard loader to around 14 with streaming from SSD, and to under 8 with fast storage. The most graphic case is a 122 billion model, 233 GB on disk, which goes from three and a half minutes to around 37 seconds. That number has a direct consequence: 37 seconds fit inside an autoscaler polling cycle of 30 to 60 seconds; three and a half minutes do not. It is the difference between scaling to zero being viable with large models or not being viable at all.

On the compilation side, the technique is cheaper still: keep the torch.compile cache between startups, and with it the 11 to 21 seconds become 3 to 6. Tools such as Tensorizer and OCI images with the weights packaged inside attack the same problem from another angle, as we saw in the article on speeding up cold start.

Knative and KServe: the two modes

KServe leans on Knative for its serverless mode, and this is what connects everything above to a real platform. KServe has two deployment modes.

The Serverless mode uses Knative Serving and the KPA, supports scaling to zero and measures concurrency and requests per second. It is enabled by setting minReplicas: 0 in the InferenceService. It is the natural path for serving many intermittent models.

The Standard or raw mode uses the Kubernetes HPA, does not scale to zero, and measures CPU and memory. You pick it when you need something Knative restricts, such as mounting several volumes, and you do not need to switch off completely.

There is a third path of particular interest to anyone who already has KEDA deployed: in Standard mode, KServe recommends KEDA for generative inference, and KEDA can scale to zero on Prometheus metrics such as queue length or tokens per second. It is a route to scaling to zero on your own metrics, outside Knative, and it fits better with autoscaling guided by the model’s real saturation signal rather than by generic concurrency.

For anyone serving many lightweight predictive models there is also ModelMesh, the high-density pattern that packs many models into the same pods and rotates them in memory, without leaning on Knative. It is the equivalent of model-swap: instead of switching off and on per model, it shares one GPU between many. It is aimed at classic short-response inference, not at generative LLMs.

The economics

The decision comes down to comparing what you save by switching off with what you risk at startup. The saving is easy to estimate. If a GPU costs $ C $ euros per hour and your service is idle $ H $ hours a day, switching it off saves on the order of

$$\text{daily saving} \approx C \cdot H$$

With an H100 at 3 euros an hour and 16 idle hours a day, that is 48 euros a day per GPU, on the order of 1,400 a month. Across a fleet of several lightly used models the figure multiplies and stops being negligible.

The cost is on the other side, and it is not measured in euros: it is measured in SLO. Scaling to zero only pays off if the cold start fits inside what your worst-case latency tolerates. With a small model and fast storage, tens of seconds at most, and for an internal assistant used in bursts that may be acceptable. With a 70 billion model and the naive loader, minutes, and no interactive SLO survives that.

Between switching off completely and never switching off there are three middle grounds, and almost always one of them is the right answer:

Warm replica, with minReplicas: 1. You remove the first user’s cold start in exchange for paying a continuous GPU-hour. The break-even point is direct: keeping it warm pays off when the cost of that GPU running hot is lower than the reputational or SLO cost of the first user waiting. For a customer-facing service, almost always.

Model-swap or sleep on a shared GPU, as we saw in serving several models on one GPU. The process stays alive and the model is unloaded and reloaded in seconds instead of starting from scratch. It is the middle ground for several models that do not justify a GPU each but do not tolerate a full startup either.

Speed up loading so that the cold start fits inside the autoscaler’s cycle, with weight streaming and a compilation cache. It is what makes real scaling to zero viable with large models, and what turns “three and a half minutes” into “37 seconds”.

Operational traps and honest scepticism

The pod starts in seconds, the model does not. Do not confuse Knative’s startup time with the time until the model answers. The difference is the whole article.

The default concurrency target is 100, and for an LLM that is a disaster. Each request occupies the GPU; a target of 1 to 4 is the reasonable one. Leaving the default 100 means accepting a hundred simultaneous requests on a replica that handles four.

The cold start is not just weight loading. Graph compilation costs what it costs, and it is attacked separately, with a cache. Optimising only storage leaves half the problem untouched.

Cilium and Gateway API are not a supported path yet. Knative supports Gateway API in beta and tests with Istio, Contour and Envoy Gateway; integration with Cilium does not appear as tested and has documented friction. If your ingress is Cilium, verify it before counting on it.

KEDA may be a better signal than the KPA. Scaling on generic concurrency is worse than scaling on the model’s real saturation metric. If you already have KEDA, Standard mode with scaling to zero on your own metrics deserves a test against the default Serverless.

For an inference factory

The decision to switch off the GPU is not ideological, it is arithmetic, and it depends on two numbers: how long your model takes to start and how long your worst user waits. Three rules close the matter.

First, switch off what is intermittent and cheap to start: development environments, small models used sporadically, the long tail of models almost nobody touches. There, scaling to zero is direct money with no appreciable SLO cost.

Second, do not switch off what is critical and expensive to start: the customer-facing model with a strict SLO, and the huge models whose cold start is measured in minutes. There a warm replica costs less than a user waiting.

Third, before deciding, measure your real startup and fix it. Many services are ruled out of scaling to zero over a three-minute startup that, with weight streaming and a compilation cache, drops to thirty seconds and changes the answer. The most profitable lever is not choosing well between switching off or not: it is making switching off cheap.

With this we close the platform and self-service run. The portal shows what is there, the ML toolbox gives what is missing without duplicating what you already have, and scaling to zero decides how much of all that is switched on when nobody is looking. All three answer the same question from different angles: how to put an expensive platform at the disposal of its users without going broke keeping it switched on.

Sources

CNCF, Cloud Native Computing Foundation announces Knative’s graduation — https://www.cncf.io/announcements/2025/10/08/cloud-native-computing-foundation-announces-knatives-graduation/
Knative Docs, Serving architecture — https://knative.dev/docs/serving/architecture/
Knative Docs, Request flow — https://knative.dev/docs/serving/request-flow/
Knative Docs, Configuring KPA-specific autoscaling — https://knative.dev/docs/serving/autoscaling/kpa-specific/
Knative Docs, Configuring concurrency — https://knative.dev/docs/serving/autoscaling/concurrency/
Knative Docs, Configuring scale-to-zero — https://knative.dev/docs/serving/autoscaling/scale-to-zero/
Knative Blog, Managing Gateway API ingress with the Knative Operator — https://knative.dev/blog/articles/gateway-api-ingress-with-knative-operator/
CNCF, KServe becomes a CNCF incubating project — https://www.cncf.io/blog/2025/11/11/kserve-becomes-a-cncf-incubating-project/
KServe Docs, Autoscaling with Kubernetes HPA (Serverless vs Standard) — https://kserve.github.io/website/docs/model-serving/predictive-inference/autoscaling/hpa-autoscaler
KServe Docs, Autoscaler for generative inference (KEDA) — https://kserve.github.io/website/docs/model-serving/generative-inference/autoscaling
KServe GitHub, docs/samples/autoscaling (InferenceService minReplicas 0) — https://github.com/kserve/kserve/blob/master/docs/samples/autoscaling/README.md
Wang et al. (MLSys 2026), Breaking the ice: analyzing cold start latency in vLLM — https://arxiv.org/abs/2606.07362
NVIDIA Developer Blog, Reducing cold start latency for LLM inference with NVIDIA Run:ai Model Streamer — https://developer.nvidia.com/blog/reducing-cold-start-latency-for-llm-inference-with-nvidia-runai-model-streamer
Microsoft Azure SDK Blog, Eliminate LLM cold starts: load models up to 6x faster — https://devblogs.microsoft.com/azure-sdk/eliminate-llm-cold-starts-load-models-up-to-6x-faster-with-azure-blob-storage-and-runai-model-streamer/
Hugging Face Blog, Llama 3.1 (VRAM and precision tables) — https://huggingface.co/blog/llama31
CloudZero, H100 GPU cost 2026 — https://www.cloudzero.com/blog/h100-gpu-cost/

KubeVela and Score: handing over the platform by CLI without teaching the developer Kubernetes

Mon, 31 Aug 2026 09:45:00 +0200

Third and last article in the batch about rebuilding the cloud on-premise. The general map placed the gaps for you and Crossplane covered the infrastructure control plane. Here comes the layer above, the one your developer touches every day: application-centric self-service by CLI.

TL;DR

Crossplane answers “give me a database”, from the infrastructure side. KubeVela and Score answer “deploy my application”, from the workload side, and they do it by CLI, which is precisely the friendly experience Crossplane deliberately does not give you. The two share an article because they cover the same layer from different angles. KubeVela is a complete delivery platform: it uses the OAM model, where your platform team writes some definitions in the CUE language once and the developer deploys their whole application with a fifteen-line file and a CLI (vela up, vela status, vela logs) that feels like Heroku on the outside and is GitOps on the inside. Score is more modest and more portable: a workload specification in a single file that gets translated by CLI into Docker Compose locally and into Kubernetes in production, without running anything in the cluster.

The honest part up front is the limits. Neither of the two appears in the “Adopt” section of the CNCF technology radar from early 2026, where Helm, Backstage and Argo CD do: they are niche against the mainstream. KubeVela version 1.11 is still in alpha, so the stable one is 1.10; its community is concentrated, with around 33 people contributing in 2025. Score is young, its Kubernetes implementation has few stars and its sponsor, Humanitec, was acquired by Akamai. And KubeVela’s power lives in CUE, an uncommon language that is hard to debug. The abstraction is real, but when it breaks, your developer ends up looking at Kubernetes YAML all the same.

The analogy: the order form against the shopping list

Deploying on raw Kubernetes is going to the wholesaler with a technical shopping list: you have to know that you want a Deployment, a Service, a HorizontalPodAutoscaler, an Ingress, a ConfigMap and a ResourceQuota, and how each one gets filled in. It works if you know the warehouse. But your data scientist, who just wants to serve their model, should not have to learn the warehouse.

KubeVela is an order form. Instead of the technical list, you tick “I want to serve this application, with these replicas, with this gateway”, and behind it somebody who does know the warehouse (your platform team) has defined exactly what gets ordered for that to work. The developer fills in the form; the translation into the technical list was done by somebody else, once, for everyone.

Score is the same form but designed so that it works for you in two different shops. You describe your order once and the same sheet serves the market next door (Docker Compose, your laptop) and the wholesaler (Kubernetes, production), without rewriting it. Whatever changes between one shop and the other is supplied by whoever translates, not by you. Both analogies point at the same thing: separating what the developer wants from how it materialises, so that they do not have to learn the warehouse.

KubeVela: the application as the unit

What it is and what state it reaches 2026 in

KubeVela is a CNCF project at incubation level since February 2023. It was born as an evolution of the OAM runtime on Kubernetes, with initial contributions from Alibaba Cloud, Microsoft and Upbound, and it rests on the Open Application Model, the specification Alibaba and Microsoft co-created to describe applications independently of the infrastructure.

Here is a realism check on its community size. The official 2025 retrospective declares seven releases in the year, 124 commits and around 33 people contributing, on a little over 7,600 GitHub stars. It is not a project in decline, but it is a concentrated community, with a growth curve that has flattened compared to its early stage under Alibaba. And an important version warning: the stable branch is 1.10, with regular patches through 2026; 1.11 is still in alpha, so do not rely on its features as if they were available.

The OAM model in five pieces

Your whole application is described in a single YAML file of type Application, with four sections you need to know by name. The components are the artefact you deploy, be it an image, a Helm chart or a service; the traits are operational requirements you hang off each component, such as scaling, the gateway or storage; the policies are global application strategy, such as multi-cluster topology or SLOs; and the workflow is the step-by-step delivery process, which can include a manual approval pause or a notification.

The key to the model is that component and trait types are not fixed: they are programmable modules maintained by your platform team, expressed as ComponentDefinition and TraitDefinition. The developer only consumes a type and its properties; they never see the Kubernetes YAML underneath. That is exactly the “do not expose raw Kubernetes” this is all about: your platform operator writes the definition once, your data scientist writes fifteen lines of Application.

CUE, the power and the toll

The definitions are written in CUE, and here you have KubeVela’s strength and weakness at the same time. CUE lets you parameterise seriously: the definition declares which inputs it exposes to the developer and which Kubernetes resources it renders from them, with access to context variables such as the application name. The real flow starts from an existing YAML, which vela def init turns into a CUE skeleton; you edit it, validate it with vela def vet and publish it with vela def apply, and it becomes available to all your developers.

The toll is that CUE is an uncommon language, with few experts and hard to debug. The cost falls on your platform team, not on the developer, which is the correct distribution, but it is a real cost you have to sign up for before adopting. Nobody arrives already knowing CUE.

The vela CLI: Heroku on the outside, GitOps on the inside

Here is the answer to the question this blog has been dragging along about CLI provisioning. Day to day with KubeVela looks like Heroku: vela up deploys the application from local files, vela status shows you its state, vela ls lists the applications, vela logs follows the logs, vela exec runs a command inside the container without you knowing the pod name, and vela port-forward opens a tunnel for you. You even have vela dry-run, which renders the Kubernetes resources without applying them, so you can see what is going to happen.

The honest qualification is that it is “imperative on the surface, declarative underneath”. When you run vela up you are not executing an imperative command like heroku scale, you are applying a declarative Application object that a controller reconciles. The CLI imitates Heroku’s comfortable flow, but the underlying model is GitOps, which means it fits with your Flux stack through the corresponding addon, instead of replacing it.

Multi-cluster, which is what matters to you

If you have a platform with several sites (site01 and site02, in my case) the multi-cluster part is the deciding one. KubeVela orchestrates from a central cluster and only the already rendered resources reach the managed clusters, with centralised governance. A topology policy decides the destination by cluster list or by label selector (by region, for example); an override policy adjusts images, replicas or traits per cluster without touching the component; and the deployment step in the workflow distributes, with the option of a phased rollout: first to the local cluster, then a manual approval gate, and only then to production. It is the kind of governance you have to build on-premise and that the cloud served you ready-made.

VelaUX, the lightweight portal thrown in

KubeVela comes with an optional web console, VelaUX, which you switch on as an addon and which gives you a dashboard, project management, environments and application lifecycle. If you want CLI first it is optional, more useful for giving operators visibility and access control than as the main route. It is a lightweight portal “thrown in” that does not aim to compete with Backstage, but it can save you from setting one up if your needs are modest.

Score: describe the workload once

What it is and what it solves

Score is an open workload specification, agnostic of platform and environment. The idea, in its own formulation, is that you define the workload once and use a Score implementation to translate it to several platforms: Docker Compose, Kubernetes, Fly.io or Cloud Run. The problem it attacks is concrete and familiar: the divergence between what you run on your laptop with Docker Compose and what gets deployed in production with Kubernetes, two descriptions that drift apart and cause you the classic “it worked on my machine”.

In a score.yaml file the developer declares their containers, the service ports and the resources they need as dependencies (a database, a DNS record, a route), and the environment-specific parameters are injected at deploy time. Score entered the CNCF sandbox in July 2024. It was created and donated by Humanitec, a developer platform company, some eighteen months before it joined the foundation.

The two implementations

Score does nothing on its own: it is a specification that needs an implementation to translate. You have two official ones. score-compose generates Docker Compose for local development, with init and generate commands, and provisions on the order of fourteen resource types, from Postgres or Redis to S3 storage or queues. score-k8s generates Kubernetes manifests ready for kubectl apply, with flags for the image, the namespace and the provisioners. Humanitec’s commercial CLI, humctl, also supports Score natively.

The most important architectural fact is that Score runs nothing in the target cluster. It is purely CLI translation: it generates manifests for you and steps aside. There is no controller, no CRD, no reconciliation. That is at once its virtue (it is lightweight, portable and without runtime lock-in) and its limit (it neither orchestrates nor deploys, it only translates).

Score against KubeVela and Helm

Score’s own official positioning clarifies the layers, and I will pin it down for you because they are easy to confuse. Score is the workload specification layer, developer-centric: it describes a workload and its dependencies and gets translated to N platforms by CLI, but it does not orchestrate or deploy. KubeVela is a complete continuous delivery platform, application-centric: it orchestrates multiple components, applies policies and workflows, does multi-cluster, and for that it needs its controller in the cluster; it is, in its own words, far richer in features than Score. Helm is the packaging and rendering of manifests, without the developer-centric abstraction or the portability to Compose.

The clean synthesis is that they are complementary layers, not substitutes: Score generates, Helm or the manifests package, KubeVela orchestrates and deploys. If they are presented to you as rivals, somebody has confused the layers.

Where each one fits: the cross-cutting comparison

With Crossplane already in hand from the previous article, the comparison of the three self-service pieces sorts itself by the question each one answers.

Score and KubeVela answer at the application level: the workload a developer deploys, “deploy this app”. Crossplane answers at the infrastructure level: the resource a developer asks for, “give me a managed Postgres”. And Backstage answers at the portal level: the web catalogue and the golden paths, the discovery layer that neither deploys nor provisions by itself, but orchestrates the ones below.

The interesting part is that they combine. A score.yaml can declare its resources and delegate the provisioning of those resources to Crossplane through a provisioner, while KubeVela orchestrates the application deployment and Backstage acts as the façade. It is not an either-or; it is a stack where each layer does its job. If your team is small, you pick a single piece and that is that; the complete stack is for mature platforms.

The fit with inference: a pattern, not an official recipe

Just as with Crossplane, honesty is called for here: there is no official, maintained KServe definition in KubeVela’s default catalogue, nor an inference resource type in Score. What follows is a design pattern with pieces confirmed separately, not a documented recipe.

The KubeVela pattern would be that your platform operator defines once, in CUE, a ComponentDefinition called for example inference-service, which underneath renders a KServe inference service with vLLM runtime, with the GPU limits, the namespace quotas and the autoscaling already wired in. Your data scientist then writes a fifteen-line Application with the inference-service type and its properties (the model, the replicas), without touching KServe, without knowing the GPU quotas or the runtime class, and operates with vela up, vela status and vela logs. Multi-site is solved for you by the topology policy, which decides which GPU cluster it lands on, and the override policy, which adjusts replicas or model per site, with a manual approval in the workflow before production.

The Score alternative would be a score.yaml with a resource of type inference-endpoint that score-k8s, with a custom provisioner, emits as a KServe inference service. More portable, but today it requires that provisioner of your own, because KServe is not a default type. In both cases, the abstraction layer rests on pieces that are already standard (KServe moved to CNCF incubation at the end of 2025 and vLLM is its reference generative runtime), but the inference abstraction is built by you. Take it as a proposal, not as a proven case.

A note for you: in 2026 KServe introduced a type dedicated to large models, LLMInferenceService, still in alpha and with vLLM as its runtime, which is the brick a KubeVela definition or a Score provisioner would render underneath. And at the infrastructure layer, not this one, the Crossplane team published Modelplane, an inference control plane made of compositions that serves vLLM directly. Neither of the two is a KubeVela or a Score type, but both confirm for you that the brick underneath is standardising, which is what makes putting a self-service façade on top viable.

Caveats, no frills

This is the section that separates the article from the brochure, and with these two projects you need it especially.

Adoption is niche, and there is a hard figure. The CNCF technology radar from early 2026, covering more than 400 developers, places Helm, Backstage and Argo CD in “Adopt”. Neither KubeVela, nor Score, nor Crossplane get a mention. They are niche tools against the mainstream, and choosing them means betting on an adoption curve that may not take off.

Score is young and its sponsor changed hands. Its Kubernetes implementation has few stars, the specification is at 0.x versions, and Humanitec, which sponsors it, was acquired by Akamai in 2025, so you have to watch the continuity of the investment in the open source part.

KubeVela has a concentrated community and v1.11 in alpha. The 33 people contributing in 2025 and the flat star curve are a sign of a plateau. Do not build on 1.11 features until it is stable.

The CUE curve is real. KubeVela’s power lives in CUE definitions, and CUE is hard. The cost falls on the platform, but it exists.

And the usual risk with abstractions: the leak. Both KubeVela and Score add an indirection over Kubernetes and KServe. When something fails underneath, the abstraction cracks and your developer ends up looking at the KServe YAML anyway, plus lock-in on your own definitions. An abstraction that does not hold up under failure hands you back exactly the complexity it promised to hide.

For your sovereign inference factory

The practical decision sorts itself by the profile of your users. If your people live in the CLI and think in applications, KubeVela gives you the Heroku-style experience on your own iron, with real multi-cluster, in exchange for your platform team learning CUE. If what hurts most is the divergence between local and production, Score solves that with the bare minimum, without putting one more runtime in the cluster. And if you want both plus a portal, the Score, Crossplane, KubeVela and Backstage stack exists, but it is for a platform that already serves many teams, not for the start.

The sobriety criterion is the same as throughout this batch: these tools plug a real gap left by the cloud you walked away from, but each one is one more system to operate. For two teams who know each other, good Helm and a tidy GitOps repository do the job without CUE or definitions of your own. Application-centric self-service starts paying you back when the number of people deploying exceeds the number of people who know Kubernetes, which is precisely the moment an inference factory stops being a project and starts being a platform.

That closes the batch. The map drew the gaps of leaving the cloud, Crossplane covered the infrastructure control plane, and KubeVela with Score cover the face the developer touches. Your own house now looks quite a lot like the hotel, and the data stayed inside.

Sources

KubeVela, State of KubeVela 2025 — https://kubevela.io/blog/2025/12/20/state-of-kubevela-2025/
CNCF, KubeVela brings software delivery control plane capabilities to the CNCF incubator — https://www.cncf.io/blog/2023/02/27/kubevela-brings-software-delivery-control-plane-capabilities-to-cncf-incubator/
KubeVela Docs, Core Concept — https://kubevela.io/docs/getting-started/core-concept/
KubeVela Docs, vela CLI reference — https://kubevela.io/docs/cli/vela/
KubeVela Docs, Custom Component with CUE — https://kubevela.io/docs/platform-engineers/components/custom-component/
KubeVela Docs, Multi-cluster delivery — https://kubevela.io/docs/case-studies/multi-cluster/
KubeVela Docs, VelaUX addon — https://kubevela.io/docs/reference/addons/velaux/
CNCF, Score accepted as a CNCF Sandbox Project — https://www.cncf.io/blog/2024/08/08/score-accepted-as-a-cncf-sandbox-project/
Score Docs, Overview — https://docs.score.dev/docs/
Score, Score vs the Open Application Model and KubeVela — https://score.dev/blog/score-vs-open-application-model-kubevela/
GitHub, score-compose — https://github.com/score-spec/score-compose
GitHub, score-k8s — https://github.com/score-spec/score-k8s
CNCF, KServe becomes a CNCF Incubating Project — https://www.cncf.io/blog/2025/11/11/kserve-becomes-a-cncf-incubating-project/
KServe Docs, Understanding LLMInferenceService — https://kserve.github.io/website/docs/model-serving/generative-inference/llmisvc/llmisvc-overview
Crossplane Blog, Building Modelplane on Crossplane — https://blog.crossplane.io/building-modelplane/ · https://modelplane.ai
CNCF y SlashData, Technology Radar Q1 2026: Platform Engineering — https://www.cncf.io/announcements/2026/03/24/cncf-and-slashdata-report-finds-platform-engineering-tools-maturing-as-organizations-prepare-for-ai-driven-infrastructure/

Crossplane: the control plane that turns your cluster into a private cloud API

Mon, 31 Aug 2026 09:30:00 +0200

Second article in the batch about rebuilding the cloud experience on-premise. The general map placed the gaps for you; this one goes into the provisioning control plane, that API which on-premise does not come for free. The third moves up a layer, to application-centric self-service with KubeVela and Score.

TL;DR

Crossplane solves the “open a ticket and wait” gap for you. In the cloud you ask for a database through an API and it appears with permissions, backups and monitoring; on-premise you do that provisioning by hand or with scripts. Crossplane brings the Kubernetes model of declarative APIs to infrastructure: your platform team defines an API of its own (a database, an inference endpoint) and the developer consumes it with a kubectl apply of a few fields, while controllers inside the cluster reconcile the desired state against reality and correct drift continuously. It graduated from the CNCF in November 2025, with more than three thousand contributors and adopters such as Nike, SAP, IBM or NASA’s science cloud.

The difference from Terraform is one of nature, not of features: Terraform applies a plan on demand and leaves; Crossplane is a control plane that lives inside the cluster and never stops reconciling. That gives you drift self-correction and native GitOps, and in exchange it takes away the safety net of plan and forces you to operate the control plane itself, with providers that create hundreds of resource definitions and state that lives in etcd. And a warning about something awkward up front: Crossplane is not an imperative aws ... style CLI, it is declarative first. On the fit with GPU inference, which until recently was only a pattern with no public case, you now have a real reference, Modelplane, from the Crossplane team itself, although at version 0.1 and composing vLLM directly without going through KServe. I go into the detail at the end.

The analogy: the thermostat against the light switch

Think of it this way: Terraform is a light switch. You flip it, the light comes on, and it stays there until you touch it again. If somebody else changes it, the light obeys whoever touched it last, and you do not find out until you go and look. It is provisioning by one-off action: you run it, it applies, it finishes.

Crossplane is a thermostat. You do not tell it “turn the heating on”, you tell it “I want twenty-one degrees”, and from then on a mechanism watches the temperature and acts on its own, continuously, to hold it. If somebody opens a window and the thermometer drops, the thermostat reacts without you intervening. That is continuous reconciliation: you declare the desired state once and a controller takes care of reality not straying from it.

The analogy also warns you about the cost. A thermostat is more convenient than a switch, but it is one more device you have to install, power and fix when it fails, and if you program it badly it adjusts the temperature to something you did not want, without asking. A switch almost never breaks and you always know what state it is in. Crossplane gives you the convenience of the thermostat in exchange for you operating the thermostat.

What Crossplane is and what state it reaches 2026 in

Crossplane was born at Upbound in 2018 and followed the CNCF maturity path without shortcuts: sandbox in 2020, incubation in 2021 and graduation on 6 November 2025. Graduation matters because it is the seal the CNCF reserves for projects it considers production-ready, the same level as Kubernetes or Prometheus. The figures that came with the announcement are solid: more than three thousand contributors from more than 450 organisations, more than a thousand pull request authors (which puts it in the top 10 % of the foundation’s projects by that metric), more than a hundred releases and two completed security audits. Among the more than seventy public adopters you have Nike, Nokia, Grafana, NASA’s science cloud, SAP and IBM.

The project’s official description is “the cloud native control plane”, and that phrase sums up exactly what it does. The main vendor behind it is still Upbound, which at graduation framed the project’s future towards “AI-native” infrastructure, a point I come back to when talking about inference, with the caveat that it is a vendor’s positioning, not a proven case.

The release cadence is quarterly, with a nine-month support window and three actively maintained versions at any time. As of writing, the v2 branch is the current one, with v1.20 as the last close of the v1 branch.

The v2 turn (which changes how you think about it)

Crossplane 2.0 reached general availability in August 2025 and brings four structural changes you need to know about, because a good deal of the old material describes a model that is no longer the recommended one.

The first is that resources are now namespaced by default. Both composite and managed resources, which used to be cluster-scoped, now live in a namespace, which enables fine-grained access control and natural multi-tenancy, exactly what you need in a sovereign platform with several teams.

The second is that a composition can include any Kubernetes resource, not only Crossplane resources. Previously a composition was limited to Crossplane’s own managed resources; now it can include Deployments, Services or third-party CRDs, so you can compose abstractions mixing application and infrastructure in a single object. This change is what makes the inference pattern you will see viable.

The third is that Claims disappear. In the v1 model there was a duality between the composite resource, cluster-scoped, and the Claim, its namespaced face which the developer consumed. Now that the composite resource is itself namespaced, that duality is redundant: the developer consumes the composite resource with no intermediary. If you come across documentation talking about Claims, it belongs to the earlier model.

The fourth is a new type, Operations, for one-off or scheduled operational tasks that run a pipeline of functions to completion, like a Kubernetes Job but inside the Crossplane model.

There is one removal that deserves its own paragraph, because it is the most surprising: in v2 native patch-and-transform was withdrawn, the classic patching mechanism of compositions. It was not relegated, it was removed. Everything now goes through composition functions, and I get to that in a moment. Migration from v1 to v2 is mostly non-breaking and the v1 APIs still work, so adopting v2 is optional, but the new model is the one you have to learn.

The model, piece by piece

Crossplane has dense jargon and I will take it apart calmly, because once the pieces are placed the whole thing is coherent.

A provider is a package that installs controllers and resource definitions to talk to an external API, be it AWS, GCP, a remote Kubernetes cluster, Helm or Terraform. It is installed declaratively, by applying a Provider object. The providers for the big cloud providers are generated with Upjet, a framework that produces the code from the Terraform providers, which explains why there are so many of them and also why some are enormous.

A managed resource is the one-to-one representation, inside Kubernetes, of a resource from the external provider: a bucket, an instance, a database. It is the unit the controller reconciles. When you create a managed resource of type bucket, the provider’s controller creates the real bucket and then watches that it keeps existing with the configuration you declared.

On those bricks you build the abstraction, which is where the real value is, with three concepts:

The CompositeResourceDefinition, or XRD, defines the schema of your own API: which fields the user can ask for. It is what turns “a database” into a resource type with its parameters.

The Composition is the template and the logic that, given one of those resources, materialises the N managed resources needed underneath. The documentation says it bluntly: a composition is a pipeline of composition functions.

The Composite Resource, or XR, is the concrete instance: the object the developer creates with kubectl apply, with the few fields the XRD exposes.

The complete pattern is this: your platform operator publishes the abstraction (the XRD plus the Composition, which say “this is what a database means here”), and the developer only applies a composite resource with four fields. How it is built underneath stops mattering to them, just as in the cloud you do not care how AWS assembles your managed database.

Composition functions: the real logic

The composition mechanism in v2, and the only one, is composition functions. A composition is a pipeline of functions that Crossplane invokes over gRPC; each function receives the observed state, the desired state, its input and a shared context, and chains its output to the next one. The YAML in the examples is illustrative, but underneath it is not processed as YAML: it is executed as functions.

The available functions cover several styles. You have function-patch-and-transform, which reimplements the old model now as just another function. There is function-go-templating for Helm-style templates, function-kcl for the KCL language, function-python for logic in Python and function-cue for CUE.

The reason for the change is what you want to understand. Classic patch-and-transform had no loops or conditionals, so an operation as simple as transforming an array of strings turned into dozens of lines of patching, and the community went as far as calling it outright bad. Functions give you full programming logic (loops, conditionals, external calls) in a real language. The honest flip side is that the complexity does not disappear: it shifts from “writing verbose patches” to “writing and operating functions”, which is a different kind of work, not less work.

Crossplane against Terraform, OpenTofu and Pulumi

This is the comparison everybody makes, and I will make it properly because the difference is one of category. According to Pulumi’s own documentation, which compares honestly, Crossplane is a declarative control plane with continuous reconciliation inside Kubernetes, while Terraform and Pulumi are infrastructure as code executed one-off, client-side and on demand.

The practical consequences line up along a few axes. Faced with configuration drift, Crossplane corrects it continuously and automatically, while Terraform and Pulumi only detect it when you run a refresh. Crossplane needs a cluster to run on, because it is controllers; Terraform and Pulumi do not. Crossplane is GitOps-native, applied with kubectl and fitting with Flux without an adapter; Pulumi needs its Kubernetes operator for the same thing. And as for language, Crossplane uses YAML and CRDs with logic in the functions, while Pulumi uses general-purpose programming languages.

The point most often forgotten is that you do not have to choose one or the other. There is a Terraform provider (and an OpenTofu one, recommended for new versions because the Terraform one was frozen at the last freely licensed version) that runs HCL inside Crossplane, wrapping your existing modules as a managed resource. You can reuse your whole investment in Terraform and still gain continuous reconciliation for whatever needs it. If you already have Terraform in production, this hybrid pattern is usually your sensible route, not the rewrite.

On comparative adoption you have to be sober: there is no official, reliable “Crossplane against Terraform” figure, and the most repeated documented criticism is that Crossplane lacks the mature module ecosystem Terraform has been accumulating for years. That is a real qualitative difference, not a detail.

The awkward question: is it an AWS-style CLI?

Since on this blog I have been pulling on the thread of CLI provisioning, I will answer directly: no. Crossplane does not offer you an imperative aws s3 create-bucket style experience. It is declarative first. You provision with a kubectl apply of a resource, and the crossplane binary is there to build, package, test and debug, not to “create a bucket” in one go.

The subcommands make that clear. crossplane render renders a composition and its functions locally, so you can test without a cluster. The crossplane xpkg build, push, install family manages the OCI packages of providers, configurations and functions. And crossplane beta validate, trace and top validate schemas, draw the resource tree and show consumption inside the cluster. It is the toolbox of whoever builds the platform, not of whoever consumes it day to day. Upbound’s up CLI adds more, but oriented towards its commercial product.

The conclusion matters for setting expectations: the imperative AWS-console-style experience, if you want it, gets built on top of Crossplane, with Backstage, a portal or your own wrapper. Crossplane gives you the declarative API; the friendly face you put on separately. If what you are after is an imperative provisioning CLI, you fit better with the model of KubeVela and Score, which is precisely the subject of the next article.

On-premise providers: provisioning without touching AWS

For a sovereign platform your question is what you can provision without setting foot in a public cloud, and the answer is more than it seems, with uneven maturity.

The most versatile is provider-kubernetes, which manages any Kubernetes object in remote clusters through an Object resource. It is the basis of the multi-cluster on-premise pattern: from a central cluster you materialise resources in your secondary RKE2s. Alongside it, provider-helm installs and manages Helm charts declaratively through a Release resource, letting you deploy things like vLLM or KServe as part of a composition.

For the identity and storage part you have concrete pieces: provider-keycloak configures Keycloak realms, clients and roles declaratively, useful for the sovereign IAM we already set up in the article on authentication with Keycloak; and provider-minio manages buckets and policies of on-premise S3-compatible storage. For vSphere you do not have a first-class native provider, but you cover it by generating one from the Terraform provider, the usual pattern for VMware infrastructure.

An honest warning: for KubeVirt, Harvester or CloudNativePG there are no mature, first-class native providers; in practice you manage them by applying their CRDs through provider-kubernetes or by wrapping their Terraform. It works, but it is not the polished experience of a dedicated provider. With the combination of provider-kubernetes, provider-helm, provider-minio and provider-keycloak, plus provider-terraform for whatever is missing, you can assemble a one hundred per cent on-premise self-service catalogue, with the caveat that maturity varies piece by piece.

The fit with GPU inference: from pattern to Modelplane

This is the section that in the first version of this text carried the most caution, because there was no public case of Crossplane orchestrating GPU and inference and it all remained a design pattern. Halfway through 2026 that changed, and the case was published by the very team that makes Crossplane.

Modelplane is an open source control plane for AI inference, Apache-licensed and with a first version 0.1 from June 2026, and the interesting thing is how it is built: entirely on Crossplane compositions and composition functions in Python, with no controllers of its own. It composes vLLM directly, with the vllm/vllm-openai image, without going through KServe, and its self-service API for the ML teams is a ModelDeployment resource where you declare the model, the engine, the topology and the hardware. Underneath, a ModelReplica resource represents each replica, and the platform team declares its GPU pools with InferenceCluster and InferenceClass resources, using an attribute model and CEL expressions taken from Kubernetes dynamic resource allocation. Its most advanced piece is a two-level fleet scheduler written as a pure function, which places replicas across the whole infrastructure and delegates to each cluster’s scheduler. It is proof that the pattern works, with the warning that it is a version 0.1 under open development, not something to put into production blindly, and that adopting it ties your platform to its resource types.

The other recent enabler is in KServe, which introduced a type dedicated to large models, LLMInferenceService, still in alpha and built on the llm-d project, with prefill and decode separation and multi-node execution, and which runs vLLM as its runtime. It is different from the classic InferenceService, reserved for predictive ML. That new type is your ideal brick if you prefer composing KServe instead of bare vLLM.

With that, your own golden path stops being a hypothesis. You define an XRD, for example InferenceEndpoint, with fields such as the model, the number and type of GPU and the replicas. Its composition, taking advantage of the fact that in v2 it can include any Kubernetes resource, materialises in a single pipeline the team’s namespace, the quota for nvidia.com/gpu and a KServe LLMInferenceService pointing at the model with vLLM runtime, and the developer only applies the InferenceEndpoint with four fields, knowing nothing about KServe or the quotas underneath. KServe supports vLLM and GPU natively, as we saw in the article on KServe.

The honest reading is that you are at the frontier, not on well-trodden ground. Outside Modelplane, made by Upbound, there is still no body of independent cases assembling Crossplane with vLLM, and the KServe type is in alpha. It is a coherent architecture, with a real reference backing it and plenty of room for you to be among the first to document it on your own iron.

What it costs you to operate your own control plane

No tool that gives you back control is free, and Crossplane has well-documented costs you need to budget for beforehand, not discover afterwards.

The learning curve is steep, acknowledged even in the coverage of its graduation. Adopters report debugging difficulties when a composition or a provider misbehaves, and they miss Terraform’s module ecosystem.

Compositions tend towards complexity. There are practitioners describing definitions spanning thousands of lines with poor navigation, and the absence of an integrated testing framework. Functions improve this, but as I said, they shift the complexity rather than remove it.

There is no plan. Unlike Terraform’s plan and apply cycle, Crossplane applies changes immediately. You lose the safety net of reviewing what is going to happen before it happens, with the risk of modifications you did not want.

Operating the control plane itself costs. Big providers create hundreds of resource definitions (the GCP one reached 347), with a risk of API server unavailability and of leaving your cluster hanging. State lives in etcd, which complicates disaster recovery and migrating control between clusters. And there is one specific danger: accidental deletion of cloud resources when upgrading XRDs, providers or compositions, because of the ownership Crossplane keeps over what it created.

Some have decided not to adopt it. The Masterpoint team ruled it out for the lack of data sources and ignore_changes, for incomplete coverage of AWS services, and for the cost of abandoning their accumulated Terraform experience. It is a legitimate decision, and quoting it to you is part of an honest article.

How it lives alongside what you already have

If you already run a platform, your question is not Crossplane yes or no, but Crossplane with what. Three relationships matter to you.

With Backstage they are complementary, not rivals. Backstage is the portal and the templates that generate manifests or pull requests, day zero; Crossplane provisions and reconciles the real state, day two. The usual pattern, which some call the golden triangle, is Backstage as the interface, Crossplane as the control plane and Argo CD or Flux as GitOps. Each one does a layer.

With Flux, which you probably already use, the relationship is additive. Flux reconciles manifests towards the clusters, but it does not give you a platform abstraction API: it does not know what “a database” is, it only applies the YAML you hand it. Crossplane adds exactly that abstraction layer, and the two coexist, with Flux deploying the composite resources and Crossplane materialising them. If you already have Flux, Crossplane’s value is precisely that self-service API layer, in exchange for the cost of operating the control plane I described above.

With KubeVela and Score, the subject of the next article, the boundary is one of layer: Crossplane answers “give me a Postgres”, from the infrastructure side; KubeVela and Score answer “deploy my application”, from the workload side. They do not compete, they stack.

For your sovereign inference factory

The practical decision boils down to this. Crossplane makes sense when you want to expose a self-service API of your own over your on-premise infrastructure and you are willing to operate a control plane to get it. If your platform is used by two teams who know each other and you provision with a tidy Flux repository, Crossplane is cost without return for now. The trigger is scale: when manual provisioning turns into a ticket bottleneck, the declarative API starts paying back its cost.

Your sensible entry route is not rewriting everything. It is starting with a provider that already serves you (provider-kubernetes and provider-helm cover a lot), wrapping your existing Terraform with the corresponding provider instead of throwing it away, and modelling a single abstraction of clear value, probably the InferenceEndpoint we discussed, as a golden path. From there, you grow on demand. And always with a clear account of having traded the safety net of plan and a handful of scripts for a thermostat you have to keep switched on.

The next article moves up a layer, from the infrastructure resource to the application workload, with KubeVela and Score, which is where the friendly CLI that Crossplane deliberately does not give you finally appears.

Sources

CNCF, Announces Graduation of Crossplane (6-nov-2025) — https://www.cncf.io/announcements/2025/11/06/cloud-native-computing-foundation-announces-graduation-of-crossplane/
Crossplane Blog, Announcing Crossplane’s CNCF Graduation — https://blog.crossplane.io/crossplane-cncf-graduation/
Crossplane Blog, Announcing Crossplane 2.0 — https://blog.crossplane.io/announcing-crossplane-2-0/
Crossplane Docs, What’s New in v2 — https://docs.crossplane.io/latest/whats-new/
Crossplane Docs, Compositions — https://docs.crossplane.io/latest/composition/compositions/
Crossplane Docs, Release Cycle — https://docs.crossplane.io/latest/learn/release-cycle/
Crossplane Docs, Crossplane CLI — https://docs.crossplane.io/latest/cli/
Upbound, Crossplane Graduates From CNCF, Upbound Redefines AI-Native Infrastructure — https://www.upbound.io/blog/crossplane-graduates-from-cncf-upbound-redefines-ai-native-infrastructure
InfoQ, Crossplane Reaches Production Maturity by Graduating CNCF — https://www.infoq.com/news/2025/11/crossplane-grad/
Pulumi Docs, Pulumi vs. Crossplane — https://www.pulumi.com/docs/iac/comparisons/crossplane/
Upbound Marketplace, provider-terraform — https://marketplace.upbound.io/providers/upbound/provider-terraform/latest
GitHub, provider-kubernetes — https://github.com/crossplane-contrib/provider-kubernetes
GitHub, provider-helm — https://github.com/crossplane-contrib/provider-helm
GitHub, provider-keycloak — https://github.com/crossplane-contrib/provider-keycloak
VSHN, provider-minio — https://github.com/vshn/provider-minio
Masterpoint, Crossplane: Why it Didn’t Work for Us — https://masterpoint.io/blog/passing-on-crossplane/
CECG, Crossplane: the good, the bad and the ugly — https://www.cecg.io/blog/crossplane-the-good-the-bad-the-ugly
Taloflow, Backstage vs Crossplane for Platform Engineering — https://www.taloflow.ai/guides/comparisons/backstage-vs-crossplane-platform-engineering
Crossplane Blog, Building Modelplane on Crossplane — https://blog.crossplane.io/building-modelplane/
Modelplane — https://modelplane.ai · https://github.com/modelplaneai/modelplane
KServe Docs, Understanding LLMInferenceService — https://kserve.github.io/website/docs/model-serving/generative-inference/llmisvc/llmisvc-overview

Kubeflow in depth: which pieces are worth it in an on-premise LLM platform and which ones you already have

Mon, 31 Aug 2026 09:30:00 +0200

Second post in the platform and self-service run, after Backstage and before Knative. The question here is not what Kubeflow is, but which parts of Kubeflow still make sense once you have built half a platform on your own.

TL;DR

Kubeflow graduated in the CNCF on 17 August 2026, and that headline hides the important one: it is no longer a monolithic platform, it is an umbrella of independent subprojects, each with its own repository, its own version and its own life. That changes how you have to evaluate it entirely. The 2020 question, should I install Kubeflow?, has become the 2026 one, which piece of Kubeflow do I install, and at what cost?

For a platform that already has RKE2, vLLM, KServe, Flux and Volcano or Kueue, the honest answer is that Kubeflow as a whole duplicates almost everything you already have. KServe no longer lives under Kubeflow: it left, and since November 2025 it has been a CNCF project in its own right, so anyone serving models with KServe does not need Kubeflow to serve. The Trainer leans on Volcano and Kueue, it does not replace them. And the full manifest drags in Istio, Dex and cert-manager as hard dependencies that clash with what you already operate. The only thing that may be worth installing on its own is Pipelines, as an orchestrator for data and training flows, and perhaps the Trainer for its fine-tuning templates. The rest you either already have, or do not need.

The analogy: the hypermarket and the specialist shop

Kubeflow was born as a hypermarket: a single building with everything an ML team might need, from notebooks to the inference server, taking in pipelines and hyperparameter tuning along the way. It made sense in 2019, when assembling each piece separately was a project in itself and there were no standards.

By 2026 the hypermarket has been converted into a street of specialist shops. The butcher’s counter, KServe, went independent, opened its own premises and is doing better alone. The fishmonger’s, the Trainer, now depends on the central market next door, Volcano and Kueue, instead of having its own cold store. And the shared building still demands that you put in its wiring, its plumbing and its security system (Istio, cert-manager, Dex) even though you already have your own.

Anyone arriving today with a half-full fridge, which is the case for anyone with a platform already running, does not need the whole hypermarket. They need to know which shop to walk into for what they are missing, and to avoid buying twice what they already have at home.

What Kubeflow is in 2026

Kubeflow was created by Google in 2017 and graduated in the CNCF on 17 August 2026, reaching the top level alongside Kubernetes or Prometheus. The graduation figures are serious: more than 6,600 contributors from over a thousand organisations, and adopters such as Bloomberg, NVIDIA, Red Hat or Spotify.

The structural change is the one that matters for the decision. The full distribution, the Kubeflow Community Distribution, is numbered by year and month (the current one is 26.03, from March 2026, on a six-monthly cadence), but each component is by now a subproject with its own repository and its own cycle, usable independently. The distribution only packages them together. That means “installing Kubeflow” is no longer an atomic decision: it is a shopping list.

Component by component

Kubeflow Pipelines: the piece you may actually want

Pipelines orchestrates containerised ML workflows. They are written in Python with its SDK, which compiles to an intermediate YAML representation so they are portable, and underneath they run on Argo Workflows. It is the piece most likely to add value to an existing platform: a flow of “ingest, embeddings, fine-tuning, evaluation, deployment” is exactly what you need to operate RAG and adapters repeatably.

The small print has two parts. The first is that if you already use plain Argo Workflows, Pipelines adds a layer of SDK, interface and metadata on top of the same engine: it does not bring a new scheduler, it brings the typed component model and the runs interface. The second is that its metadata and lineage system, MLMD, is being removed from Pipelines itself, so it is not wise to build the platform’s traceability on that particular piece.

Kubeflow Trainer: it leans on your scheduler, it does not replace it

The old Training Operator, the one with PyTorchJob and TFJob, has been rewritten as Kubeflow Trainer, with a single TrainJob API that unifies all frameworks. Version 2.2 positions itself explicitly for distributed training and LLM fine-tuning, with support for PyTorch, DeepSpeed, HuggingFace and company.

A common expectation needs correcting here. The Trainer does not replace Volcano or Kueue: it leans on them. The TrainJob carries a podGroupPolicy field that automatically creates the Volcano PodGroup objects for gang scheduling, and it integrates with Kueue’s queues. If you already built that scheduler, as we saw in its post, the Trainer adds the TrainJob abstraction and some fine-tuning templates on top of what you already have, not a new scheduling layer. It may be worth it for the templates; not for the scheduler.

Katib: only half relevant

Katib does Kubernetes-native AutoML: hyperparameter optimisation with algorithms such as Bayesian optimisation, TPE or Hyperband, plus architecture search. In the LLM era it has repositioned itself, with official documentation for tuning fine-tuning hyperparameters and even for tuning RAG pipelines.

The practical reading is lukewarm. Architecture search is irrelevant for anyone doing LoRA and RAG. Hyperparameter optimisation can be useful for learning-rate or LoRA-rank sweeps, but it competes with doing it from the SDK of the QLoRA runbook itself or with Ray Tune, and it rarely justifies dragging in Katib for that alone.

KServe: the piece that is no longer in Kubeflow

This is the fact that settles half the evaluation. KServe left Kubeflow. It was born inside the project in 2019, was donated to the Linux Foundation in 2022, was renamed from KFServing to KServe and graduated from Kubeflow that same year, and since November 2025 it has been a CNCF incubating project in its own right. In the 2026 taxonomy it appears as an ecosystem project, external, not as a core component.

The consequence is direct: anyone already serving with KServe does not need Kubeflow to serve. And KServe is exactly where the LLM action is: its 0.15 release strengthened the vLLM backend and introduced an LLMInferenceService resource with disaggregated serving, prefix caching, per-variant autoscaling and OpenAI-compatible APIs. If your serving layer is vLLM on KServe, you already have that layer complete and Kubeflow adds nothing to it.

Model Registry, Notebooks and the rest

The Model Registry, now reorganised under the name Kubeflow Hub, is an index of models, versions and metadata. It stores pointers and states, not the bytes. Against MLflow, it lacks the experiment and metric tracking that is MLflow’s strength, to the point that the guides recommend combining them rather than substituting one for the other; and against an OCI registry, it does not package the binary, which still lives wherever you put it. It is still a young component, below version 1.0.

The Notebooks live in two versions, the stable 1 and a 2 redesigned on CRDs that is still in alpha. The Spark Operator remains in the core; Feast, by contrast, has left the core and is now an ecosystem project, a nuance worth being clear about before counting on it.

Multi-tenancy: powerful and tied to Istio

Isolation between teams is done with Profiles, a CRD that wraps a namespace and gives it RoleBindings, ServiceAccounts and Istio authorisation policies that validate an identity header derived from OIDC. It is a complete model, but it has an Achilles heel: it depends entirely on the Istio sidecar and on that header. If traffic bypasses the mesh or someone forges the header, isolation breaks, and all multi-tenant security ends up coupled to Istio, which clashes head-on if you already operate another mesh or no service mesh at all.

The cost of installing the whole thing

Here is the main reason not to install the full distribution on top of an existing platform. The 26.03 manifest drags in as hard dependencies: Istio, cert-manager, Dex for OIDC, OAuth2-Proxy, and Knative Serving and Eventing for KServe. It recommends 16 GB of RAM and 8 vCPU as a minimum, and the aggregate of all components comes to around 4.4 CPU cores and 12 GB of memory for the control plane alone.

The project itself acknowledges the problem. There is an open thread in its repository, bluntly titled “community feedback on Kubeflow’s complexity”, where it is admitted that Kubeflow practically forces a dedicated cluster because of the assumptions it makes about what is installed, that Istio and Dex should be swappable and not fixed dependencies, and that a lighter Helm that installs only what is needed is missing. For anyone already operating cert-manager, an ingress and GitOps with Flux, the full manifest does not coexist: it duplicates and tramples the base infrastructure.

Kubeflow against the alternatives

The field is wide and almost all the comparative literature is written by vendors with a product of their own, so it has to be read with tweezers. With that caution:

Plain Argo Workflows is the engine Pipelines uses underneath; if all you need is container graphs, Pipelines is overhead. MLflow is strong in experiment tracking and lightweight registry, installs in minutes, and does not orchestrate distributed training: it is complementary, and in fact we use it in the post on prompt versioning. Flyte is a typed orchestrator that many perceive as lighter to operate than full Kubeflow. And Ray on Kubernetes is the most serious competitor for LLM workloads, because with a single runtime it covers training, serving and tuning, overlapping at once with the Trainer, Katib and KServe.

Where Kubeflow wins is in breadth and in governance: a CNCF-graduated umbrella with adoption by large companies. Where it loses is in operational weight, in hard dependencies, and in the fact that its best pieces are already independent projects that do not require the rest.

The real fit with LLM workloads

The project has moved towards GenAI, and that has to be acknowledged: there is an SDK with LLM fine-tuning templates, the Trainer adds support for reinforcement post-training methods, and new agent-oriented pieces have appeared. It is not empty marketing, there is real work there.

But the question for a specific platform is not whether Kubeflow does LLM: it is which part I have not already duplicated. On a stack of RKE2, vLLM, KServe, Flux and Volcano or Kueue, the balance looks like this:

Component	Does it add anything new to your stack?
KServe	No: you already have it, and it is no longer Kubeflow’s
Trainer	Only the fine-tuning templates; you already have the scheduler
Katib	Marginal: LoRA HP sweeps, little more
Pipelines	Possibly yes, as a flow orchestrator, but it runs on Argo
Model Registry	Optional: competes with MLflow and with your OCI registry
Notebooks	Depends on what you use for notebooks today
Multi-tenancy	No, if you already isolate by namespace without tying everything to Istio

Operational traps and honest scepticism

Do not install the full manifest on a platform that already works. You are going to duplicate Istio, cert-manager and the rest, and fight a dependency war with no prize at the end. If something in Kubeflow interests you, install it as a standalone subproject.

KServe is not Kubeflow. It is the most common category error of 2026. If someone justifies building all of Kubeflow “to serve models”, you already know they have not checked that KServe left long ago and lives on its own.

The Trainer does not save you the scheduler. It needs Volcano or Kueue underneath. If you expected it to solve GPU gang scheduling by itself, it does not.

Multi-tenancy ties you to Istio. Before adopting Profiles, ask yourself whether you want security between teams to depend on one specific mesh. If you already isolate well by namespace and RBAC, you may not need that layer.

MLMD is on its way out. Do not build your pipelines’ traceability on the metadata system the project itself is retiring.

For an inference factory

The conclusion is comfortable to apply. Kubeflow in 2026 is not a yes-or-no decision, it is a menu you order from à la carte. On a platform that already serves with KServe, schedules with Volcano and deploys with Flux, the menu shrinks a great deal.

If you need to orchestrate data and training flows repeatably, and you do not want to build plain Argo Workflows with its ergonomics, Pipelines is the piece worth evaluating, installed standalone. If you do distributed fine-tuning frequently and want a uniform abstraction on top of your scheduler, the Trainer may save you some templates, knowing that it leans on the Volcano you already have. Everything else you either have, or a specialist piece covers better.

The expensive mistake would be installing the whole hypermarket to buy bread. Kubeflow stopped being that; treat it as what it is, a street of shops, and walk only into the one you are missing. The next piece in this run, Knative, is precisely one of those specialist shops, the one that decides whether switching off the GPU when nobody is using it pays off or costs you dear.

Sources

CNCF, CNCF announces Kubeflow’s graduation — https://www.cncf.io/announcements/2026/08/17/cncf-announces-kubeflows-graduation-solidifying-the-standard-for-cloud-native-ai-operations/
CNCF, Kubeflow unveils new cloud native innovations to supercharge AI — https://www.cncf.io/blog/2026/07/28/kubeflow-unveils-new-cloud-native-innovations-to-supercharge-ai/
CNCF, KServe becomes a CNCF incubating project — https://www.cncf.io/blog/2025/11/11/kserve-becomes-a-cncf-incubating-project/
Kubeflow Docs, Introduction / components — https://www.kubeflow.org/docs/started/introduction/
Kubeflow Docs, Pipelines overview — https://www.kubeflow.org/docs/components/pipelines/overview/
Kubeflow Trainer Docs, Volcano gang scheduling and Kueue — https://trainer.kubeflow.org/en/latest/operator-guides/job-scheduling/volcano.html
Kubeflow Docs, Migrating to Kubeflow Trainer v2 — https://www.kubeflow.org/docs/components/trainer/operator-guides/migration/
Kubeflow Docs, Katib overview — https://www.kubeflow.org/docs/components/katib/overview/
Kubeflow Docs, Hyperparameter optimization for LLM fine-tuning — https://www.kubeflow.org/docs/components/katib/user-guides/llm-hp-optimization/
Kubeflow Docs, Model Registry overview — https://www.kubeflow.org/docs/components/model-registry/overview/
Kubeflow Docs, Multi-tenancy design — https://www.kubeflow.org/docs/concepts/multi-tenancy/design/
GitHub, kubeflow/manifests (dependencias y recursos) — https://github.com/kubeflow/manifests
GitHub, kubeflow/manifests #2451: community feedback on complexity — https://github.com/kubeflow/manifests/issues/2451
InfoQ, Kubeflow expands AI capabilities as CNCF graduation nears — https://www.infoq.com/news/2026/08/kubeflow/

From public cloud to private: what you really lose and which projects fill the gap

Mon, 31 Aug 2026 09:15:00 +0200

With this one I open a run about the biggest gap a sovereign platform leaves you: how to give your teams the self-service experience of a public cloud, without the public cloud. Here you have the map of the problem; the next two go down into the detail of the tools, Crossplane as the infrastructure control plane and KubeVela with Score as application-centric self-service.

TL;DR

Leaving the public cloud has a solid business case, and I am not making it up. 37signals declared savings of close to two million dollars in 2024 after repatriating, the classic a16z analysis puts cloud spend at around half of the cost of revenue of a software company, and the Barclays CIO survey of 2024 recorded the highest repatriation intent in its history. If your workload is a predictable, sustained base (which is exactly the continuous LLM inference this blog is about) the numbers work out, and in Europe the sovereignty reasons pile on top: the Data Act, the tension with the US CLOUD Act, and that dependence on three hyperscalers that control around 70 % of the European cloud market.

But the problem is not the compute. Your on-premise GPU rack serves tokens just as well as a rented one. What the cloud was selling you and you now have to rebuild is something else: the elasticity of asking for a hundred machines at ten and handing them back at eleven, the catalogue of hundreds of managed services with an on-call team behind them, the unified provisioning API with fine-grained permissions, the multi-region network, the bill that arrives itemised, and that model where operating all of it was somebody else’s problem. All of that you rebuild piece by piece with open source projects, and the rebuild has a cost that the savings figures do not usually tell you about. What follows is the map of what you lose, which project fills each gap, and where a gap remains that nothing will close completely.

The analogy: leaving the hotel to set up your own house

Think of it like this: living in the public cloud is living in a hotel. You pay a lot per night, but you do not change light bulbs, reception is open at three in the morning, if you need another room you ask for it and it appears, and when you leave you take nothing with you to maintain. Setting up your own house works out much cheaper per square metre if you are going to stay for years, and on top of that you decide who comes in and where your things are kept. The problem arrives the day a pipe bursts: then you do not call reception, because reception is you.

Repatriation is that move. The savings are real and so is data ownership, but what in the hotel came included in the price of the night is now your to-do list: the plumbing is your network, the boiler is your storage, the concierge who got you anything is the catalogue of managed services you no longer have, and the itemised bill they slid under your door is a cost model you have to build, because electricity and depreciation do not arrive broken down per tenant. The house pays off, but only if you count what it costs to furnish it and fix the pipes, not just the price per square metre against the hotel.

The rest of the article walks through that to-do list and tells you, for each item, which open source tool does the job the hotel used to do for you.

Why this conversation belongs to 2026, not to always

Repatriating is nothing new, but in 2026 three things come together that make it urgent if you run a European AI platform.

The first is economic, and Andreessen Horowitz put it on the table in 2021 with The Cost of Cloud, a Trillion Dollar Paradox. Their thesis, which you should read knowing that a16z has interests in the ecosystem it analyses, is that cloud spend averages around 50 % of the cost of revenue in a software company, and that recovering half of that spend would free up margins the market penalises. The cleanest case, and one you can verify, is 37signals: DHH published that their cloud bill dropped from 3.2 to 1.3 million dollars a year after repatriating, with an initial investment of about 700,000 dollars in Dell servers. Now the honest caveat, which the specialist press itself underlines: those savings figures did not include the extra operations staff, nor the energy, nor the cooling. I come back to it in the hidden costs section.

The second is declared intent. The Barclays CIO survey from the first half of 2024 recorded that 83 % planned to repatriate some workload to private cloud or on-premise in the following twelve months, the highest reading in the series. But read that 83 % precisely: it is the proportion of managers who plan to move some workload, not the percentage of workloads or of spend that moves. Flexera, in its 2025 report, gives you the more grounded figure: large companies have repatriated around 21 % of the workloads they had in the public cloud, and cost saving is priority number one for almost 60 %.

The third is regulatory and European, and it is the one that turns repatriation into something more than a FinOps decision. The European Union Data Act came into force in January 2024 and has applied since September 2025; it mandates portability between providers and, from January 2027, bans the exit fees that today chain you to your provider. And underneath it beats a deeper legal tension: the US CLOUD Act allows United States authorities to demand data from an American provider even if it is stored in Frankfurt, which is exactly the argument for why a European region of a hyperscaler does not give you sovereignty. With AWS, Microsoft and Google controlling around 70 % of the European cloud market, the conversation about sovereign cloud stopped being theoretical. I developed this already in sovereign on-premise versus hyperscalers.

What you lose when you leave the cloud, specifically

Let us get to the core, ordered by what is going to hurt you most. And one thing before starting: none of these gaps is about compute. They are all about what the cloud put around the compute.

Hardware elasticity disappears. On-premise, the GPU you did not buy does not exist. The cloud bills you by the hour precisely because it takes on the risk of idle capacity: it lets you ask for a peak and hand it back. With your own iron, you buy the peak up front and pay for it whether it is in use or not, so your profitability depends on high sustained utilisation. Below a certain usage threshold, the cloud beats you again. This is the irreducible gap, the one no tool fills, and if you repatriate a workload with extreme peaks you are making the wrong decision.

Managed services become your operations. Every box you took for granted (the database, the queue, the object storage, the load balancer, DNS, the secrets manager) stops being a checkbox you tick and becomes a system you install, update and watch yourself. The 37signals case teaches it well: getting out of S3 object storage was the last and hardest part, with petabytes of data and a contract they dragged along for years. Your managed Postgres turns into operating CloudNativePG; your S3 turns into operating MinIO; and so on with every service.

You have to build the provisioning control plane yourself. AWS gives you a unified API where you ask for a resource with fine-grained permissions and it appears. On-premise that does not come for free: you have to build the control plane that translates “I want a database” into real resources. It is exactly the gap that Crossplane fills, and that is why it has its own article.

You replace the managed network piece by piece. The VPC, the security groups, NAT, the load balancer: every network abstraction in the cloud has its on-premise equivalent (Cilium for network policies on eBPF, MetalLB for the load balancer without ELB, Gateway API for L7 ingress), but they are pieces you assemble, not a service you switch on. A good part of this we already saw in the networking vertical with Cilium and eBPF.

Billing stops existing as such. In the cloud the cost reaches you broken down by service and by tag; you have a Cost Explorer that answers “how much does this team cost me”. On-premise there is no natural bill: there is hardware depreciation, an electricity bill and some payroll, and splitting that per workload is a model you have to build. OpenCost helps you with the split per workload inside the cluster, but the total physical cost (depreciation plus energy plus staff) you model separately, as we saw in OpenCost and cost allocation.

You design multi-zone availability yourself. The three availability zones the cloud gives you for free turn into several sites, several racks and several utility feeds that you have to plan. 37signals solved it with two datacenters and replication between them. It is neither free nor automatic.

And above all, the on-call is yours. The cloud was not selling you only compute: it was selling you operations included. When something falls over at three in the morning, in the cloud there is an on-call team at the provider; with open source, that team is you. This is the cost that savings spreadsheets almost never include, and it is the one that decides whether your repatriation was a good idea or a hidden bill.

The map of projects that fill each gap

With the gaps ordered, the catalogue of substitutes reads itself. I group it by layer, from the lowest to the closest to the developer.

In the infrastructure substrate, what replicates EC2 virtual machines, virtual networking and disks, you have OpenStack as the classic standard for a complete private cloud, SUSE’s Harvester as bare-metal hyperconvergence on KubeVirt managed from Kubernetes, and emerging projects such as Spinifex that rebuild an AWS-compatible API on your own iron. It is the heaviest layer and the one I touch least in this blog, because I assume your cluster already exists.

In the provisioning control plane, what replicates the AWS IAM and resource API, there is Crossplane, graduated in the CNCF in November 2025, which brings the Kubernetes declarative API model to infrastructure. Alongside it, Cluster API for declarative cluster provisioning. And so you can see how far this reaches in our own field, the Crossplane team itself published Modelplane in mid-2026, an inference control plane built on compositions that serves models with vLLM. This layer is developed by the Crossplane article.

In the developer self-service experience, what replicates the AWS console and CLI, you have three different approaches you should not confuse. Backstage is the web portal, the catalogue and the templates, and it already has its own article. KubeVela brings self-service to the application model with a friendly CLI. Score defines a workload specification that is portable between environments. KubeVela and Score share the third article because they cover the same layer from angles that complement each other.

In on-premise managed services, CloudNativePG operates Postgres for you with high availability and backups to object storage, MinIO gives you S3-compatible storage, and OpenCost covers that cost allocation you lost with Cost Explorer.

The way to read this map is that none of these pieces is surplus if you want the cloud experience: they are the boxes the cloud ticked for you and that you now tick one by one.

The gap you will not close

I will tell you plainly, because it is the part the “leave the cloud” brochures skip. No combination of open source projects reproduces 100 % of the public cloud for you, and not for lack of maturity, but by design.

What you will not replicate is, first, instant hardware elasticity: you still buy the peaks up front. Second, the depth and breadth of the managed services catalogue, hundreds of services with a service level agreement and an on-call team behind them, which no internal team matches. Third, the global multi-region network the cloud gives you almost for free. And fourth, that model of transferred operational responsibility: with open source, operations come back home.

The correct reading is not that repatriating is a bad idea, but that you swap a predictable operating expense and a delegated responsibility for a cheaper capital investment and a responsibility of your own. For sustained base load it works out. For everything else, think about it.

When NOT to repatriate

This section has to be here, because repatriation badly applied is expensive. There are three clear signs that the cloud is still your right answer.

The first is a workload with extreme peaks or very variable traffic. Most inference deployments operate between 40 and 65 % GPU utilisation, and below roughly 70 % sustained usage the cloud beats you on total cost. If your workload has long troughs, you are buying iron so it can sleep.

The second is uncertainty or the start of a project. The break-even point in favour of buying only applies to you when utilisation is consistently high and, above all, measured and not projected. Buying GPUs against an optimistic forecast is the fastest way to end up with a warehouse of idle silicon. Sizing it properly is worth a whole article: sizing and justifying the GPU investment.

The third is the lack of an operations team. The total cost of ownership on-premise includes between half and one full-time engineer per GPU server. Without that team, your saving is illusory, because the on-call the cloud included is now covered by nobody. a16z itself frames its thesis as a hybrid approach, not as “leave the cloud” at all.

The numbers, to ground it

So as not to leave you with the discussion in the abstract, here are the orders of magnitude, triangulating two sources with opposite biases, a cloud provider and a neutral FinOps tool. An H100 in the cloud runs you between 2.90 and almost 7 dollars per GPU-hour depending on provider; bought, an H100 card is around 31,000 dollars and an eight-way system between 250,000 and 320,000. At full utilisation, a rented H100 passes its purchase price in less than a year, and an eight-way system with second-hand hardware can pay for itself in about seven months. The three-year total cost of an eight-H100 system, counting staff, colocation, energy and cooling, is estimated at between 700,000 and 950,000 dollars, and the threshold where on-premise beats the hyperscalers is around 80 % sustained utilisation.

The two sources disagree on the exact number, as you would expect, but they agree on the deciding factor: sustained utilisation, measured and not projected. The full working of this calculation is in the TCO of an on-premise GPU cluster. Here the conclusion is enough for you: for continuous base load the numbers work out, and that is exactly the profile of an inference factory.

For your sovereign inference factory

Everything above arranges itself into a practical decision. If you operate an LLM inference platform on your own iron, you are not deciding whether to repatriate (you already took that decision when you built the cluster), but how to rebuild the cloud experience your teams miss.

The sensible sequence starts with the substrate, which you probably already have in the form of RKE2 and storage. It goes on to the provisioning control plane, where Crossplane turns “open a ticket and wait” into a declarative API. It continues with the developer experience, where you choose between the Backstage portal, the KubeVela application model or the portable Score specification, depending on whether your people prefer a web or a CLI. And do not forget the on-premise managed services or the cost allocation, because without a bill your platform loses the discipline the cloud imposed by default.

The next two articles go down into the two layers least covered in this blog: Crossplane for the infrastructure control plane, and KubeVela with Score for CLI self-service centred on the application. With them, the map of this layer is covered, and your own house starts to look like the hotel you left, with the difference that the data stays inside.

Sources

Andreessen Horowitz (S. Wang, M. Casado), The Cost of Cloud, a Trillion Dollar Paradox — https://a16z.com/the-cost-of-cloud-a-trillion-dollar-paradox/
Data Center Dynamics, 37signals claims it saved almost 2M USD last year from cloud repatriation — https://www.datacenterdynamics.com/en/news/37signals-claims-it-saved-almost-2m-last-year-from-cloud-repatriation/
The Register, Developer pockets 2M USD in savings from going cloud-free — https://www.theregister.com/2024/10/21/37signals_aws_savings/
Barclays, Technology: 1H24 CIO Survey (PDF) — https://8198920.fs1.hubspotusercontent-na1.net/hubfs/8198920/Barclays_Cio_Survey_2024-1.pdf
Channelnomics, Breaking Down the 83% Public Cloud Repatriation Number — https://channelnomics.com/breaking-down-the-83-public-cloud-repatriation-number/
The New Stack, Updated Stats on Cloud Sustainability, Repatriation and Cost Optimization (Flexera 2025) — https://thenewstack.io/updated-stats-on-cloud-sustainability-repatriation-and-cost-optimization/
Comisión Europea, Data Act | Shaping Europe’s digital future — https://digital-strategy.ec.europa.eu/en/policies/data-act
EU Data Act, Article 29 — Gradual withdrawal of switching charges — https://www.eu-data-act.com/Data_Act_Article_29.html
Kiteworks, How the EU Data Act and GDPR Conflict with U.S. CLOUD Act — https://www.kiteworks.com/gdpr-compliance/eu-data-act-gdpr-cloud-conflict/
Computerworld, EU takes first steps to reduce reliance on US hyperscalers — https://www.computerworld.com/article/4181816/eu-takes-first-steps-to-reduce-reliance-on-us-hyperscalers.html
CNCF, Announces Graduation of Crossplane (6-nov-2025) — https://www.cncf.io/announcements/2025/11/06/cloud-native-computing-foundation-announces-graduation-of-crossplane/
CNCF, Score accepted as a CNCF Sandbox Project — https://www.cncf.io/blog/2024/08/08/score-accepted-as-a-cncf-sandbox-project/
Harvester HCI (SUSE) — https://harvesterhci.io/
CloudNativePG — https://cloudnative-pg.io/
Crossplane Blog, Building Modelplane on Crossplane — https://blog.crossplane.io/building-modelplane/ · https://modelplane.ai
Spheron, LLM Inference On-Premise vs GPU Cloud: 2026 Cost and Break-Even Analysis — https://www.spheron.network/blog/llm-inference-on-premise-vs-cloud/
CloudZero, H100 GPU Cost in 2026: Buy, Rent, and Cloud Pricing Compared — https://www.cloudzero.com/blog/h100-gpu-cost/

Backstage as a self-service portal: giving your teams the LLM platform without giving them the cluster

Mon, 31 Aug 2026 09:00:00 +0200

This article opens the platform and self-service layer: the one that decides how you hand everything underneath to the teams that are going to use it. The three articles in this batch are Backstage (the portal), Kubeflow (the ML toolbox) and Knative (switching the GPU off when nobody is using it).

TL;DR

Backstage solves a real problem of a platform that has grown: nobody knows what is there, who maintains it, or how to ask for a new one without opening a ticket. Its answer is three pieces, a software catalogue that models every component and its relations, templates that create new resources through a form instead of through copy and paste, and TechDocs to keep documentation next to the code. On top of those three sits the concept that holds up everything else, the golden path: the supported route made also the easiest one, so that people take it because it is paved, not because they are forced onto it.

The price is high and it should be said plainly. Backstage is not a product you install, it is a Node and React application that you compile and maintain as your own code, at a declared cost of two to three full-time engineers, and with upgrades as its biggest pain point. As of August 2026 it is still a CNCF incubating project, not graduated, and it has no native entity type for models, datasets or inference endpoints: that gets modelled by hand, or you wait for a proposal that is still open. The decision is not about features, it is about scale: below a certain organisation size, a portal costs more than it saves.

The analogy: the industrial estate directory

A single unit needs no sign. You know what is inside because you built it. An estate with two hundred units, on the other hand, needs a directory at the entrance: which company sits on each plot, what it does, who is responsible, and an office where you can request a new plot with the utility connections already in place instead of negotiating each one separately.

An LLM platform starts out as a single unit. One cluster, one model, the team that built it. Six months later it is an industrial estate: fifteen models served, three RAG pipelines, LoRA adapters everywhere, and a question that comes back in every meeting, who maintains this? Backstage is the directory for that estate, with two additions an ordinary directory does not have: the office that hands over turnkey plots, and the archive of blueprints for each unit. The catalogue is the directory, the templates are the office, TechDocs is the archive.

The analogy also warns about the cost. The directory does not maintain itself. Somebody has to update it when a company moves, and a directory with half the plots mislabelled is worse than no directory at all, because people stop trusting it. In Backstage that failure has a name, catalog rot, and it is the leading cause of a portal being abandoned.

What Backstage is and what state it reaches 2026 in

Backstage was born inside Spotify and donated to the CNCF, which accepted it in September 2020 and moved it to incubating level in March 2022. That is where it still is: as of August 2026 it is incubating, not graduated, a detail that matters when someone sells it as a de facto standard. It is by far the most widely adopted developer portal (the official 2025 review declares more than 3,400 organisations using it, 31,000 GitHub stars and more than 250 plugins), but adoption is not governance maturity, and a large share of those plugins is maintained by the community, not by the core.

The architecture has kept changing pieces under the feet of whoever operates it. The New Backend System stabilised in 2024 and is now the standard. The New Frontend System became the default mode for new applications in version 1.49, of March 2026. The cadence is one minor release a month, and that rhythm is precisely the origin of the biggest operational problem, which we will see further down.

The three pieces

The software catalogue

The heart of Backstage is an entity catalogue. Every piece of the platform is described in a catalog-info.yaml file that lives next to its code, and Backstage discovers it by crawling the repository. The entity types are fixed: Component (something that runs or compiles), API (a contract that a component exposes), Resource (infrastructure that a component needs), System (a set of components that form a unit), Domain (a set of systems), plus Group and User for ownership.

The power is in the relations rather than in the entities, which Backstage computes and navigates: ownedBy connects each thing to its team, providesApi and consumesApi draw who depends on whom, partOf groups components into systems, dependsOn traces the dependency graph. With that, the meeting question (who maintains this, and what breaks if we touch it?) stops being answered from memory and starts being answered by query.

There is a design decision that conditions everything that follows: Backstage advises against creating new entity types. Plugins are compiled expecting the usual types, and an invented type leaves them blind. The supported route is to reuse an existing type and distinguish it with the spec.type field (for example a Resource with type: llm-model), adding whatever custom fields you need to the spec.

The templates

The second piece is the Scaffolder, the template engine. A template is an entity of type Template with three parts: the parameters requested through a form, the steps that get executed, and the actions that do the real work (create a repository, write the catalog-info.yaml, register the component, trigger the GitOps pipeline). The developer goes to /create, picks a template, fills in the form, and in a single operation has the resource created, catalogued and deploying.

This is what turns a directory into an office. Without templates, Backstage only lists what already exists. With templates, it is the door through which new things are created, and that door is where the golden path materialises.

TechDocs

The third piece is TechDocs, documentation as code. Documents are written in Markdown next to the code, generated with MkDocs in the CI pipeline, and served inside the portal itself, linked from each component’s page. The advantage is in the incentives: documentation that lives far from the code rots; documentation that lives next to it and gets reviewed in the same pull request has some chance of staying true.

The golden path, which is what all of this is about

Spotify coined the term golden path for the recommended and supported way to build something. It has two halves: the blessed tooling, visible in the portal, and a step-by-step tutorial in the documentation. The idea, in the formulation that platform reports keep repeating, is that nobody forces you to use the paved road; people take it because it is paved.

The data supports the golden path being the deciding variable. The 2026 State of Platform Engineering report, covering 518 professionals, puts voluntary adoption above 80 % where there is a well-designed golden path and below 20 % where there is not, and estimates that around 70 % of platform initiatives fail to reach significant adoption without one. It is a figure that reorders priorities: a portal without golden paths is a pretty catalogue nobody uses.

In Backstage, a golden path is implemented as a Scaffolder template. For an LLM platform, a typical golden path would be “serve a new model”: the form asks for the model, the GPU size, the owning team and the SLO; the template creates the repository with the InferenceService, writes the catalog-info.yaml, registers it under the right system, and opens the pull request against the GitOps repository you already have with Flux. What used to be half a day of copying a colleague’s YAML and guessing the values becomes a two-minute form with the correct defaults.

Authentication and permissions

For a sovereign platform the identity piece matters as much as the catalogue. Backstage ships connectors out of the box for GitHub, GitLab, Google, Microsoft, Okta, generic OIDC and SAML, among others. For on-premise the relevant one is Keycloak, which fits in like any other OIDC provider, with one nuance worth keeping in mind: the Keycloak connector is maintained by the community, not by the core, so its maintenance and its tightness fall on a third-party plugin. Once you have set up Keycloak as we saw in the article on authenticating MCP, the portal hooks into the same identity issuer.

The permissions framework appears today as stable in the documentation and supports RBAC, ABAC and custom logic. The small print: by default the endpoints are not protected, and the permission policy is written by the integrator. The most widely used RBAC solution with a graphical interface comes with the Red Hat distribution, not with the core. In other words, serious access control means work on top of what comes in the box.

The real cost, no frills

This is where an honest article parts ways with the brochure. Backstage is not an installable product, it is a framework that you compile and maintain as your own code. That has measurable consequences.

The most quoted analysis, from the managed Backstage vendor Roadie, puts the effort at around three full-time engineers in the first year and two from the second onwards, with a time to production of six to twelve months in self-managed mode. The figure comes from a vendor with a commercial interest in you buying its managed version, so it should be taken as an order of magnitude and not as a neutral data point, but the direction is confirmed by independent sources: ongoing maintenance takes up around a fifth of engineering time, and upgrades are the biggest pain point, cited by more than half of those who adopt it. With one minor release a month and architectural pieces shifting under your feet, keeping up to date is a permanent job.

The other cost is about adoption, not engineering. A widely repeated critical analysis points out that many organisations settle at around 10 % internal usage, against the 99 % voluntary usage Spotify achieved at home, and attributes it to three recurring failures: the catalog rot we already mentioned, the lack of a product owner treating the portal as a product, and the illusion that Backstage can be adopted over what already exists when in practice it only standardises new things well. The sentence that sums up the disenchantment is that Backstage is a free portal in the same way you would have a free car if somebody tipped all the parts of a Chevrolet onto your desk.

Applied to an LLM platform: what is there and what is not

Here an expectation needs correcting. In 2026 Backstage has no native entities for models, datasets or inference endpoints. There are three things and none of them is what you would hope for:

An open proposal (RFC 33060, from February 2026) to model model servers as entities of type API with type: ai-model-server. It is under discussion, not implemented.

An official module that introduces an AiResource type, which sounds like what we are after but is for something else: governing the skills and rules of coding assistants, not registering inference models.

And one real pattern, the MLOps templates that AWS documents on top of SageMaker, where Backstage acts as the Scaffolder that creates environments and pipelines, not as the model registry. It is a cloud case, not on-premise, and it uses Backstage for its template half, not for its model catalogue.

The practical conclusion for your own stack is twofold. First, what Backstage brings to an LLM platform today is above all the template half: the “create a new inference service” golden path wired into your GitOps. Second, if you want to catalogue models, today that is done by hand with custom spec.type values, treating the model as a Resource and its endpoint as an API, until the proposal matures. Registering the model’s bytes remains the job of the OCI registry, not of the catalogue.

The infrastructure plugins that are worth it in this stack do exist and are reasonably alive: the Kubernetes one (core), the Flux one for seeing GitOps state from the portal, the Grafana one for embedding dashboards in a component’s page, and the ArgoCD one for anyone using Argo instead of Flux.

Alternatives: when not to set up Backstage

The honest decision is not which portal, but whether a portal. The SaaS-type alternatives (Port, Cortex, OpsLevel, Humanitec) win on time to first value, measured in days instead of months, and on not having to maintain the framework, in exchange for a per-user fee that scales badly and for giving up pure on-premise. Roadie offers managed Backstage, which removes the framework maintenance while keeping its extensibility, in exchange for a per-developer fee and for taking your catalogue outside.

For a sovereign platform, where on-premise and data ownership are requirements, pure SaaS is usually ruled out by design, which leaves the real choice between self-managed Backstage and no portal at all. And there the criterion is size. Below a handful of teams, a well-maintained README and a tidy GitOps repository do the same job as the catalogue, without the cost of two engineers. The portal starts paying off when the cognitive load of knowing what is there and how to ask for it exceeds the cost of maintaining the tool that answers it.

Operational traps and honest scepticism

The catalogue rots if it is not automated. A catalog-info.yaml written by hand and never updated is lying within a few weeks. The only defence is generating and validating those files in the pipeline, not trusting people to maintain them.

Without a product owner, the portal dies. Backstage is not infrastructure you install and forget, it is an internal product with users you have to understand. Without somebody treating adoption as an objective, it ends up as a catalogue nobody looks at.

Upgrades are a recurring tax. One release a month and architectural pieces mid-migration mean the portal is never finished. That maintenance has to be budgeted from the start, not discovered in the second quarter.

Backstage deploys nothing. It is a portal, not an engine. Behind each template there has to be your GitOps, your operators and your cluster doing the real work. If that base is not solid, the portal just puts a pretty façade over chaos.

The data model for AI does not exist yet. Anyone setting up Backstage expecting a ready-to-use model catalogue is going to find themselves modelling by hand. It is doable, but it is work, and you should know that beforehand.

For an inference factory

Backstage fits at the high maturity level of a platform, not at the start. In the five maturity levels it is a tool of the level where the platform already serves several teams and the bottleneck has stopped being technical and become one of coordination. Three concrete decisions follow from all of the above.

The first is not setting it up too early. If the platform is used by two teams who know each other, the portal is cost without return. The trigger is cognitive load, not the calendar.

The second is starting with the templates, not with the catalogue. The value you notice from day one is the golden path that turns half a day of copying YAML into a two-minute form. The complete catalogue, with all its relations, is a bigger job that can come later.

The third is budgeting maintenance as a function, not as a project. Two part-time engineers indefinitely, an owner watching adoption, and automatic generation of the catalogue entries. Without that, the portal degrades until somebody asks why nobody uses it, and the answer is that it stopped being true.

The portal is the face your platform shows to whoever uses it. It is worth it when there is enough platform behind it that showing it tidily saves more than maintaining the shop window costs. Before that, it is the shop window of a shop that has no stock yet. The other two articles in this batch go into that stock: the ML toolbox with Kubeflow and the savings of switching the GPU off with Knative.

Sources

CNCF, Backstage (project page) — https://www.cncf.io/projects/backstage/
Backstage, Backstage Wrapped 2025 — https://backstage.io/blog/2025/12/30/backstage-wrapped-2025/
Backstage Docs, Software Catalog: descriptor format — https://backstage.io/docs/features/software-catalog/descriptor-format/
Backstage Docs, Well-known relations — https://backstage.io/docs/features/software-catalog/well-known-relations/
Backstage Docs, Extending the model — https://backstage.io/docs/features/software-catalog/extending-the-model/
Backstage Docs, Software Templates — https://backstage.io/docs/features/software-templates/
Backstage Docs, TechDocs — https://backstage.io/docs/features/techdocs/
Backstage Docs, Authentication — https://backstage.io/docs/auth/
Backstage Docs, Permissions overview — https://backstage.io/docs/permissions/overview/
Backstage GitHub, RFC #33060: modeling AI model servers in the catalog — https://github.com/backstage/backstage/issues/33060
Roadie, Backstage: how much does it really cost? — https://roadie.io/blog/backstage-how-much-does-it-really-cost/
Earthly, Backstage is at the peak of its hype — https://earthly.dev/blog/backstage-is-at-peak-hype/
AWS Prescriptive Guidance, Accelerate MLOps with Backstage and SageMaker templates — https://docs.aws.amazon.com/prescriptive-guidance/latest/patterns/accelerate-mlops-with-backstage-and-sagemaker-templates.html
Spotify Engineering, How we use golden paths to solve fragmentation — https://engineering.atspotify.com/2020/08/how-we-use-golden-paths-to-solve-fragmentation-in-our-software-ecosystem
The New Stack, DORA 2024: AI and platform engineering fall short — https://thenewstack.io/dora-2024-ai-and-platform-engineering-fall-short/
bex.co, State of Platform Engineering Report Vol. 4: golden path — https://bex.co/blog/2026/08/06/platform-engineering-report-vol-4-golden-path

Speeding up model cold start: from minutes to seconds

Thu, 11 Jun 2026 13:00:00 +0000

This is the third batch of an operational series on squeezing a generic on-premise LLM cluster of 4×H100 SXM 80 GB with NVLink. The sister pieces in this batch are Multimodal: serving a VLM on-premise with vLLM, adding vision to the same engine, and FinOps and GPU multi-tenancy with LiteLLM, sharing out and charging for the GPU across teams. This post takes the problem that From disk to HBM laid out conceptually and turns it into a runbook: how you actually bring the cold start down, knob by knob, so that elasticity is usable.

TL;DR

The post From disk to HBM left an uncomfortable idea behind: moving the weights from disk to HBM is only one of the five line items of cold start. The other four, starting the process, creating the CUDA context, setting up the allocator and, the most expensive one that hardly anyone looks at, capturing the CUDA graphs and compiling the JIT kernels, are untouched by any fast loader. That is why a server that loads the weights in 6 s can still take 90 s to be ready. This runbook attacks all five. For the weights: safetensors (mmap, zero-copy, no bounce through a host FP32 buffer) versus the pickle of torch.load; Tensorizer (CoreWeave), which serialises the model into one file and streams it tensor by tensor straight to the GPU from object storage/HTTP/S3; and the Run:ai Model Streamer (from NVIDIA), which reads concurrently and overlaps the read with the H2D copy. All three are switched on with one flag, --load-format, in vLLM. For the non-weight part: torch.compile/Inductor compilation cache (persisted and reused across start-ups), bounding the CUDA graph capture to the batch sizes you really use (the default capture can eat ~54 s), and sleep mode as a shortcut that skips all five line items. The maths: $t = W/B$ for the weights, the saving from overlapping read and H2D instead of doing them in series, and a 70B cold start broken down item by item. And the economics: when scale-to-zero with pre-warm beats keeping a warm replica. On the generic 4×H100 SXM 80 GB cluster.

The analogy: opening the kitchen in the morning

A restaurant does not open by flicking a switch. Anyone who has worked in one knows it: opening time is decided by everything that happens beforehand, and almost none of it is “bringing in the food”.

The first cook arrives at a cold, dark place. They have to: turn on the lights and start the systems (the process init); fire up the ovens and let them come up to temperature (the CUDA context, which takes its time to become operational); get the boards out, sort the cold room, organise the work space (the memory allocator, which reserves and structures the HBM); go down to the store and bring up the produce, the crates of vegetables, the meat, the fish, and put it in its place (loading the weights from disk to HBM, the entire subject of From disk to HBM); and, the most forgotten of all, sharpen every knife, set up the mise en place, make the stocks, pre-heat the mother sauces, all the preparation that is not an ingredient but without which not a single dish goes out (CUDA graph capture and JIT compilation of the kernels).

Here is the trap that From disk to HBM already anticipated: if you only optimise the produce, you still open late. You can hire the best delivery service in the world, a blazingly fast service lift, crates that put themselves away. But if the cook is still sharpening knives and waiting for the ovens to come up to temperature, the kitchen does not serve the first dish. The produce arrived early and the restaurant is still shut.

Today’s runbook is, literally, the list of everything that has to be done for the kitchen to open on time, and which of those tasks can be sped up, which can be left done from the night before (cached), and when it pays better never to switch the kitchen off (warm replica) than to light it again every morning (scale-to-zero).

The five line items of cold start

From the moment an inference pod is born until it returns its first token, the clock runs in five different places. It is worth naming them, because each optimisation attacks one or two, never all of them:

Process init. Starting the Python interpreter, importing the engine (vLLM and its dependencies), parsing the configuration. Seconds, and it grows with the size of the environment and the CUDA/PyTorch imports.
CUDA context. The first cudaSetDevice / cudaFree(0) initialises the CUDA runtime against the driver: it loads the context and maps the GPU. It is not instant, on the order of one to several seconds per GPU depending on driver and number of devices.
Allocator. vLLM sets up its memory manager over the HBM (PyTorch’s caching allocator plus the KV-cache layout). Reserving and structuring tens of GB has its cost.
Weight loading disk→HBM. The journey that From disk to HBM dissected: disk → page cache → host buffer → PCIe → HBM. What almost everyone believes cold start is, and it is only one of the five.
CUDA graph capture + kernel JIT. vLLM captures CUDA graphs for the various batch shapes and compiles kernels with torch.compile/Inductor (and backends such as DeepGEMM or FlashInfer). This item is the great ignored one: the default graph capture in vLLM takes on the order of 54 s because it covers a wide range of batch sizes (vLLM, torch.compile integration; Red Hat, vLLM with torch.compile).

The lesson the diagram is shouting: on a modern server with torch.compile and CUDA graphs, item 5 can be as big as or bigger than item 4. Speeding up weight loading from 40 s to 6 s is a huge advance, but if you leave graph capture at 50 s, the cold start falls from 104 s to ~70 s, not to 12 s. You have to attack both halves.

Speeding up item 4: loading the weights

safetensors versus pickle: the format rules

The first knob is the on-disk format. The old torch.load uses pickle, which has two problems. The security one is well known: deserialising a pickle executes arbitrary Python code, since the __reduce__ protocol lets the file invoke any callable on load, so every model download was a remote execution vector (HuggingFace, Safetensors). The performance one is what concerns us: loading a pickle rebuilds Python objects and usually passes through an intermediate host buffer before reaching the GPU.

safetensors, now under the PyTorch Foundation (HuggingFace, Safetensors joins PyTorch Foundation), solves both. The file is just a JSON header + raw tensor bytes: loading it cannot do anything except populate tensor buffers. And the key property for us is physical: the data region is aligned to a page boundary (the header is padded so that the first tensor starts at a multiple of the OS page size). That is what makes zero-copy possible: a loader can mmap the file, cudaHostRegister over the mapped region and DMA straight from the page cache into VRAM, with no torch.load deserialisation and no temporary host FP32 buffer (HuggingFace, Safetensors). It is the base format; everything else is built on top of it.

The mmap trap (already flagged in From disk to HBM): mmap reads nothing immediately, it defers the cost to the first access to each page. If you do not force the read, the cold start looks short and the first token pays the page faults. And the “blazingly fast second load” is the page cache lying to you: in production pods are born cold, on nodes where those files are not cached.

Tensorizer: streaming tensor by tensor straight to the GPU

Tensorizer, from CoreWeave, serialises the model’s weights and its tensors into a single file and, instead of loading the whole model into RAM before moving it to the GPU, streams the data tensor by tensor from disk, an HTTP/HTTPS endpoint or an S3 bucket, deserialising on the fly straight onto the GPU (vLLM, Loading Models with CoreWeave’s Tensorizer). The operational advantage: near-instant loading and low host RAM usage during initialisation, which matters especially in serverless and autoscaling scenarios, exactly the scale-to-zero case. And it decouples the weights from the container image: the model lives in object storage, not inflating the image pull.

In vLLM it is enabled with --load-format tensorizer (or load_format="tensorizer" via the API). It requires serialising the model once into Tensorizer’s format; from then on, any pod streams it.

Run:ai Model Streamer: reading and copying at the same time

The Run:ai Model Streamer (from NVIDIA) attacks the bottleneck from another angle: it reads the tensors concurrently, N OS threads reading from storage into the CPU buffer, while streaming them to VRAM, so that the storage read and the H2D copy overlap instead of happening in series (vLLM, Loading models with Run:ai Model Streamer; NVIDIA, Reducing Cold Start Latency). It reads safetensors directly, with no reconversion.

The numbers published by NVIDIA give the order of magnitude: the streamer reaches 4.88 s reading from S3 at concurrency 32 and 7.53 s from an SSD IO2 at concurrency 8; integrated into vLLM, the total time to ready drops to 23.18 s from S3, 28.28 s from SSD IO2 and 35.08 s from GP3 (NVIDIA, Reducing Cold Start Latency). Note the gap between the weight loading time (~5–8 s) and the total to ready (~23–35 s): those ~18–27 s of difference are the other four line items, above all item 5. The streamer fixed item 4 and the rest is still there, exactly what this post warns about.

Concurrency is the tuning parameter: it controls how many OS threads read tensors into the CPU buffer (and, for S3, how many client connections the host opens) (vLLM, Run:ai Model Streamer). 16 is usually enough for local NVMe; 32 for high-throughput object storage. One thread does not saturate a Gen5 NVMe; overlapping does.

# default safetensors (the slowest of the fast ones)
vllm serve <model> --load-format safetensors

# Run:ai Model Streamer (concurrent reads, overlaps read + H2D)
vllm serve <model> --load-format runai_streamer \
 --model-loader-extra-config '{"concurrency": 32}'

# Tensorizer (tensor-by-tensor stream straight to GPU from object storage/S3)
vllm serve <model> --load-format tensorizer

The maths of the weights: $t = W/B$ and the saving from overlapping

The floor of item 4 is simple: moving $W$ gigabytes at a bandwidth $B$ takes

$$t = \frac{W}{B}$$

Take a large model, a 70B. In BF16 ($b=2$ bytes/param) that is $W = 70 \times 10^9 \cdot 2 = 140$ GB; in FP8 ($b=1$), $W = 70$ GB. Bandwidth depends on the tier (see the table below): local Gen5 NVMe reads on the order of ~14 GB/s per disk; PCIe Gen5 x16 copies host→GPU at ~50 GB/s; network storage, ~1–3 GB/s. The theoretical floor for reading 140 GB from an NVMe is $140/14 = 10$ s; from the network at 2 GB/s, 70 s. The bottleneck rules.

The overlapping trick. The default loader does the two steps, reading from disk and copying H2D, in series: first it fills a buffer, then it copies, then the next one. The time is the sum:

$$t_{\text{series}} = t_{\text{read}} + t_{\text{H2D}}$$

Tensorizer and the Run:ai Model Streamer overlap them: while one thread copies a chunk to VRAM, another is already reading the next one from disk. With enough concurrency, the total time tends towards the maximum of the two, not the sum:

$$t_{\text{overlapped}} \approx \max(t_{\text{read}},\, t_{\text{H2D}})$$

With the 70 GB of the 70B in FP8, from NVMe at 14 GB/s and PCIe at 50 GB/s: $t_{\text{read}} = 70/14 = 5.0$ s, $t_{\text{H2D}} = 70/50 = 1.4$ s. In series, $5.0 + 1.4 = 6.4$ s; overlapped, $\max(5.0,\,1.4) = 5.0$ s. The saving here is modest (~22%) because the disk dominates comfortably. Overlapping shines when the two steps are comparable: from network storage at 5 GB/s, $t_{\text{read}} = 70/5 = 14$ s and $t_{\text{H2D}} = 1.4$ s, series 15.4 s vs overlapped 14 s; but with many concurrent streams saturating a fast disk and a matched H2D, going from the sum to the maximum can almost double the effective throughput. The flip side: overlapping only gives you what the physical bottleneck allows. If the disk only delivers 14 GB/s, no streamer brings you below 5 s for 70 GB, and that is what the other lever is for, moving fewer bytes (FP8 versus BF16 halves $W$).

The storage tier

Tier	Typical bandwidth	140 GB (70B BF16)	70 GB (70B FP8)	Notes
Local Gen5 NVMe	~14 GB/s per disk	~10 s	~5 s	the short path; one disk
Local Gen5 NVMe (several, RAID0)	~28–50 GB/s	~3–5 s	~1.5–2.5 s	if the loader saturates several streams
Object storage (S3/RGW)	~1–3 GB/s per stream	~47–140 s	~23–70 s	streamer concurrency helps a great deal
NFS / shared network	~1–2 GB/s	~70–140 s	~35–70 s	puts the network and its contention in the path

The operational conclusion is the same as in From disk to HBM: the weights that serve the cold start live on the node’s local NVMe, not on the network. Network storage is the repository; the inference node has a hot local copy (pre-pull with an initContainer or a per-node cache DaemonSet). Tensorizer is the interesting exception: it streams from object storage so efficiently that it sometimes makes serving from S3/RGW viable with no local copy, at the price of depending on the bandwidth of the storage network.

Speeding up item 5: everything that is not weights

Here is the half of the cold start that fast loaders do not touch. Three fronts.

Caching the torch.compile/Inductor compilation

torch.compile has a built-in cache system: the compiled artefacts are saved after the first start-up and reused across start-ups, even across machines if it is configured properly. In vLLM, the result of the Dynamo compilation is stored in ~/.cache/vllm/torch_compile_cache/ (vLLM Blog, Introduction to torch.compile; vLLM, torch.compile integration). The consequence for the runbook: if that directory is a persistent volume shared between pods (a PVC, or a hostPath on the node’s NVMe populated by the first start-up), subsequent pods skip the compilation and start faster. The first pod pays for the compilation; the rest inherit the cache.

It is exactly “sharpening the knives the night before”: the expensive preparation is done once and reused. The nuance: the cache is sensitive to the vLLM/PyTorch version, the GPU, the quantisation and the configuration, and changing any of them invalidates the cache and you pay for the compilation once again.

Bounding the CUDA graph capture

vLLM captures CUDA graphs for a wide range of batch sizes by default, and that is expensive: the default capture is around 54 s, and on large configurations up to 294 s has been measured. By limiting the capture to the batch sizes your workload really uses (for example 1, 2, 4, 8, 16, 24, 32, 64) the cold start is cut by more than 70%, from 294 s to ~82 s in the measured case (Red Hat, vLLM with torch.compile). The knob is the engine’s compilation/CUDA graphs configuration; the principle is not to capture graphs for batch sizes you will never see.

A real trade-off: fewer captured batch sizes = faster start-up but less coverage. A batch outside the captured set falls back to the eager path (slower per request). You bound to the frequent sizes of your traffic, not of any traffic. And there is always the extreme of disabling CUDA graphs (enforce_eager): near-instant start-up on item 5 at the cost of throughput in the hot state, valid for debugging or for very low-traffic services where cold start weighs more than steady state.

Warm pools: not paying for item 5 on the critical path

The definitive shortcut for all five items is not to run them while the user waits. A warm pool, a replica already started, with weights loaded, graphs captured and kernels compiled, waiting for traffic, turns the cold start into zero for that request. It is GPU cost (partly or wholly idle) in exchange for start-up latency; we analyse it in the economics section below.

Sleep mode: the shortcut that skips all five items

And there is a shortcut halfway between “start from scratch” and “keep a whole replica warm”: vLLM’s sleep mode, the subject of Serving several models on a single GPU. Instead of killing the process, it puts it to sleep: it parks the weights in host RAM (level 1) or discards them (level 2), but keeps the process alive, and with it the CUDA context, the allocator, the CUDA graphs and the already compiled JIT kernels. The wake pays for neither item 1, 2, 3 nor 5; it only puts the weights back into VRAM (item 4, and from RAM, not from disk). That is why a wake is 18–200× faster than a full cold start, and even level 2, which reloads the weights from the same disk, is still 23–45× faster, because it skips the other four items (vLLM Blog, Sleep Mode). Sleep mode is living proof of this post’s thesis: if preserving items 1, 2, 3 and 5 gives you 18–200×, then moving bytes was never the whole cost.

An example cold start, item by item

Let us put numbers on a cold start of 70B in FP8 (70 GB) from local NVMe and see which optimisation attacks each item:

#	Line item	Default	Optimisation	Optimised
1	Process init	~4 s	(little room; minimal environment)	~3 s
2	CUDA context	~6 s	(driver; fixed)	~6 s
3	Allocator	~4 s	(fixed)	~4 s
4	Weight loading	~25 s	Run:ai streamer / Tensorizer (concurrency + overlap)	~5 s
5	CUDA graphs + JIT	~54 s	torch.compile cache + bounded capture	~10 s
	Total	~93 s		~28 s

Two readings. First: optimising only item 4 (a fast loader and nothing else) brings the total down from 93 to ~73 s, a real but disappointing improvement, because item 5 is still intact. Optimising only item 5 (cache + bounded capture) brings it down to ~49 s. Attacking both brings it down to ~28 s, and that is what turns theoretical scale-to-zero into usable scale-to-zero. Second: items 2 and 3 are practically irreducible, they depend on the driver and the hardware; they constitute the floor of the cold start, on the order of 10 s, which no software optimisation crosses. Below that floor, the only answer is not to start from scratch: sleep mode (wake from sub-second to a few seconds) or a warm replica (zero).

The economics of scale-to-zero

The capacity question all of this serves is: switch off or keep warm? Scale-to-zero saves GPU while there is no traffic, but pays the cold start when it comes back. The decision is a balance between the cost of an idle GPU and the latency SLA.

The basic criterion: scale-to-zero is only viable if the cold start fits inside the start-up SLA your users tolerate, or if you have a pre-warming mechanism that starts the replica before the request arrives (load prediction, an early signal from the queue). Without pre-warm, a 90 s cold start means the first user after an idle period waits 90 s, unacceptable for almost any interactive SLO. With the cold start brought down to ~28 s it is still a lot for an interactive request, but it is already tolerable for batch workloads or for a pre-warm triggered by autoscaling with KEDA on queue depth.

The decision rule, in one sentence: keep a warm replica when the cost of the idle GPU during the troughs is lower than the cost of missing the SLA on every start-up; do scale-to-zero (with pre-warm) when the troughs are long and deep and the optimised cold start fits in your tolerance. For a 24×7 interactive service with irregular but continuous traffic, a floor of warm replicas almost always wins. For a large model invoked a few times a day (a 70B for hard tasks), scale-to-zero with an optimised cold start, or sleep mode if it shares a GPU with another model, is the sensible option. It is the same capacity planning boundary: cold start is a parameter of the replica cushion, not a start-up detail.

Applied to the generic 4×H100 cluster

Let us bring the runbook down to the 4 H100 SXM 80 GB with NVLink. The concrete decisions:

Loader and format. Weights in safetensors on disk (never pickle). For the short path from local NVMe, the Run:ai Model Streamer (--load-format runai_streamer, concurrency 16–32) is the default option: concurrent reads, overlapping read and H2D, and you only change one flag. Keep Tensorizer for the case where you serve from object storage (RGW/S3) and want to decouple the weights from the image with no local copy, since its tensor-by-tensor streaming straight to the GPU is what makes that pattern viable. In both cases, FP8 versus BF16 halves item 4 almost for free (measure the quality, do not assume it).

Storage tier. Model repository on the network (Ceph RGW), hot copy on the node’s local NVMe populated by pre-pull. The cold start that counts is the cold one, on a node where the files are not in page cache; serving that start-up from the network puts its latency and contention into the critical path.

Caching item 5. The torch_compile_cache directory on a persistent volume or a hostPath NVMe shared between pods of the same model: the first pod compiles, the rest inherit. And bound the CUDA graph capture to the real batch sizes of each service’s traffic.

When scale-to-zero with pre-warm vs a warm replica. The agent LLM (main service, continuous traffic) never does scale-to-zero: a floor of warm replicas on one or two GPUs. The occasional 70B does: scale-to-zero with an optimised cold start, or, if it shares a GPU with a mid-sized model, sleep mode so the wake is sub-second instead of a 28 s start-up (exactly the play in Serving several models on a single GPU). The optimised cold start in this post is what makes swap and failover between replicas tolerable: a replica that falls over and is replaced in 28 s degrades less than one that takes 90 s. And it connects directly with multi-tenancy: a short cold start lets you share GPUs between teams with per-tenant scale-to-zero without the first user of each tenant paying for an endless start-up.

The cross-cutting principle: cold start is the ceiling on elasticity. The five line items, with both halves attacked, a fast loader for the weights and cache plus bounded capture for the graphs, bring it down from minutes to seconds. And below the irreducible floor of ~10 s (CUDA context + allocator), the only way out is not to start from scratch: sleep mode or a warm replica.

Traps and things that are not what they seem

“I switched to a fast loader and start-up barely improved.” Your bottleneck probably was not item 4. If CUDA graph capture eats 54 s, bringing weight loading down from 25 to 5 s takes 20 s off 93, which is barely noticeable. Measure where the time goes before optimising: vLLM’s start-up logs break down weight loading versus graph capture.

“The second time it started in a flash.” The page cache (for the weights) and the torch.compile cache (for the graphs) lying to you at the same time. The start-up that matters is the cold one, on a clean node. Benchmarking the second start-up measures a situation that almost never occurs in the peak that triggers the autoscaler.

“Tensorizer/streamer will give me the 18–200× of sleep mode.” No. Those loaders speed up only item 4. The 18–200× of sleep mode comes from preserving the other four by keeping the process alive. They are tools for different problems: a fast loader is for a real cold start (a new process); sleep mode is for alternating models without killing the process.

"enforce_eager fixes the cold start." It fixes item 5 (no graph capture, near-instant start-up) but degrades throughput in the hot state, since CUDA graphs exist because they speed up the steady state. It is a trade: it wins for very low traffic services where start-up weighs more than the hot path; it loses for a service with sustained load.

“I bound the CUDA graphs to a single batch size and that is it.” You bound to the sizes your traffic uses, not to one. A batch outside the captured set falls back to the eager path and runs slower. Too aggressive and you penalise the steady state to gain a few seconds of start-up.

“Moving the weights to FP8 also speeds up item 5.” Not directly. FP8 halves the bytes (item 4) and doubles inference throughput, but graph capture and kernel compilation do not depend on the size of the weights, they depend on the shape of the graph and the batch sizes. Item 5 is attacked with caching and bounded capture, not with quantisation.

Conclusion

From disk to HBM said it conceptually: moving the weights is one of the five line items of cold start. This runbook turns it into action. For the weights, item 4, there is a clear menu: safetensors as the base (mmap, zero-copy, no pickle), Run:ai Model Streamer for the short path from NVMe (concurrent reads overlapping read and H2D), Tensorizer for streaming from object storage with no local copy, and the cross-cutting lever of moving fewer bytes with FP8. But the forgotten half of the start-up is item 5, CUDA graph capture and JIT compilation, which on a modern server can be as big as the weight loading: you attack it by caching the compilation across pods and bounding the graph capture to the real batch sizes. Attacking only one half leaves the cold start half done; attacking both brings it from ~90 s to ~28 s. And below the irreducible floor of ~10 s imposed by the CUDA context and the allocator, the only way out is not to start from scratch: sleep mode (sub-second wake) or a warm replica (zero). The kitchen opens on time when somebody took care of the produce and of sharpening the knives the night before. Optimising only the delivery from the store leaves the ovens cold and the restaurant shut.

References

HuggingFace, Safetensors (format, zero-copy, security versus pickle): https://huggingface.co/docs/safetensors/index
HuggingFace, Safetensors is Joining the PyTorch Foundation: https://huggingface.co/blog/safetensors-joins-pytorch-foundation
vLLM, Loading Models with CoreWeave’s Tensorizer: https://docs.vllm.ai/en/stable/models/extensions/tensorizer/
vLLM, Loading models with Run:ai Model Streamer: https://docs.vllm.ai/en/stable/models/extensions/runai_model_streamer/
NVIDIA, Reducing Cold Start Latency for LLM Inference with NVIDIA Run:ai Model Streamer (S3/SSD benchmarks, concurrency): https://developer.nvidia.com/blog/reducing-cold-start-latency-for-llm-inference-with-nvidia-runai-model-streamer/
vLLM Blog, Introduction to torch.compile and How It Works with vLLM (compilation cache): https://blog.vllm.ai/2025/08/20/torch-compile.html
vLLM, torch.compile integration (cache in ~/.cache/vllm/torch_compile_cache/): https://docs.vllm.ai/en/latest/design/torch_compile/
Red Hat Developer, vLLM with torch.compile: Efficient LLM inference on PyTorch (CUDA graph capture ~54 s, bounding batch sizes −70%): https://developers.redhat.com/articles/2025/09/03/vllm-torchcompile-efficient-llm-inference-pytorch
vLLM Blog, Zero-Reload Model Switching with vLLM Sleep Mode (wake 18–200×): https://blog.vllm.ai/2025/10/26/sleep-mode.html

FinOps and multi-tenancy on the GPU cluster: who pays for what

Thu, 11 Jun 2026 12:20:00 +0000

Third instalment in an operational series on how to squeeze a generic on-premise LLM inference cluster of 4×H100 SXM 80 GB. The sibling pieces are Multimodal VLM on-premise with vLLM, serving vision-language models on the same cluster, and Speeding up cold start with Tensorizer, cutting the model load time from disk to HBM. This post closes the economic question the other two leave open: once the cluster serves several models for several teams, who pays for what?

TL;DR

An on-premise GPU cluster does not have the property that makes FinOps easy in the cloud: no monthly invoice arrives telling you what each compute hour cost. The cost has to be manufactured from capex and energy. The right unit is the token, and it is derived in two steps. First the €/GPU-hour: capex depreciation spread over the hours of useful life, plus energy (power × PUE × price_kWh). For an H100 SXM with an assumed capex of 30,000 € and 4 years of life at 90 % availability, that gives ≈ 1.07 €/GPU-hour (0.95 of depreciation + 0.12 of energy at 0.12 €/kWh and PUE 1.4). Second, the €/1M tokens: divide that €/GPU-hour by the sustained throughput. At 2,000 useful tok/s, ≈ 0.148 €/1M tokens; but that number only holds if the GPU is at 100 % useful utilisation. At 20 % utilisation the same token costs 4× more (0.74 €/1M), because capex and energy keep being paid even if the GPU is idle. That is the central thesis: the FinOps goal is not to negotiate the price of the token down, there is no provider to negotiate with, but to raise useful utilisation, which is the only term that moves cost by an order of magnitude. Attribution (chargeback / showback) is made real with LiteLLM virtual keys: per-team and per-user budget, RPM/TPM rate limit, automatic spend tracking and tags to assign spend to cost centres. Isolation between tenants is achieved with MIG (hard GPU partitioning), namespaces, ResourceQuota and PriorityClass. The FinOps Foundation rule worth carving in stone: showback always, chargeback only if the organisation’s accounting policy supports it; neither of the two is “more mature”. And the governance metric of the shared cluster is not the €/token but DCGM_FI_DEV_GPU_UTIL.

The problem the cloud hides from you and on-premise forces you to solve

In the cloud, the cost of a GPU is an observable fact: AWS, GCP or Azure charge you per GPU-hour and the invoice arrives itemised. FinOps in that world consists of reading an invoice that already exists and attributing it. On-premise there is no such invoice. The GPU was bought once, eighteen months ago, on a purchase order nobody remembers; the electricity bill arrives for the whole building with no breakdown of what fraction was the cluster’s; and the datacenter PUE is a number the facilities team knows but the AI team never asked about.

The first job of on-premise FinOps, then, is not to attribute a cost but to manufacture one. You have to build, from explicit assumptions, the equivalent of the “price per GPU-hour” that in the cloud is given to you. And once manufactured, you have to understand that this number is not a property of the hardware, like VRAM or TDP, but a function of how the hardware is used. An idle H100 costs exactly the same as an H100 at 100 %: the capex is already paid and idle energy is not zero. The only thing that changes is how many useful tokens that fixed cost produced. That is where the whole thesis of this post comes from.

The analogy: the coworking space with metered services

A coworking space manages a finite physical space and rents it to teams. The cost model has exactly the structure of the GPU cluster, and the analogy holds down to the last detail.

The desk-hour is the spread capex. The manager paid the rent on the premises, the furniture and the refurbishment, a one-off outlay or a fixed monthly cost, and spreads it over the available desk-hours in the month. Each desk has a base cost that does not depend on whether anybody uses it: the walls, the table and the chair cost the same empty as occupied. This is the depreciation of the GPU: capex divided by the hours of useful life.

The metered kWh is the energy. On top of the desk-hour, the coworking space measures the actual electricity consumption, the plugged-in laptop, the screen, the air conditioning in that room, and bills it separately. This is the energy of the GPU: power × hours × price_kWh, multiplied by the datacenter’s PUE, which is the coworking space also charging you for the fraction of the building’s air conditioning that cools your room.

The empty desk is paid for by whoever booked it. Here is the heart of the matter. If a team books ten desks for the whole month and only uses two, it pays for all ten anyway. The empty desk still takes up space the manager cannot resell, still depreciates furniture, still counts as committed capacity. In the cluster: a GPU assigned to a tenant that uses it at 20 % costs the same as if it used it at 100 %, and the idle 80 % is wasted capex that somebody is paying for.

The manager optimises occupancy, not tariff. A competent coworking manager does not obsess over lowering the price of the desk-hour, it is a sunk cost, the premises are already paid for. They obsess over the occupancy rate: every empty desk is margin lost forever, because an unsold desk-hour cannot be stored. The FinOps equivalent: the goal is not to lower the €/token, it is to raise utilisation, because the €/token falls only as a consequence of utilisation rising. There is no provider to squeeze; the only margin lies in not leaving GPUs idle.

The analogy has an honest limit worth naming: in a coworking space, two people cannot physically share the same chair. On a GPU you can, with time-slicing, MPS or MIG (see Sharing a GPU), and that is precisely the technical lever that lets you raise occupancy beyond what the physical coworking space would allow. But the economics of fixed cost + metered cost is identical.

The token as the unit of cost

Why the token and not the GPU-hour, or the request, or the model? Because the token is the only unit comparable across heterogeneous workloads. A GPU-hour of Llama 8B and one of Llama 70B produce radically different amounts of “useful work”; a RAG request with a 50-token answer and one generating a 3,000-token report are not comparable. The token, specifically the pair (input_tokens, output_tokens), normalises all of that. It is the unit LiteLLM accounts for natively, the one that appears in the OTel attributes gen_ai.usage.input_tokens and gen_ai.usage.output_tokens (see LLM tracing with OpenTelemetry GenAI), and the one commercial providers already use to charge. Adopting the token as the internal currency makes on-premise chargeback directly comparable with the cloud alternative, which is exactly the comparison management wants to see.

The two derived metrics that matter:

Tokens/€ — how many useful tokens each euro of total cluster cost buys. It rises when utilisation rises. It is the metric of economic efficiency.
Tokens/W — how many tokens each watt consumed produces. It is the metric of energy efficiency and, in a datacenter with limited power (the usual case on-premise), it is often the real hard constraint: you cannot add more GPUs because there are no more kW in the rack, so every watt has to perform.

Cost model of a GPU: from capex to €/token

The hourly cost of a GPU has two addends. Let us put all the assumptions in writing, because, as in capacity planning, a cost calculation without written assumptions is a disposable calculation.

Depreciation (spread capex)

$$\text{depreciation cost/h} = \frac{\text{capex}_{\text{GPU}}}{\text{hours of useful life}}$$

where the hours of useful life are the depreciation years × 8,760 h/year × the effective availability (the GPU is not available 100 % of the time: there is maintenance, restarts, update windows). Explicit, generic assumptions (they are not figures from any provider or any real purchase):

Capex per GPU: 30,000 € — a reasonable assumption that includes not just the GPU but its pro-rata share of the server, power supply, cooling, NVLink/InfiniBand network and rack. A “bare” H100 SXM costs less, but honest FinOps spreads the cost of the whole node across its GPUs.
Accounting useful life: 4 years. That is aggressive-realistic for a datacenter GPU; some depreciate over 3, others over 5.
Effective availability: 90 %.

$$\text{hours of life} = 4 \times 8{,}760 \times 0.90 = 31{,}536 \text{ h}$$ $$\text{depreciation cost/h} = \frac{30{,}000}{31{,}536} \approx 0.95 \text{ €/GPU-hour}$$

Energy (power × PUE × price_kWh)

$$\text{energy cost/h} = P_{\text{GPU}} \cdot \text{PUE} \cdot \text{price kWh}$$

Assumptions:

GPU power: the TDP of the H100 SXM is 700 W according to the NVIDIA datasheet (H100 datasheet). Under sustained inference load it hovers around that value; we use 0.7 kW as the average power under load. (At idle it drops a lot, 60–100 W, but the FinOps of reserved capacity reasons about power under load, which is what limits the rack.)
PUE (Power Usage Effectiveness): how many watts enter the datacenter for every watt that reaches the chip. A modest, well-managed on-premise datacenter sits at 1.4; a bad one at 2.0; the hyperscalers boast of 1.1. We use 1.4.
Price per kWh: 0.12 €/kWh — a generic assumption for an industrial tariff; it varies enormously by country and contract.

$$\text{energy cost/h} = 0.7 \cdot 1.4 \cdot 0.12 \approx 0.1176 \approx 0.12 \text{ €/GPU-hour}$$

The total €/GPU-hour

$$\boxed{\text{€/GPU-hour} = 0.95 + 0.12 \approx 1.07 \text{ €/GPU-hour}}$$

Two important readings of this number. First: depreciation dominates (89 % of the cost); energy is 11 %. This inverts many people’s intuition, who believe “the expensive part is the electricity”. With these assumptions, the expensive part is having bought the GPU, and that is why leaving it idle hurts so much. Second: the number is per-GPU; for the 4×H100 node it is ≈ 4.28 €/hour, and for the complete generic cluster of 4 nodes (16 GPUs), ≈ 17.1 €/hour ≈ 150,000 €/year of fixed cost that is paid whether it is used or not.

From €/GPU-hour to €/token

Now we divide the hourly cost by the tokens the GPU produces in an hour. Here sustained throughput comes in. Let us suppose, an order-of-magnitude figure, checkable with vllm bench serve, that a Llama 70B replica on 4×H100 (TP=4) sustains 2,000 useful tokens/s aggregated under good concurrency.

$$\text{tokens/hour} = 2{,}000 \times 3{,}600 = 7.2 \times 10^6 \text{ tok/h}$$

The cost of that replica (4 GPUs) is $4 \times 1.07 = 4.28$ €/hour. Therefore:

$$\text{€/1M tokens} = \frac{4.28}{7.2} \approx 0.59 \text{ €/1M tokens}$$

Per individual GPU, if we take a smaller replica (e.g. Llama 8B FP8 on 1 GPU at ~2,000 tok/s):

$$\text{€/1M tokens} = \frac{1.07 \text{ €/h}}{7.2 \times 10^6 \text{ tok/h}} \times 10^6 \approx 0.149 \text{ €/1M tokens}$$

These numbers are the theoretical floor at 100 % utilisation. Nobody operates at 100 %. And that is the whole story.

Utilisation is the lever (with numbers)

The €/token in the previous section assumes the GPU produces 7.2 M tokens every hour, hour after hour. In practice it produces that only during peak hours. The rest of the time it is partly idle: at night, at weekends, between traffic peaks. The useful utilisation $u$ is the fraction of the theoretical capacity that really turns into billable tokens.

The hourly cost is fixed (1.07 €/GPU-hour is paid come rain or shine), but the tokens produced scale with $u$:

$$\text{€/1M tokens}(u) = \frac{\text{€/GPU-hour}}{\text{max tok/hour} \cdot u} = \frac{1.07}{7.2 \times 10^6 \cdot u} \times 10^6 = \frac{0.149}{u}$$

The requested contrast, worked out:

Useful utilisation $u$	Real tokens/h	€/1M tokens	Multiplier vs 80 %
100 %	7.2 M	0.149 €	0.80×
80 %	5.76 M	0.186 €	1.00× (baseline)
50 %	3.6 M	0.298 €	1.60×
20 %	1.44 M	0.744 €	4.00×
10 %	0.72 M	1.488 €	8.00×

Direct reading: going from 20 % to 80 % utilisation divides the cost per token by four ($0.744 \to 0.186$). No tariff negotiation, no quantization, no hardware change gives you that factor of 4 so cheaply. Quantizing from BF16 to FP8 can double throughput, and therefore halves the €/token, but it degrades quality and demands evals (see Quantization for inference); raising utilisation from 20 to 80 % does not touch the quality of a single token.

This reformulates GPU cluster FinOps in a single sentence: the €/token is not something you negotiate, it is something you earn by filling the GPUs. And raising utilisation is an engineering problem that already has its pieces:

Autoscaling that switches replicas off when traffic drops, so as not to pay for idle GPU (see LLM autoscaling on Kubernetes with KEDA). Switching off a 4-GPU replica for eight hours overnight saves ≈ 34 € a day of real cost; multiplied by replicas and by days, that is the difference between a profitable cluster and one that bleeds.
Sharing a GPU between small workloads with MIG or time-slicing, so that two tenants that individually would use 30 % together fill a GPU to 60 % (see Sharing a GPU).
Overnight batch that fills the off-peak hours with non-urgent work (re-embeddings, evals, light fine-tuning) instead of leaving the GPUs switched off or ticking over.
Reducing cold start so that scaling to zero and starting again is cheap and therefore viable as a utilisation strategy (see the sibling piece Speeding up cold start with Tensorizer).

The diagram: the €/token cascade by utilisation

The cost per token is not a number, it is a cascade that multiplies as utilisation falls:

Attributing the cost: LiteLLM virtual keys

With the €/token manufactured, it remains to attribute it to whoever consumed it. The piece that already sits in front of the engines, the L7 inference router, is also the natural place to count. LiteLLM Proxy makes chargeback real with four mechanisms, all documented:

Virtual keys with a budget. Each team or user receives a virtual key with a max_budget and a budget_duration. You can define several independent budget windows (for example one of 24 h and another of 30 d) that reset on their own cycle (Virtual Keys). When a team exhausts its monthly token budget, the proxy rejects with a budget exceeded error instead of carrying on spending GPU-hours nobody will be able to charge anywhere.

RPM/TPM rate limit. Each key carries requests per minute (RPM) and tokens per minute (TPM) limits (Budgets, Rate Limits). This is not just anti-abuse protection: it is the tool that stops a noisy tenant monopolising the cluster’s utilisation at everybody else’s expense. The rate limit is the capacity quota; the budget is the spend quota. They are orthogonal and you use both.

Automatic spend tracking. LiteLLM accounts for the spend of all known models by key, user and team, recording the API key, the user, the team_id, the request’s tags, the end user, the model group and the token counts (Spend Tracking). For the cluster’s self-hosted models, the cost per token is configured with the €/token we manufactured above, and that is the hook between the cost model and the real accounting.

Tags for cost centres. The tags let you track spend and set budgets per label, categorising costs by project, department or cost centre (Setting Tag Budgets). A tag is attached when the key is created and every request made with that key inherits the tag automatically; the proxy applies the label’s budget. That way chargeback maps cleanly onto the organisation’s accounting hierarchy without every team having to remember to tag by hand.

A declarative catalogue fragment, which fits into the litellm-config of the router post, materialising the manufactured cost and two tenants:

litellm_settings:
 # €/token manufactured in this post for the self-hosted model.
 # input/output in €/token (not per 1M); 0.186 €/1M = 1.86e-7 €/token at u=80%.
 model_cost_map:
 llama-70b-onprem:
 input_cost_per_token: 0.000000186
 output_cost_per_token: 0.000000186

# Virtual keys per team (via the /key/generate API or config):
# equipo-datos: max_budget=500 €/month, tpm_limit=200000, tags=["cc-datos"]
# equipo-soporte: max_budget=150 €/month, tpm_limit=60000, tags=["cc-soporte"]
# The proxy counts tokens, multiplies by model_cost_map, deducts from the budget,
# and attributes the spend to the tag → showback per cost centre, without touching the engine.

The complete attribution flow, from the request to the per-team report:

Showback vs chargeback: the distinction the FinOps Foundation insists on

The two words are constantly confused and the difference is one of accounting formality, not technology:

Showback — giving each team visibility of what it consumed and its cost, without really billing it to their budget. The report arrives, the team sees it, but the money does not move between cost centres.
Chargeback — really transferring the cost to the team’s or product’s budget, putting it in their P&L. The spend stops being a central IT cost and becomes an allocated cost.

The FinOps Foundation is explicit on two points worth carving in stone (Invoicing & Chargeback, Chargeback vs Showback). First: showback is a requirement of any FinOps practice; chargeback depends on the organisation’s accounting policy. Not all organisations can or want to move money between departments for GPU consumption; showback you can always do. Second, and counterintuitive: neither of the two is “more mature” than the other. The narrative that chargeback is the “grown-up” version of showback is false according to the framework itself. The practical recommendation: start with showback, giving visibility, then build cost allocation aligned to the organisational hierarchy, and only then, if the accounting policy supports it, switch chargeback on.

For the on-premise GPU cluster this means: the LiteLLM machinery (keys, tags, spend tracking) always produces the showback. Turning it into chargeback is a decision for finance, not for the platform team. The platform team guarantees that the numbers are correct and reproducible; who really pays is a governance decision.

Isolation: making the cost genuinely attributable

Chargeback is only honest if consumption is isolable. If two tenants share a GPU without partitioning and one saturates the HBM, the other suffers degradation it did not cause but which contaminates its cost attribution. Isolation has two planes.

Hard GPU isolation: MIG. Multi-Instance GPU physically partitions an H100 into up to 7 instances with dedicated memory and SMs; in a 7-way partition each instance has ~10 GB of HBM3 and its own SMs. MIG gives the strongest isolation: tenant A in its MIG instance cannot touch tenant B’s performance, and attribution is trivial because each instance is accountable separately (see Sharing a GPU for the detail of MIG vs time-slicing vs MPS). The cost: MIG instances are fixed, they are not resized hot, and if they sit empty they are fragmented and wasted capex, the coworking desk subdivided into booths nobody rents.

Logical Kubernetes isolation. On top of the cluster:

Namespaces per tenant — the boundary for RBAC, NetworkPolicy and quotas.
ResourceQuota — limits how many nvidia.com/gpu (or nvidia.com/mig-1g.10gb) a namespace can request. It is the GPU capacity quota at the scheduler level: the tenant cannot claim more GPUs than its quota grants, which bounds its maximum cost by construction.
PriorityClass — defines which workloads can evict others under pressure. The overnight re-embeddings batch runs with low priority and gives way to interactive inference; that way it fills the utilisation valleys without risking the paying tenant’s SLO. It is the piece that stops “raising utilisation with batch” from cannibalising quality of service.

The combination that works in the generic cluster: MIG to partition the GPUs between tenants that need hard isolation and clean attribution, ResourceQuota per namespace to bound each tenant’s cost, and PriorityClass so that filler work raises utilisation without touching the priority tenants.

Applied to the generic 4×H100 cluster

Let us bring it all down to the series’ cluster: 4 nodes × 4×H100 SXM 80 GB, 16 GPUs, ≈ 17.1 €/hour of fixed cost (≈ 150,000 €/year). Three teams share it: Data (production RAG, office-hours traffic), Support (ticket assistant, daytime peaks) and Platform (evals and re-embeddings batch, no urgency).

Splitting the cost. The cluster’s fixed cost (150,000 €/year) is split through LiteLLM’s spend tracking in proportion to the tokens consumed by each team, valued at the manufactured €/token. If in one month Data consumed 1,700 M tokens, Support 480 M and Platform 320 M, at 0.186 €/1M the showback is ≈ 316 € / 89 € / 60 €. The residue, the fixed cost of the GPUs nobody used because average utilisation was, say, 45 %, is the cost of idleness, and the governance decision is whether it is split between the tenants (penalises everybody equally) or charged to Platform as “cost of unsold capacity” (gives Platform an incentive to raise utilisation). The FinOps Foundation would say: make it visible first (showback of the cost of idleness), decide the split afterwards.

Quotas per tenant. Each team has its namespace with ResourceQuota: Data can claim up to 8 GPUs (2 replicas at TP=4), Support up to 4, Platform up to 4 but with a low PriorityClass, giving up its GPUs when Data or Support need them at peak. In LiteLLM, each team has its virtual key with a monthly max_budget and a tpm_limit that reflects its capacity quota.

MIG to isolate and attribute. For the small models (embeddings, reranker, a utility Llama 8B), an H100 split 7-way with MIG gives seven separately attributable instances: three for Data’s embeddings, two for Support’s reranker, two free for filler. Each instance is an independent accounting “desk”; the per-tenant showback comes straight from the scheduler.

The governance metric is utilisation, not the €/token. Here we close the circle. The KPI the cluster owner should look at daily is not “what the token costs”, that number only falls as a consequence, but DCGM_FI_DEV_GPU_UTIL, the DCGM exporter metric that indicates what fraction of the time the GPU is not idle (DCGM exporter). And with a critical nuance that already appeared in capacity planning: DCGM_FI_DEV_GPU_UTIL measures “not idle”, it does not measure useful work. A GPU can show 100 % GPU-util with the HBM saturated and low SM occupancy, producing few tokens. That is why serious FinOps crosses three signals: DCGM_FI_DEV_GPU_UTIL (is the GPU busy?), DCGM_FI_PROF_SM_OCCUPANCY and DCGM_FI_PROF_GR_ENGINE_ACTIVE (is it doing real work?), and vllm:gpu_cache_usage_perc (is the HBM well used?). The economic governance metric is useful tokens per GPU-hour, and it is watched from the GPU observability with DCGM dashboard.

An end-to-end assembly detail, LibreChat on top of LiteLLM with RAG, which closes the flow from tenant to cost, is covered in the (forthcoming) post on the end-to-end sovereign assistant; here it is enough to know that the metering point is always the proxy, never the engine.

Four traps of on-premise GPU FinOps

Trap 1 — comparing the on-premise €/token with the commercial API’s list price. The commercial API has margins, SLA and scale the on-premise cluster does not, but on-premise has the cost of idleness the API hides from you (they fill their GPUs with thousands of customers). The honest comparison is on-premise at its real utilisation vs the API at its real negotiated price, not the fantasy of on-premise at 100 %.

Trap 2 — forgetting the cost of idleness in the showback. If you only charge teams for the tokens they consumed, the cost of the idle GPUs disappears from the report and nobody sees it. That cost exists and somebody is paying it. Making it visible is the first step to reducing it.

Trap 3 — confusing high GPU-util with efficiency. A GPU at 100 % DCGM_FI_DEV_GPU_UTIL with the HBM saturated and low SM occupancy spends capex without producing proportional tokens. The goal is not “GPU at 100 %”, it is “useful tokens per euro”. Always cross util with SM occupancy and real throughput.

Trap 4 — chargeback before showback. Switching chargeback on, moving money, before the teams trust that the numbers are correct generates disputes that burn the whole FinOps programme. First visibility, then trust in the data, then, if the accounting policy allows it, the money.

References

NVIDIA — H100 Tensor Core GPU Datasheet (TDP 700 W, HBM3 3,35 TB/s): resources.nvidia.com/en-us-gpu-resources/h100-datasheet-24306.
LiteLLM — Virtual Keys (docs.litellm.ai/docs/proxy/virtual_keys), Budgets & Rate Limits (/docs/proxy/users), Spend Tracking (/docs/proxy/cost_tracking), Setting Tag Budgets (/docs/proxy/tag_budgets), Team Budgets (/docs/proxy/team_budgets).
FinOps Foundation — Invoicing & Chargeback Capability (finops.org/framework/capabilities/invoicing-chargeback/) y Data Analysis and Showback (finops.org/framework/previous-capabilities/analysis-showback/).
CloudZero — Chargeback vs. Showback: Cloud Cost Allocation Models Explained (cloudzero.com/blog/chargeback-vs-showback/).
NVIDIA — DCGM Exporter (docs.nvidia.com/datacenter/dcgm/latest/gpu-telemetry/dcgm-exporter.html): DCGM_FI_DEV_GPU_UTIL, DCGM_FI_PROF_SM_OCCUPANCY, DCGM_FI_PROF_GR_ENGINE_ACTIVE.

Multimodal on-premise: serving a VLM with vLLM (vision + language)

Thu, 11 Jun 2026 11:40:00 +0000

Third batch of an operational series on squeezing a generic on-premise LLM cluster of 4×H100 SXM. Its sibling pieces in this batch are FinOps and GPU multi-tenancy with LiteLLM, which puts a price on every token, and a visual token costs the same as a text one, and Speeding up cold start with Tensorizer, which matters twice as much here because a VLM loads two models (the vision encoder and the LLM). If the VLM ends up feeding a document assistant, that end-to-end setup with LibreChat and RAG is another story (in preparation).

TL;DR

A vision-language model (VLM) is not magic: it is a normal LLM with a sensory organ bolted on the front. Three pieces in series: a vision encoder (a ViT) that looks at the image, a projector or connector (typically an MLP) that translates the encoder’s output into the LLM’s embedding space, and the usual LLM, which receives those embeddings as if they were just more tokens. The consequence that governs the entire operational design is brutal in its simplicity: an image becomes visual tokens, and those tokens cost exactly the same as text tokens. A high-resolution document page is not “a couple of image tokens”: it is hundreds or thousands of tokens entering the context, inflating the prefill (compute-bound) and occupying KV cache (memory-bound) just as if you had pasted a thousand words. In Qwen2.5-VL the arithmetic is literal: the number of visual tokens is $H\times W / (14\times14\times4)$, so an $896\times896$ image is 1,024 tokens and an A4 page scanned at a decent resolution easily reaches 1,500–2,500 tokens for the image alone. This post opens up the anatomy, works through the token-cost maths and translates it into TTFT and VRAM on an 80 GB H100, explains vLLM’s multimodal support (--limit-mm-per-prompt, pixel budget via mm_processor_kwargs, chat template) and answers the question that really matters: classic OCR or VLM. The short answer: use Tesseract/PaddleOCR for the bulk of documents with clean text, and reserve the VLM, expensive in tokens and in GPU, for the subset that genuinely needs it: complex layout, nested tables, handwriting, stamps, charts. On the generic 4×H100 SXM cluster, with FP8 so it fits.

The analogy: the reader who sees the page versus the one who only hears the transcript

Picture two experts you ask to review a scanned contract. To the first you read the contract aloud, a blind transcript, word by word. It is fast, efficient, and for 90 % of the clauses it is enough: the text is the text. But there are things you cannot read out: that the signature at the end is handwritten and does not match the typed name; that there is a “PAID” stamp struck diagonally across the amount; that the table on page 3 has a merged cell that changes who each row applies to; that there is a pen annotation in the margin. The blind transcript loses all of that or flattens it into gibberish.

To the second expert you hand the whole page, to look at. They see the signature, they see the stamp, they understand the structure of the table because they perceive the lines and the cells, they read the marginal note. They grasp the document as a visual object with layout, not as a river of characters. They are strictly more capable.

Why not always use the second one, then? Because seeing costs more mental bandwidth. The first expert receives the contract as a flow of words: cheap, linear. The second receives it as an image that their brain has to parcel up, attend to region by region, and reconstruct. They process a great deal more information per page, most of which, in a clean text document, is redundant with what the transcript would already have given them. If you only need to read the clauses, hiring the expert-who-sees for every page is overpaying for capability you do not use.

That is the entire thesis of this post. The VLM is the expert who sees; OCR is the blind transcript; and “mental bandwidth” is tokens. The engineering consists of sending the expert-who-sees only the pages where seeing changes the answer, and resolving the rest with the cheap transcript.

Anatomy of a VLM: encoder, projector, LLM

A modern VLM of the Qwen-VL family (Qwen2.5-VL model card; Qwen3-VL Technical Report) is made up of three blocks in series:

Vision encoder (ViT). A Vision Transformer that receives the image, cuts it into patches and produces an embedding per patch (or per group of patches). In Qwen2.5-VL it is a dynamic-resolution ViT: it accepts images of arbitrary size, resizing them to multiples of 28 and splitting them into 14×14 pixel patches, with window attention to speed up the processing of large images. There is no “canonical size” everything is squashed to: a large image produces more patches than a small one.
Projector / connector. The ViT’s embedding space is not the LLM’s. The projector acts as the translator. In Qwen2.5-VL it is elegant and cheap: it groups every 2×2 block of adjacent patches (four ViT tokens), concatenates them and projects them with a two-layer MLP into a single fused token in the LLM’s dimension. That fusion factor of 4 is the reason for the $4$ that appears in the denominator of the token formula below: four ViT patches collapse into one visual token that the LLM sees.
The LLM. The usual language transformer. It receives a sequence of embeddings that is a mixture of text tokens (from your prompt) and visual tokens (from the image), all in the same space, and does what it knows how to do: attention over the complete sequence and autoregressive generation. To the LLM, a visual token and a text token are indistinguishable in cost: both go through the same QKV projections, both occupy an entry in the KV cache, both take part in the $O(C^2)$ attention.

The idea to internalise: the encoder and the projector are the sensory organ, but the inference cost lives in the LLM, and that cost is paid in tokens. The ViT adds a fixed prefill overhead (processing the image once), but the recurring bill, the LLM prefill and the KV cache throughout the whole generation, is set by the number of visual tokens the projector spits out. That is why the control lever is not “which encoder”, but how many pixels you let through.

The maths: what it costs to see a page

Let us put numbers on “an image costs many tokens”, with the real Qwen2.5-VL formula (Qwen team, Qwen2.5-VL blog):

$$N_{\text{tok}} = \frac{H \times W}{14 \times 14 \times 4} = \frac{H \times W}{784}$$

where $H$ and $W$ are height and width in pixels (rounded to the multiple of 28 the ViT imposes). The $14\times14$ is the patch size; the $\times4$ is the projector’s 2×2 fusion. A sanity check against the model card figure: an $896\times896$ image gives

$$N_{\text{tok}} = \frac{896 \times 896}{784} = \frac{802{,}816}{784} = 1{,}024 \text{ tokens.}$$

It checks out. Now an A4 page scanned at a resolution reasonable for reading small text, say $1120 \times 1568$ px (approximate height $\times$ width, already rounded to multiples of 28):

$$N_{\text{tok}} = \frac{1120 \times 1568}{784} = \frac{1{,}756{,}160}{784} \approx 2{,}240 \text{ tokens.}$$

A single page is ~2,240 visual tokens before you write a word of prompt. For context: that is roughly what 1,700 words of Spanish text would take up (at ~1.3 tokens/word). If that A4 page contains 600 words of actual text, the VLM is paying almost 4× in tokens to see it rather than read its transcript. If the page is a dense table or a handwritten document that OCR cannot read, that 4× is justified. If it is a paragraph of clean text, it is pure waste.

Qwen2.5-VL lets you control this with min_pixels / max_pixels: the number of tokens per image is dynamic and ranges from about 4 up to 16,384 by default, adjustable with those parameters (HF Qwen2.5-VL docs). Lowering max_pixels cuts tokens at the expense of resolution, and of the ability to read the fine print. It is exactly the lever of “how much mental bandwidth do I give the expert”.

From token cost to TTFT

Visual tokens are not free in latency. The prefill, processing the whole context before the first token, is compute-bound and its cost grows with the context length $C$ (linear in the projections, quadratic in the attention); this is worked through in detail in vLLM’s attention backend. Every visual token enters that $C$.

Let us model the TTFT as the time to process the prefill tokens at a given prefill throughput. Take a VLM of around 7–8B on an H100 with an illustrative prefill throughput of ~12,000 tok/s (an example figure; measure it, do not assume it, and it depends on how much the ViT forward pass weighs on top). For a request with an 80-token text prompt plus one page at 2,240 visual tokens:

$$\text{TTFT} \approx \frac{80 + 2240}{12{,}000 \text{ tok/s}} \approx \frac{2320}{12{,}000} \approx 0{.}19 \text{ s}$$

plus the fixed cost of the vision encoder’s forward pass over the ~9,000 ViT patches (another compute-bound slice of prefill). Now five pages in the same request (a short PDF):

$$\text{TTFT} \approx \frac{80 + 5 \times 2240}{12{,}000} = \frac{11{,}280}{12{,}000} \approx 0{.}94 \text{ s}$$

The TTFT multiplies almost fivefold, because the visual tokens completely dominate the context. And attention scales as $O(C^2)$: five pages are not “5× more expensive” in the attention component, but ~25× relative to one. The operational lesson: the number of images and their resolution are your TTFT budget, not a detail. Lowering max_pixels from 2,240 to, say, 1,100 tokens/page (lower resolution, enough for large text) cuts the prefill almost in half.

VRAM: the VLM’s weights and what is left for KV

On an 80 GB H100 SXM, let us budget a VLM of the Qwen-VL family. Take an 8B (LLM) plus the vision encoder (a ViT of a few hundred million parameters; call the whole thing ~8.5B effective parameters for weight purposes). In BF16 (2 bytes/parameter):

$$M_{\text{BF16}} \approx 8{.}5 \times 10^9 \times 2 \text{ B} \approx 17 \text{ GB.}$$

In FP8 (1 byte/parameter), the weights halve (Qwen3-VL-8B-Instruct-FP8, with reported quality almost identical to BF16):

$$M_{\text{FP8}} \approx 8{.}5 \times 10^9 \times 1 \text{ B} \approx 8{.}5 \text{ GB.}$$

For an 8B model this fits comfortably in BF16 on an H100; FP8 matters more for leaving VRAM free for KV cache, and here the KV is the problem, because visual tokens inflate it. The detail of end-to-end FP8 (weights and KV) is in FP8 end to end. What is relevant: one page is ~2,240 KV entries; serving several sessions that send long documents consumes KV at a rate a text-only case would never see. With a 32B VLM in FP8 (~32 GB of weights) on an 80 GB H100 you are left with ~45 GB for KV and activations, which with long multimodal contexts run out sooner than “text-only” intuition suggests. The rule: with a VLM, plan the VRAM counting the visual tokens in the KV, not just the weights.

Serving a VLM with vLLM

vLLM supports image multimodal input natively for the Qwen-VL family and many other VLMs (Multimodal Inputs, vLLM docs). The gears that matter:

--limit-mm-per-prompt. Limits how many multimodal items each prompt accepts, per modality. By default the limit is generous (999 per modality), but in production it is worth setting it low, --limit-mm-per-prompt '{"image": 2}', because every accepted image reserves token and KV budget. If you set a limit of 8 images and someone sends 8 A4 pages, that is ~18,000 visual tokens in a single request: it blows up max_model_len or the KV. The limit is a budget firebreak, not a cosmetic restriction. (A curious case: setting it to 0 for a modality lets you pass precomputed embeddings without loading the encoder module, saving VRAM, useful if you do the vision encoding outside vLLM.)

Pixel budget via mm_processor_kwargs. Here the token formula connects with the configuration. When serving Qwen2.5-VL you pass, for example, --mm-processor-kwargs '{"min_pixels": 12544, "max_pixels": 254016}' (vLLM Qwen2.5-VL discussion). With $254{,}016$ maximum pixels per image, the token ceiling per image is $254{,}016 / 784 \approx 324$ tokens, an aggressive trim that keeps the cost in check at the price of not being able to read very small print. This is the dial that really governs your token bill: raise it for dense documents, lower it for images where fine detail does not matter.

The chat template and the image placeholder. The model’s processor (loaded via AutoProcessor) inserts special tokens marking where the image goes in the sequence: in Qwen2.5-VL the sequence <|vision_start|><|image_pad|><|vision_end|>, where <|image_pad|> expands to the exact number of visual tokens the image produced. vLLM applies this template automatically; it is worth knowing there is a documented quirk, the image block tends to be placed before the user’s text in the sequence order, regardless of where you put it in the content (issue #15125, vllm-project/vllm), which rarely causes trouble but is worth bearing in mind if the image/text order is semantically important for your prompt.

Current models. As of June 2026, the reference open family is Qwen3-VL (dense 2B/4B/8B/32B variants and MoE 30B-A3B / 235B-A22B, interleaved contexts of up to 256K tokens combining text, image and video), with published FP8 checkpoints of almost identical quality to BF16 (Qwen3-VL Technical Report, arXiv 2511.21631; Qwen3-VL Usage Guide, vLLM recipes). Qwen2.5-VL remains perfectly serviceable and its token formula is the one we have used because it is public and clean; the cost figures are of the same order in both generations.

Classic OCR vs VLM: when to use which

Here is the decision that pays the bill. Parsing documents to feed them into a document ingestion pipeline has two paths, and choosing the split between them well is the difference between a healthy cluster and a choking one.

Classic OCR (Tesseract, PaddleOCR). Text-recognition engines that run on CPU, are cheap, fast and mature. Tesseract is the portable workhorse (edge, offline kiosks, embedded scanners). PaddleOCR is more capable on layout and Asian scripts, with its PP-Structure module for preserving structure, extracting tables and detecting key-value fields (PaddleOCR vs Tesseract, CodeSOTA 2026). On clean, well-scanned text, classic OCR is hard to beat on cost per page: zero GPU, millisecond latency, very high throughput. Its weakness is the “hard” document: complex multi-column layout, tables with merged cells, handwriting, stamps, charts, poor scan quality. There, pipeline-based OCR either fails or returns gibberish that destroys the structure.

VLM. It takes the image of the document and generates structured markdown or HTML in a single end-to-end pass, understanding the layout because it sees it. It shines exactly where classic OCR struggles: nested tables, reading charts, handwriting, understanding the spatial relationship between elements. The price is the one we have been calculating all through this post: expensive in tokens and in GPU. Worth noting a 2026 nuance: the frontier has moved. VLM models specialised in document parsing have appeared, compact (~1–3B), open source, scoring very high on benchmarks such as OmniDocBench (PaddleOCR-VL in the 94–96 point band; dots.ocr 3B at ~88 with full markdown including tables) (LLM-OCR vs Traditional OCR, Parsli 2026; Open-Source OCR Models, E2E 2025). In other words: for pure parsing, today you may not want a 32B generalist VLM, but a small specialised VLM-OCR. But it is still GPU and it still pays tokens per image: the cost calculation does not disappear, the model just gets cheaper.

The split rule. It is not “OCR or VLM” as a global choice, but per-document triage. Run everything through a cheap first filter: classic OCR already returns a confidence score; if it is high and the document has no tables or visual elements, the OCR transcript will do and you have spent CPU. If confidence is low, there are tables, there is handwriting or there are graphical elements, escalate that page to the VLM. In a typical corpus, 80–90 % of pages are resolved with cheap OCR and only the rest consumes GPU. This is what makes an on-premise VLM sustainable: you do not serve everything with it, you use it as the expensive specialist you send only the hard cases.

Applied to the generic 4×H100 cluster

How this fits into a cluster of 4×H100 SXM 80GB:

Where to place the VLM. A VLM deserves a whole H100 or a large slice, for a VLM-specific reason that text-only intuition does not anticipate: the token cost per image means not many concurrent sessions fit. If each request brings one or several pages, each consumes thousands of KV tokens; the KV cache runs out with few simultaneous sessions. Compared with a text-only chat service where dozens of short conversations fit per GPU, a document service saturates the KV with a handful of heavy requests. That is why it makes no sense to carve the GPU finely for the VLM with aggressive MIG (GPU sharing is covered in sharing a GPU: time-slicing, MPS and MIG): the bottleneck is not idle compute but the KV, and a small slice runs out of KV before it makes use of the compute.

FP8 so it fits and to free up KV. Serve the VLM in FP8: at equal reported quality, the weights take up half and you leave the maximum VRAM for the KV cache, which is the scarce resource with multimodal contexts. For a 32B in FP8 (~32 GB) on an H100, those ~45 GB free for KV are what determine how many concurrent pages you can handle.

The VLM only for the subset that needs it. The most important architectural piece: do not put the VLM on the critical path of every document. The triage from the previous section lives upstream; the bulk (clean text) is resolved with OCR/PaddleOCR on CPU, and in a cluster with 4×H100 there is probably spare CPU to run it in parallel with the GPUs, and only the hard subset reaches the VLM. This frees the H100s for what really needs them and keeps a reasonable cost per document. It connects directly with FinOps and GPU multi-tenancy with LiteLLM: when you put a price on every token, a visual token costs the same as a text one, so a page through the VLM can cost 4× its transcript, and that should show up in the chargeback of the team that decides to send everything to the VLM for convenience.

Double cold start. An operational detail that bites: a VLM loads two models into memory, the vision encoder and the LLM, so its cold start is heavier than that of an equivalent LLM. If you are going to do model swapping or on-demand scaling, fast loading matters twice as much; that is where speeding up cold start with Tensorizer comes in. If instead you serve several models on one GPU by rotating them, the swap+sleep pattern (serving several models on one GPU) has to account for the VLM taking up more room and loading more slowly.

Consumer GPU, for reference. If someone wants to prototype outside the cluster, an RTX 5090 (Blackwell, 32 GB) comfortably serves a 7–8B VLM in FP8 (~8.5 GB of weights) with KV to spare for a few pages per request; a 32B in FP8 (~32 GB) no longer fits with useful KV on that card and calls for the H100. The 5090 is good for validating the triage pipeline and the prompt; the real service belongs on the H100s.

What we have not covered

Video. Qwen3-VL handles video with dynamic FPS sampling; each frame is an image and the token cost multiplies by the number of sampled frames. One minute of video can be tens of thousands of tokens. It is the most expensive multimodal beast and deserves its own analysis.
Prefix caching of images. If the same image (a logo, a template) appears in many requests, can its KV be cached? It depends on the backend and on the stability of the placeholder; it connects with the attention backend.
Evaluation. How to measure that the VLM extracts the table correctly (it is not enough for it to “look right”): faithfulness against a set with structured ground truth.
Fine-tuning the VLM for a specific document domain (invoices, particular forms), which can put part of the domain’s “knowing how to see” into the weights and reduce what the prompt has to explain.

References

vLLM, Multimodal Inputs (limit_mm_per_prompt, precomputed embeddings): https://docs.vllm.ai/en/stable/features/multimodal_inputs/
vLLM, Vision Language Multi Image (mm_processor_kwargs, min/max pixels): https://docs.vllm.ai/en/latest/examples/offline_inference/vision_language_multi_image/
vLLM, Qwen3-VL Usage Guide (recipes): https://docs.vllm.ai/projects/recipes/en/latest/Qwen/Qwen3-VL.html
Qwen team, Qwen2.5-VL (blog, token formula, dynamic resolution): https://qwenlm.github.io/blog/qwen2.5-vl/
Qwen2.5-VL (Transformers docs, min/max pixels, token range): https://huggingface.co/docs/transformers/model_doc/qwen2_5_vl
Qwen3-VL Technical Report, arXiv 2511.21631: https://arxiv.org/abs/2511.21631
Qwen3-VL-8B-Instruct-FP8 (FP8 with quality almost identical to BF16): https://huggingface.co/Qwen/Qwen3-VL-8B-Instruct-FP8
vLLM issue #15125 (image placeholder order): https://github.com/vllm-project/vllm/issues/15125
PaddleOCR vs Tesseract, CodeSOTA 2026: https://www.codesota.com/ocr/paddleocr-vs-tesseract
LLM-Based OCR vs Traditional OCR, Parsli 2026: https://parsli.co/blog/llm-ocr-vs-traditional-ocr
Open-Source OCR Models 2025, E2E Networks: https://www.e2enetworks.com/blog/complete-guide-open-source-ocr-models-2025

Hardening and secrets in the sovereign LLM stack: defence in depth

Thu, 11 Jun 2026 11:00:00 +0000

Part of the operational series on squeezing a generic on-premise 4×H100 SXM 80GB LLM cluster. The sibling pieces: the document ingestion from PDF to indexed chunk that fills the vector database, the embeddings and rerankers service with TEI in production that feeds it, and, the most directly coupled to this post, the GitOps of the inference stack with Flux, because GitOps and secrets share a chicken-and-egg problem we solve here. Assembling all of this into a conversational assistant (LibreChat + LiteLLM + RAG) is covered by another post in the series, still in draft.

TL;DR

A sovereign LLM inference stack is, at minimum, six services: a gateway (LiteLLM or equivalent), an inference engine (vLLM), an embeddings/rerankers service (TEI), a vector database, a state database (conversations, users) and a front end (LibreChat). Six services are six attack surfaces, and the assistant “works” as soon as the gateway returns tokens, long before it is hardened. This post walks through defence in depth layer by layer:

Secrets: never in the clear in git. sealed-secrets (asymmetric encryption, controller in the cluster) vs SOPS/age (file encryption, simple, pure GitOps) vs External Secrets Operator + Vault (dynamic secrets, native rotation). GitOps’s chicken-and-egg problem.
Network: default-deny NetworkPolicy + explicit allow; L3/L4 with the standard NetworkPolicy, L7 and DNS-based egress with Cilium (eBPF); egress control so the data does not leave the perimeter; internal mTLS.
Pod security: runAsNonRoot, readOnlyRootFilesystem, dropped capabilities, seccomp RuntimeDefault, restricted Pod Security Standards, nothing privileged.
Supply chain: pin by digest, scan with Trivy, sign with cosign, admission control that rejects anything unsigned.
AuthN/Z: LiteLLM virtual keys, OIDC/LDAP in the front end, Kubernetes RBAC with least privilege.
Runtime: detection and, optionally, enforcement with Tetragon.
Data at rest: storage encryption, credentials for the vector database and the state database.

The thesis: hardening reduces the blast radius, it does not eliminate it. That is why you prioritise by impact, and the first job on a 4×H100 is the gateway↔vector store pair and egress-deny.

The analogy: the sovereign office

Picture an office holding sensitive documentation: the corpus that feeds the RAG, the users’ conversations, the engine’s credentials. It is not enough for the street door to lock. Real security is defence in depth: several layers, each assuming the previous one can fail.

The mapping is direct:

The building’s perimeter is egress control: making sure a leaked secret or a compromised process cannot exfiltrate the corpus to an external server. It is the most underrated layer and the first one I put in place in a sovereign deployment.
Access control at every interior door is the default-deny NetworkPolicy: the front end does not talk to the vector database directly because it has no reason to; only the strictly necessary pairs are open.
The fire partitions are pod isolation (restricted) and namespaces: if one service catches fire, the fire does not jump to the one next door.
The doorman who checks credentials is the admission control that verifies each image’s cosign signature before letting it into the cluster.
The cameras are Tetragon/Falco: they record what each process did and detect (or kill) anything anomalous.
The safe with a rotated key is the secrets: encrypted at rest, out of git, and rotated so a stolen key expires.

No layer is sufficient on its own. The building is secure because crossing them all is expensive.

Layer 1 — Secrets: GitOps’s chicken-and-egg problem

The stack is deployed by GitOps: the sibling piece on Flux reconciles the state declared in git against the cluster. That is excellent for manifests, but there is a foundational problem: the inference engine needs the Hugging Face token to download the model, the state database needs its password, the gateway needs its master key. Those secrets cannot go into git in the clear. Anyone with read access to the repo, and in an organisation with generous read that is a lot of people, plus every repo backup, plus every fork, would read them.

The chicken and the egg: GitOps wants all state to be in git, but secrets cannot be in git in the clear. The solution comes in three families:

Family A — sealed-secrets (asymmetric encryption, controller in the cluster)

Bitnami Sealed Secrets is a controller running in the cluster plus a client CLI, kubeseal. It uses asymmetric cryptography: there is a key pair. The public one is held by developers and is used to encrypt; the private one lives only in the cluster’s controller and is used to decrypt. The flow:

The developer takes a normal Secret and encrypts it with kubeseal, which obtains the controller’s public key. Result: a SealedSecret resource.
That SealedSecret is committed to git in the clear — it is encrypted, it is not readable.
The controller in the cluster detects it, decrypts it with its private key and creates the real Kubernetes Secret in the target namespace.

The key property: because the private key never leaves the cluster, neither the developer nor anyone with access to git can decrypt. And the encryption includes the namespace name: a SealedSecret sealed for inferencia cannot be moved to front and decrypted there — it behaves as if each namespace had its own key. The controller also manages rotation of the sealing keys, labelling them as active or compromised.

Advantage: it fits pure GitOps perfectly, the encrypted secret lives with the rest of the state. Limitation: the secret, once decrypted, ends up as a normal Kubernetes Secret — it sits in etcd, and etcd has to be encrypted separately (we will see that in layer 7).

Family B — SOPS + age (file encryption, simple)

SOPS encrypts the whole YAML/JSON file (or only its values) and leaves it in git encrypted, decrypting it only at deployment time. It supports cloud KMS (AWS/GCP/Azure), PGP and, the relevant one for on-premise sovereignty, age, a simple serverless offline encryption scheme. The GitOps operator (Flux brings native SOPS integration) decrypts on reconciliation using the age key stored in the cluster.

Advantage: simple, scriptable, with no server to maintain; the operational weight falls on safeguarding the age key. It is the usual recommendation for small teams or for getting started. Limitation: rotation is manual (re-encrypt everything with the new key) and there are no dynamic secrets.

Family C — External Secrets Operator + Vault (dynamic secrets, native rotation)

The External Secrets Operator (ESO) keeps no secrets in git at all. All that goes into git is an ExternalSecret: a reference saying “the password field comes from path secret/data/vectordb in such-and-such a SecretStore”. The operator reads from an external store, typically HashiCorp Vault, and synchronises the Kubernetes Secret. Vault keeps the secrets in its own encrypted backend and serves them over an authenticated API, so that they never reside in git and, with its dynamic engines, it can issue short-lived database credentials that expire on their own.

Advantage: native rotation, dynamic secrets, centralised auditing, a single point of governance. Limitation: you have to operate Vault (sealing/unsealing, high availability, access policy), which is real work.

How to choose, and rotation

Criterion	sealed-secrets	SOPS + age	ESO + Vault
Secret in git	encrypted	encrypted	reference only
Extra server	lightweight controller	none	Vault (heavy)
Rotation	automatic sealing keys	manual (re-encrypt)	native / dynamic
Dynamic (short-lived) secrets	no	no	yes
Fit with pure GitOps	excellent	excellent	good (references)
Operational curve	low	very low	high

For a sovereign 4×H100 getting started, SOPS/age or sealed-secrets cover 90% at minimal cost. When the system grows and requirements for frequent rotation and auditing appear, typical in ENS High Category scenarios, you migrate to ESO + Vault. The rotation rule: a secret that never rotates is a secret that, once leaked, is leaked forever. Rotation does not prevent the leak; it bounds its window of validity. We will come back to that when we talk about blast radius.

Layer 2 — Network: default-deny, egress and the arithmetic of the surface

The arithmetic of communication pairs

Without a network policy, in Kubernetes every pod can talk to every pod. That is the default posture, and it is the worst one for a sovereign system. With $N$ services, the number of possible ordered communication pairs (who-can-call-whom) is:

$$P_{\text{abierto}} = N \cdot (N-1) \approx N^2$$

With our $N = 8$ components (gateway, vLLM, TEI, vector database, state database, front end, plus the secrets controller and the observability one), that is $8 \cdot 7 = 56$ possible directed pairs. Fifty-six paths along which a compromised service could pivot laterally.

Now we apply a default-deny NetworkPolicy: nothing talks to anything except what is explicitly allowed. The real whitelist of an LLM stack is small. The strictly necessary pairs:

front end → gateway
gateway → vLLM
gateway → TEI
gateway → vector database
gateway → state database
front end → state database (sessions/users)

That is $E = 6$ edges. The communication surface falls from $56$ to $6$:

$$\frac{E}{P_{\text{abierto}}} = \frac{6}{56} \approx 0.107$$

Almost 89% of the possible paths are closed. The front end can no longer touch vLLM or TEI or the vector database directly; if somebody compromises the front end, they have no network route to the corpus. This is the difference between $\sim N^2$ and a whitelist $E \ll N^2$.

Standard L3/L4, L7 and egress with Cilium

Kubernetes’ native NetworkPolicy operates at L3/L4: it selects pods by label and allows/denies by port and protocol. That covers the bulk of the whitelist. But it has limits: it does not understand DNS or HTTP. This is where Cilium comes in, applying policies in the kernel via eBPF and extending them with CiliumNetworkPolicy:

Egress by FQDN/DNS: instead of pinning IPs (which change), allow egress only to huggingface.co so vLLM can download the model and block the rest. Critical for egress control.
L7: allow only certain HTTP methods/paths between the gateway and an internal service.
Enforcement modes: in the default mode, an endpoint with no policy selecting it has everything open; in always mode, everything is denied until a policy explicitly opens it. For a sovereign system, always + default-deny is the target posture.

Egress control is the sovereign piece par excellence. An internal service being unable to open connections to the internet except to a handful of explicit destinations means that, even if vLLM or the gateway is compromised, the corpus cannot leave the perimeter. The RAG is the asset of value; egress-deny is what stops it leaking out.

Internal mTLS

The NetworkPolicy says who can talk to whom, but it neither encrypts nor authenticates east-west traffic. For that, mTLS (mutual TLS: client and server authenticate each other). Cilium offers native mutual authentication; alternatives such as Linkerd or Istio give you a full service mesh. For a six-service stack, Istio’s full mesh is usually over-engineering; Cilium’s mutual authentication or Linkerd (lighter) is proportionate. The effect: an attacker on the cluster network cannot impersonate the gateway or eavesdrop on internal traffic in the clear.

Layer 3 — Pod security: restricted, and nothing privileged

The Pod Security Standards define three profiles: privileged (no restrictions), baseline (the reasonable minimum) and restricted (hardened). The target for the whole LLM stack is restricted, applied via Pod Security Admission with a namespace label in enforce mode. The restricted profile requires:

runAsNonRoot: true and a runAsUser other than 0. No container runs as root.
readOnlyRootFilesystem: true. The root filesystem is read-only; anything that needs to write goes to an emptyDir or an explicit volume. An attacker cannot leave persistent binaries in the container.
drop: ["ALL"] of Linux capabilities. No capabilities that are not needed.
seccomp RuntimeDefault, which applies the runtime’s profile and blocks dangerous syscalls.
allowPrivilegeEscalation: false, nothing privileged, no host namespaces.

The nuance with GPUs: pods using an H100 load NVIDIA’s device plugin, which historically tempted people to relax the securityContext. You do not need to run the inference container as privileged in order to use the GPU; device access is managed by the plugin, and the vLLM pod can and should run restricted. This is a fire partition: if vLLM is compromised, the attacker has a non-root process, with no capabilities, a read-only FS and trimmed syscalls — a terrible starting point for escalation.

Layer 4 — Image supply chain

The container image is third-party code running with access to your data. Three composed controls:

Pin by digest, not by tag. vllm/vllm-openai:latest is a moving target; @sha256:... is immutable. Pinning the digest guarantees that what was deployed is exactly what was audited.
Scanning with Trivy. Before promoting an image, Trivy enumerates its CVEs and composes the SBOM (component inventory). The pipeline fails if there are unmitigated critical vulnerabilities.
Signing with cosign and verification at admission. The Sigstore project (cosign for signing, Fulcio as a CA for ephemeral certificates via OIDC, Rekor as an immutable transparency log) allows the image to be signed. And Sigstore’s policy-controller, or Kyverno, is an admission controller that verifies the signature before admitting the pod: an image without a valid signature does not enter the cluster.

This is the doorman from the analogy: he checks each image’s credential (signature) at the door. A poisoned image uploaded to a registry, or a hijacked tag, stays outside because it carries no signature from the trusted issuer.

Layer 5 — AuthN/Z: virtual keys, OIDC and minimal RBAC

Three identity control points, from the outside in:

Front end (OIDC/LDAP). The human user authenticates against the corporate identity provider. The front end does not invent its own user system; it delegates to OIDC. This gives SSO, MFA and centralised revocation.
Gateway (LiteLLM virtual keys). The gateway issues virtual keys: each team, application or user has its own key with a budget, a rate limit and permitted models. The gateway’s master key, and the real API keys towards the engine, are never seen by the client; clients only handle their virtual key, revocable individually. If a virtual key leaks, that one and only that one is revoked.
Kubernetes RBAC with least privilege. Each service’s ServiceAccount has exactly the permissions it needs. The vLLM pod does not need to list Secret objects in other namespaces or create pods. Restrictive RBAC means a stolen service account token opens very little.

Layer 6 — Runtime: the cameras with Tetragon

The previous layers are preventive. What is missing is detection: what if something, despite everything, runs where it should not? Tetragon is observable security and runtime enforcement over eBPF, Kubernetes-aware. It hooks kernel events — process_exec, tcp_connect, security_file_open — with typical overhead below 1%, and it can move from observing (recording the event) to enforcement (killing the process or cutting the connection in the kernel, Sigkill).

The operational rule, which I set out in detail in the bubblewrap + Tetragon runbook: adopt first, block later. First you deploy in observation mode to build a baseline of the stack’s normal behaviour — which binaries vLLM executes, what the gateway connects to — without false positives. Only afterwards do you promote the clear rules to enforcement: kill any process that tries to read secret paths, or cut every tcp_connect to destinations outside the egress whitelist. Falco is the detection-only alternative over eBPF; Tetragon adds enforcement. These cameras also produce the audit evidence: what each service ran and which attempt was blocked.

Layer 7 — Data at rest

The last thing we protect is data standing still:

Storage encryption. The persistent volumes, where the vector database and the state database live, on top of disk encryption (LUKS/dm-crypt) or Ceph-level encryption. A disk stolen from the datacenter does not reveal the corpus.
etcd encryption. Remember that Kubernetes Secret objects, once decrypted by sealed-secrets or ESO, are normal objects in etcd. You have to enable etcd encryption at rest, or the secret sits in the clear in the control plane.
Database credentials. The vector database’s password and the state database’s are first-class secrets (layer 1), never embedded in the manifest or in environment variables in the clear in git.

Exposure table: service × port × external egress?

This table is the input for writing the NetworkPolicy resources. The egress column is the one that decides what leaves the perimeter.

Service	Internal port	Permitted callers	External egress?
front end	3080	(human ingress via OIDC)	No
gateway (LiteLLM)	4000	front end	No
vLLM	8000	gateway	Only `huggingface.co` for the initial download; zero in operation
TEI (embeddings/reranker)	8080	gateway	Only model download; zero in operation
vector database	6333	gateway	No
state database	5432	gateway, front end	No
secrets controller	—	(control plane)	No
observability	9090	internal scraping	No

The reading: no service needs external egress in normal operation. vLLM and TEI need it once to pull the model, and that permission can be temporary or restricted by FQDN to huggingface.co. Everything else is total egress-deny. If a service starts attempting outbound connections this table does not contemplate, Tetragon records it and, in enforcement, cuts it.

Blast radius: a leaked secret, with and without defence

The blast radius measures how much damage a compromise does. Let us model it for the realistic worst case: the state database’s credential leaks.

Without hardening (flat network, unrotated secret, no egress-deny, no detection):

The secret gives access to the state database from any pod (flat network → 56 open pairs).
The secret does not rotate → it is valid indefinitely; the exploitation window is $\infty$ until somebody notices.
Without egress-deny, the attacker dumps the whole database to an external server.
Without Tetragon, nobody finds out until the public incident.
Blast radius: the entire state database, exfiltrated, undetected, for an indefinite time.

With hardening (default-deny, ESO + Vault with dynamic credentials, egress-deny, Tetragon in enforcement):

The secret is only usable from the pod that has a network route to the state database (1-2 pairs, not 56).
With Vault’s dynamic credentials, the secret expires — say within a window $T$ of hours; past $T$, it is worthless.
Egress-deny prevents the dump to the internet: the attacker can read, but cannot take anything out.
Tetragon records the anomalous access and, in enforcement, kills the process attempting the outbound connection.

The qualitative reduction is enormous, but let us be quantitative about the window. If a static secret is valid forever and a rotated one with period $T$ is valid for at most $T$, and compromises arrive at rate $\lambda$, the expected number of live and exploitable secrets at any given instant goes from growing without bound to being capped at $\lambda \cdot T$. With rotation every 24 h ($T = 1$ day) against “never”, a given secret’s window falls from months to a day: a reduction of one to two orders of magnitude in temporal exposure. Combined with the reduction in network pairs ($56 \to \sim 2$, a factor of $\sim 28\times$) and egress-deny (from exfiltration possible to impossible by the direct route), the blast radius is drastically reduced.

But let us be honest: it does not reach zero. The attacker with the credential can read the state database during the window $T$ from the compromised pod. Hardening turned “indefinite, silent catastrophe” into “a bounded, detected incident with no exfiltration by the direct route”. That is exactly what defence in depth promises: not invulnerability, but that the cost of a full compromise is high and its radius small.

The ENS / NIS2 angle

These layers are not voluntary hygiene: they implement specific controls. The mapping is developed in detail in the technical controls ENS × ISO 42001 × EU AI Act post; here, the actionable summary:

Hardening layer	ENS measure (RD 311/2022)	NIS2 / framework link
Encrypted secrets + rotation	`op.exp.11` (cryptographic keys), `mp.info.3` (encryption)	Credential management; in High Cat., HSM
Default-deny NetworkPolicy + segmentation	`mp.com.1` (perimeter), `mp.com.4` (flow separation)	NIS2 art. 21: network security measures
Egress control	`mp.com.1` + `op.mon.1`	Exfiltration prevention
Internal mTLS	`mp.com.2-3` (confidentiality/integrity in transit)	Mandatory TLS
Restricted Pod Security	`op.exp.2` (hardened configuration)	Configuration hardening (CIS)
Trivy + cosign + admission	`op.exp.6` (malicious code), `op.ext.3` (supply chain)	NIS2 supply chain
OIDC + virtual keys + RBAC	`op.acc.1-2-5` (identification, access, authentication)	MFA in High Cat.
Tetragon runtime	`op.mon.1` (intrusion detection)	Monitoring and response
Encryption at rest + etcd	`mp.info.3` (encryption), `mp.si` (media)	Data at rest

The honest note for auditing: hardening reduces risk, it does not eliminate it, and maturity is demonstrated by prioritising by impact. A competent auditor does not want to see nine half-finished layers; they want to see that egress-deny on the critical asset (the corpus) and secrets management are solid before mTLS is perfect between low-sensitivity services. For the management and governance context, see also ISO/IEC 42001 as an AIMS and the EU AI Act mapping onto the architecture.

Applied to the generic 4×H100 cluster

In a real deployment on 4×H100 SXM 80GB, you do not harden the nine layers at once. The order, prioritised by impact:

Egress-deny on vLLM, TEI and the vector database, first. It is the barrier that stops the corpus leaving the perimeter, and it goes in as soon as the services start. Allow egress by FQDN to huggingface.co only during the model’s initial download; afterwards, zero. It is the layer with the highest return per hour invested.
Secrets for the inference engine and the databases, out of git. The Hugging Face token vLLM uses to download the model, the vector database’s and the state database’s passwords, the gateway’s master key: into sealed-secrets or, if there is already a rotation requirement, into ESO + Vault. Never in values.yaml in the clear.
Default-deny NetworkPolicy + the 6-edge whitelist. The gateway and the vector database are the most exposed services — the gateway because it receives all the traffic, the vector database because it holds the embedded RAG. Closing everything that is not on the whitelist cuts lateral pivoting from $\sim N^2$ to the 6 real edges.
restricted Pod Security in the inference namespace, including the vLLM pod with a GPU (it needs no privileges to use the H100).
cosign + admission so only signed images get in; Trivy in the GitOps pipeline of Flux.
Tetragon in observation, baseline, and then enforcement over the egress of the sensitive pods.
Encryption at rest for the databases’ volumes and for etcd.

The gateway and the vector database are the two I harden first inside the whitelist: the gateway for being the exposed face, the vector database for holding the asset that egress-deny protects. The rest is built on top, layer by layer, always assuming the previous one can fail.

What hardening does NOT solve

To close honestly, what these layers do not cover and what needs other pieces:

Prompt-level attacks (jailbreak, indirect injection via the RAG corpus): that is the job of guardrails and LLM Guard, not of NetworkPolicy.
Agents with legitimate permissions doing something harmful: the runtime isolation from the agent isolation post bounds what an agent can touch, but a granted permission is a usable permission.
The human factor: a well-managed secret shared over Slack is still leaked. Rotation bounds the window, it does not eliminate the mistake.
Zero-day vulnerabilities in the components themselves: Trivy detects what is known; the unknown gets through until the CVE is published.

Hardening is a multiplier on the cost of attacking, not an impassable wall. Its value lies in making an individual compromise bounded, detected and with no way out — and that, for a sovereign system holding sensitive data, is exactly the goal.

GitOps for the inference stack with Flux: operating the assistant as code

Thu, 11 Jun 2026 10:20:00 +0000

This post is part of the operational series on how to squeeze a generic on-premise 4×H100 SXM 80 GB LLM cluster. The sibling pieces: the document ingestion pipeline for RAG that fills the vector store we deploy here, the embeddings and reranker service with TEI that is one of the services GitOps manages, and the secrets hardening of the sovereign stack, which goes deep into the secrets problem we only state here. The complete end-to-end assistant (LibreChat + LiteLLM + RAG) that orchestrates all of this will get its own post.

TL;DR

An LLM assistant in production is a distributed system, not a binary. At a minimum it has an inference engine (vLLM or similar), an L7 gateway that routes per model and applies rate limiting, a chat front end, a vector store for RAG, an embeddings/reranker service, and an observability stack. Six substantial services, spread across three or four namespaces, with start-up dependencies (the front end is useless if the gateway is not there, the gateway is useless if the engine has not loaded the model, RAG does not answer if the vector store is empty). Operating it by hand, a kubectl apply here and a helm upgrade there, produces a cluster whose real state nobody can reconstruct: there is no record of what was applied, or when, or why. GitOps solves this with four principles (OpenGitOps): the desired state is declarative, it is versioned and immutable in git, it is applied automatically by an agent (nobody SSHes in to deploy) and it is continuously reconciled by comparing the actual state against the declared one. Flux is that agent: six controllers (source, kustomize, helm, notification, image-reflector, image-automation) that clone the repo, render Kustomize/Helm, apply to the cluster, correct drift and, optionally, write back to git to bump the tag of a new image. The unit of work is the Kustomization or the HelmRelease, with interval (how often it reconciles), prune (deletes what is no longer in git), dependsOn (orders the start-up) and health checks. The interval sets the MTTR of drift: with interval=1m, a manual edit to the cluster is reverted in ≤ 1 min on average. Secrets cannot sit in the clear in git, a chicken-and-egg problem, and it is solved with SOPS/age, sealed-secrets or External Secrets + Vault. Rollback is a git revert. GitOps is not free: the learning curve is real, and debugging why a Kustomization is not reconciling is a new skill.

The analogy: the master plan and the foreman who does not negotiate

Picture a large building site with a signed and filed master plan, and a foreman who has a single order: the site must be identical to the plan, at all times. The foreman does not improvise. Every so often he walks the site with the plan in hand and compares: if a wall is where the plan says, he leaves it; if someone moved a partition overnight without updating the plan, he puts it back; if the plan says there is a column and it does not exist, he builds it; if a column exists but no longer appears on the plan, he knocks it down. The plan is the single source of truth: to change the site you do not touch the site, you change the plan, and the foreman takes care of the rest on his next round.

That is exactly the mechanics of GitOps. Git is the master plan: declarative (it describes the end state, not the steps), versioned (every change is signed into the history, with author and reason), immutable (a commit is not rewritten). Flux is the foreman: every interval it walks the cluster, compares against git and converges. And here is the lesson that separates GitOps from “keeping the YAML in a repo”: the foreman undoes manual changes. If an operator comes in with kubectl edit and raises the inference engine replicas from 2 to 4 at three in the morning to put out a fire, on the next round Flux takes it back to 2, because the plan says 2. This enrages anyone coming from the imperative world, and it is precisely the point: if you want 4 replicas permanently, you update the plan. The cluster stops being a system with its own opaque memory and becomes a reproducible projection of git. Delete the entire cluster, point a fresh Flux at the same repo, and the site is rebuilt identically.

The analogy also marks the boundary: the plan describes the building, not who holds the keys to the materials store. Secrets (passwords, tokens, keys) cannot go into the public plan. That is the chicken-and-egg problem we cover further down and that the hardening piece develops in full.

The problem: an assistant is six services, not one

Before Flux, it is worth fixing the size of the problem. A minimally serious sovereign LLM assistant, deployed on the generic reference cluster, has this service topology:

Service	Function	Typical namespace	State
Inference engine (vLLM)	Serves the tokens of the general LLM and the code one	`llm-serving`	Stateless (model from object store)
Embeddings + reranker (TEI)	Vectorises queries and reorders candidates for RAG	`llm-serving`	Stateless
Vector store (Qdrant)	Stores and searches document embeddings	`data`	Stateful (PVC)
L7 gateway (Envoy AI Gateway / LiteLLM)	Per-model routing, rate limiting, auth	`gateway`	Semi-stateless
Chat front end	Assistant UI	`apps`	Stateless
Observability (Langfuse, OTel Collector, dashboards)	LLM-aware traces, metrics and logs	`observability`	Stateful (Langfuse Postgres)

Six services, four or five namespaces, two components with persistent state, and a real dependency graph: the front end calls the gateway, the gateway calls the engine and the embeddings service, RAG depends on the vector store being full, and everything emits traces to observability. Deploying this by hand means remembering the order, the values of each helm install, the ConfigMaps, the Secrets, the nodeSelectors that pin the GPU pods to the right nodes. Do it twice (a staging environment and a production one) and the two diverge within days. That is exactly what GitOps eliminates.

The six Flux controllers

Flux is not a monolithic binary but a set of cooperating controllers, the GitOps Toolkit. A default installation brings four; the two image automation ones are added with --components-extra (Flux installation):

source-controller: clones and keeps the sources up to date, whether git repos (GitRepository), Helm repos (HelmRepository), buckets or OCI artifacts. It is the one “holding the plan”: it exposes the repo contents as an internal artifact that the others consume.
kustomize-controller: takes a source artifact, renders Kustomize (bases + overlays) and applies the result to the cluster. It is responsible for prune, dependsOn and the health checks of the Kustomizations.
helm-controller: reconciles HelmRelease objects, installing and updating Helm charts declaratively, without anybody running helm from a terminal (helm-controller).
notification-controller: the bridge to the outside world in both directions. It receives webhooks (to reconcile instantly on every push, instead of waiting for the interval) and emits events and alerts to Slack, an internal chat or an incident system.
image-reflector-controller: scans container registries and stores the tags it finds in an internal database. It is the “eyes” of image automation.
image-automation-controller: uses what the eyes see to write back to git, committing the new image tag into the manifests when a version appears that satisfies the policy.

A cluster without image automation does not need the last two. But for an inference stack where the model server is updated fairly often, they are what automates version promotion without touching anything by hand.

The diagram has three paths. The grey one is the main flow: git → source-controller → kustomize/helm-controller → cluster. The dashed red one is the reconciliation loop that runs every interval: it compares actual state against desired, fixes drift and prunes what is left over. The purple one is image automation: the reflector sees a new tag in the registry, the policy decides whether it qualifies, and the automation-controller writes back to git. That last path is the one that closes the circle and turns git into a system that updates itself.

Repo structure: clusters, infrastructure, apps

The most widespread convention separates three levels of responsibility. It is not the only one, but it ages well:

gitops-repo/
├── clusters/
│ └── prod/
│ ├── infrastructure.yaml # Kustomization → ./infrastructure
│ └── apps.yaml # Kustomization → ./apps (dependsOn infrastructure)
├── infrastructure/
│ ├── controllers/ # ingress, cert-manager, GPU operator, KEDA...
│ └── configs/ # ClusterIssuer, RuntimeClass, StorageClass...
└── apps/
 ├── base/ # manifests common to all environments
 │ ├── llm-engine/ # vLLM HelmRelease
 │ ├── gateway/ # L7 gateway HelmRelease
 │ ├── chat-front/ # Deployment + Service + Ingress
 │ ├── vector-store/ # Qdrant HelmRelease (+ PVC)
 │ └── observability/ # Langfuse + OTel HelmRelease
 ├── staging/ # overlay: low replicas, small model
 └── prod/ # overlay: high replicas, large model, MIG

The key piece is Kustomize’s base / overlay separation. The base/ describes the service once; each overlay (staging/, prod/) applies a patch with the environment’s differences: number of replicas, model size, MIG profile, nodeSelector, --gpu-memory-utilization. That avoids duplicating complete manifests per environment. The clusters/ folder is the entry point Flux reconciles first: it holds the root Kustomizations that point to infrastructure/ and apps/, with a dependsOn that guarantees the infrastructure (CRDs, operators, storage classes) is ready before the applications.

How many manifests: the calculation that justifies overlays

This is where the numbers make the argument. Without overlays, each service needs a complete set of manifests per environment. With $N$ services and $M$ environments, the cost in files to maintain is:

$$ \text{files}_{\text{naïve}} = N \times M \times k $$

where $k$ is the average number of manifests per service (Deployment/HelmRelease + Service + ConfigMap + PVC + Ingress ≈ 5). For our assistant, $N=6$ services and $M=3$ environments (dev, staging, prod), with $k=5$:

$$ 6 \times 3 \times 5 = 90 \text{ complete files, each maintained separately.} $$

With the base/overlay structure, the base is written once and each overlay contains only the patch with the differences (typically 1 short file per service per environment):

$$ \text{files}_{\text{overlay}} = \underbrace{N \times k}_{\text{base}} + \underbrace{N \times M}_{\text{patches}} = 6 \times 5 + 6 \times 3 = 30 + 18 = 48 $$

It is not just that there are fewer files (48 against 90): it is that the bulk of a change happens in a single place. Changing the engine’s memory limit for every environment is one edit in base/, not three synchronised edits. The saving factor grows with $M$: for 5 environments, the naïve version is 150 files and the overlay one is 60. Duplication is the enemy of auditability, and overlays attack it at the root.

The reconciliation loop: desired vs actual

The heart of Flux is a control loop that never ends: every interval it reads the desired state (git), reads the actual state (cluster), computes the difference and applies it. It is the same principle as a thermostat: it reads the target temperature (git), reads the actual temperature (cluster) and switches the boiler on or off until they match. It does not “deploy once”; it converges forever.

A Kustomization manifest for the applications, with the pieces that matter (Kustomization reference):

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
 name: apps
 namespace: flux-system
spec:
 interval: 1m # reconciles every minute: sets the drift MTTR
 path: ./apps/prod  # production overlay
 prune: true # deletes from the cluster whatever is removed from git
 sourceRef:
 kind: GitRepository
 name: gitops-repo
 dependsOn:
 - name: infrastructure  # does not apply apps until infra is Ready
 wait: true # waits for the resources to be healthy
 timeout: 5m  # fails the reconciliation if it does not converge in 5 min
 healthChecks:
 - apiVersion: apps/v1
 kind: Deployment
 name: gateway
 namespace: gateway

Four fields do the heavy lifting:

interval defines how often the foreman’s round runs. It is the direct determinant of the drift MTTR (next section).
prune: true switches on garbage collection: objects that were applied before but are no longer in the current git revision are deleted from the cluster automatically. Without prune, a service withdrawn from the repo keeps running, and the plan and the site diverge silently.
dependsOn orders the graph: Flux does not apply the manifests of a Kustomization until every referenced one is in state Ready: True. It is the mechanism that guarantees data → gateway → front end.
wait + healthChecks: with wait: true, Flux monitors every applied resource and waits for them to be ready before marking the reconciliation as successful; healthChecks lets you tune exactly which resources to watch. That is what makes a dependsOn mean something: the dependency is not satisfied until it is healthy, not merely applied.

MTTR and drift detection as a function of the interval

The interval is the only parameter the operator chooses to govern the speed of the loop, and it translates directly into operational metrics. If a deviation (drift) happens at a random instant within the reconciliation period $T$, the waiting time until Flux detects it is uniformly distributed over $[0, T]$, with mean:

$$ \mathbb{E}[t_{\text{detection}}] = \frac{T}{2} $$

To that you add the correction time $t_c$ (render, diff, apply), normally a few seconds. The drift MTTR comes out as:

$$ \text{MTTR}_{\text{drift}} = \frac{T}{2} + t_c $$

With interval=1m and $t_c \approx 10\text{s}$, drift is corrected in $\frac{60}{2} + 10 = 40$ s on average, with a worst case of $60 + 10 = 70$ s. With interval=10m, the mean rises to $5\text{min}\;10\text{s}$ and the worst case to more than 10 min: a window in which a manual change stays active. The temptation is to set interval=10s and forget about it, but there is a cost: every reconciliation consumes CPU and, above all, makes requests to the cluster API and to the registry. With dozens of Kustomizations reconciling every 10 s, the kube-apiserver and the image-reflector start to feel the pressure. The practical rule: a short interval (1m) for critical applications whose drift hurts, a long interval (10–30m) for stable infrastructure that almost never changes, and notification-controller webhooks for instant reconciliation on every push, so that the interval governs only drift and not deployment latency.

Helm as code: the inference engine HelmRelease

For the engine, the gateway and the front end, the usual approach is to package them as Helm charts and declare them with HelmRelease. The helm-controller reconciles them without anyone ever running helm:

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata:
 name: llm-engine
 namespace: llm-serving
spec:
 interval: 10m
 chart:
 spec:
 chart: vllm
 sourceRef:
 kind: HelmRepository
 name: vllm-charts
 values:
 image:
 tag: "0.8.4" # managed by image automation (see below)
 replicaCount: 2
 nodeSelector:
 nvidia.com/gpu.product: H100-SXM
 resources:
 limits:
 nvidia.com/gpu: 1
 extraArgs:
 - "--gpu-memory-utilization=0.90"
 - "--tensor-parallel-size=1"

Everything that in the imperative world would be a vllm serve flag, --gpu-memory-utilization or --tensor-parallel-size, now lives as versioned data. Changing the fraction of VRAM the engine reserves is a commit, reviewable in a PR, with history. The relationship with sharing a GPU is direct: the MIG profile and the --gpu-memory-utilization are deployment parameters, and here they are code.

Image automation: bumping the tag without touching anything by hand

The inference engine is updated fairly often (vLLM patches, security fixes). Without automation, every new version requires somebody to edit the tag in the HelmRelease. Flux image automation does it on its own, with three objects (automate image updates):

apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImageRepository
metadata:
 name: vllm
 namespace: flux-system
spec:
 image: registry.example.local/inference/vllm
 interval: 5m  # scans the registry tags every 5 min
---
apiVersion: image.toolkit.fluxcd.io/v1beta2
kind: ImagePolicy
metadata:
 name: vllm
 namespace: flux-system
spec:
 imageRepositoryRef:
 name: vllm
 policy:
 semver:
 range: ">=0.8.0 <0.9.0" # only patches and minors within 0.8.x–0.8.x
---
apiVersion: image.toolkit.fluxcd.io/v1beta1
kind: ImageUpdateAutomation
metadata:
 name: vllm
 namespace: flux-system
spec:
 interval: 5m
 sourceRef:
 kind: GitRepository
 name: gitops-repo
 git:
 commit:
 author:
 name: fluxbot
 email: fluxbot@example.local
 messageTemplate: "auto: bump vllm to {{ .NewTag }}"

The division of roles: ImageRepository scans all the tags of the image repository and stores them in the reflector’s internal database. ImagePolicy reads those tags and picks the “latest” according to the policy; the policy field is mandatory and defines how the selection is made (ImagePolicy). ImageUpdateAutomation takes the chosen tag, edits the manifest in the repo (wherever there is a # {"$imagepolicy": "flux-system:vllm"} marker) and commits (ImageUpdateAutomation).

The choice of policy matters:

semver: interprets tags as semantic versions and picks the highest one that satisfies the range (range: ">=0.8.0 <0.9.0"). By default it excludes prereleases (0.8.0-rc.1 does not qualify) unless they are asked for explicitly. It is the right option for production: you stay inside a tested version range and do not jump to a new major by accident.
regex / alphabetical / numerical: for tag schemes that are not pure semver, for example main-<sha>-<timestamp> from an internal pipeline. More flexible, but it forces you to trust that tag order reflects version order, which is fragile.

A safety guideline: instead of having ImageUpdateAutomation push directly to main, configure it to write to a branch and open a PR. That way the version bump goes through human review or a validation pipeline before reaching the cluster. Automation proposes; the human (or an automatic gate) disposes. And for production, the version bump is not the deployment: it is the trigger for the progressive rollout described in canary, blue-green and shadow.

Secrets in GitOps: the chicken-and-egg problem

GitOps demands that all the desired state is in git. But secrets, the Langfuse Postgres password, the registry token, the gateway API keys, cannot go into a repo in the clear, not even a private one: git history is immutable, and a secret committed once stays there forever. This is the fundamental tension: GitOps wants everything in git; security forbids secrets in git. There are three families of solution, all with the same underlying idea, that only the encrypted secret goes into git and the cluster holds the key to decrypt it:

SOPS + age/KMS: you encrypt the sensitive values of a manifest with SOPS (it leaves the keys readable and encrypts only the values). The encrypted YAML goes to git; the kustomize-controller carries an age key or reaches a KMS to decrypt it at apply time. Simple, with no extra components in the cluster.
sealed-secrets: a controller in the cluster holds a private key. You encrypt the secret against its public key (with kubeseal), the encrypted SealedSecret goes to git, and the controller decrypts it into a normal Secret inside the cluster. The private key never leaves the cluster.
External Secrets + Vault: not even the encrypted secret goes into git, only a reference (ExternalSecret) saying “the value of this key is in Vault, at this path”. The External Secrets operator resolves it at apply time. It is the cleanest pattern for many secrets and frequent rotation, at the cost of operating a Vault.

The choice depends on the volume of secrets and on whether you already have a central manager. For a small stack, SOPS+age is enough and has no dependencies. The sibling hardening piece goes into the detail of each option, key rotation and the threat model.

Promotion and rollback: git revert is the panic button

The most elegant consequence of having the cluster as a projection of git is that rollback is a git revert. If a deployment breaks production, the new engine gives worse latency or the gateway starts returning 5xx, there is no need to remember which version was there before or to rebuild the state by hand. You revert the commit that introduced the change, and on the next reconciliation (≤ interval, or instantly with a webhook) Flux returns the cluster to the previous state. The git history is, literally, the deployment history: every commit is a restore point with author, date and reason.

This fits with the promotion strategies of the canary, blue-green and shadow post: Flux manages what is declared, and the progressive rollout tool manages how traffic moves between the old version and the new one. The git revert is the last-resort rollback (it returns everything to a known state); the canary is the mechanism that avoids needing it. For services where the version change is delicate, the right approach is for image automation to do the bump on a branch, for the canary to validate the regression gates, and only then to promote the commit to main.

The seams: GitOps is not free

It would be dishonest to sell GitOps as magic. It has real costs that any team pays:

Learning curve. The declarative model is a change of mindset. The operator who has spent years fixing incidents with kubectl edit has to unlearn the reflex: now the cluster reverts their changes and that, at first, feels like Flux “fighting them”. Understanding that the change is made in git, not in the cluster, takes weeks of discomfort.

Debugging the reconciler. When a Kustomization does not apply, the error is not in the pod; it is in the reconciliation chain. Did the source-controller clone the right revision? Did the kustomize-controller render the overlay properly? Did a dependsOn health check fail? Did the timeout fire before the large model finished loading? Diagnosing this needs flux get, flux logs, reading the status.conditions of the Flux objects and understanding which link got stuck. It is a new skill, different from debugging plain Kubernetes.

Timeouts and slow loads. The timeout of a Kustomization with wait: true has to be longer than the start-up time of the slowest service. An inference engine that takes 4 minutes to load a large model from the object store will fail a Kustomization with timeout: 2m, even though everything is fine. Calibrating timeouts per service is fine-grained work.

The drift you did want. Sometimes the operator needs an urgent temporary change and Flux reverts it. The correct answer, suspending reconciliation of that Kustomization with flux suspend, making the change, then reflecting it in git and resuming, is discipline that has to be built. Without that discipline, people end up fighting the foreman at 3 in the morning.

None of these costs cancels the benefit. But a team adopting GitOps expecting “everything to be easier from day one” gets frustrated. It is easier from month two onwards, once auditability, reproducibility and trivial rollback have paid off the curve.

Applied to the generic 4×H100 cluster

On the reference cluster (4×H100 SXM 80 GB per GPU node, NVLink, plus CPU and control nodes), the assistant stack is managed entirely by GitOps. Hardware decisions become data in the overlays.

nodeSelectors and MIG profiles as code. The prod overlay pins each service to the right node and declares the GPU profile. The general engine uses a whole GPU; embeddings and the small LLM fit in MIG slices, exactly the split from the sharing a GPU post, but now versioned:

# apps/prod/llm-engine-patch.yaml
spec:
 values:
 nodeSelector:
 nvidia.com/gpu.product: H100-SXM
 extraArgs:
 - "--gpu-memory-utilization=0.92"
 - "--tensor-parallel-size=1"
---
# apps/prod/embeddings-patch.yaml — on a MIG slice
spec:
 values:
 nodeSelector:
 nvidia.com/mig.config: "3g.40gb"
 resources:
 limits:
 nvidia.com/mig-3g.40gb: 1

The MIG profile (3g.40gb), the VRAM fraction (0.92), the tensor-parallel setting: all of it is text in a PR. Changing the GPU split between services is a reviewable commit, not a hand-run nvidia-smi mig session that nobody records.

Start-up order with dependsOn. The assistant’s dependency graph is encoded with chained dependsOn. The right order is data → embeddings/engine → gateway → front end:

# apps/prod/gateway.yaml
spec:
 dependsOn:
 - name: llm-engine  # the gateway is useless without the engine
 - name: embeddings
 # ...
---
# apps/prod/chat-front.yaml
spec:
 dependsOn:
 - name: gateway  # the front end is useless without the gateway

With wait: true on each Kustomization, Flux does not mark llm-engine as Ready until the engine pod passes its health check, which in the case of vLLM means model loaded and endpoint responding, not just pod started. Only then does it begin applying the gateway. This avoids the classic cascade of failures of manual deployment: bring up the front end first, see a 502 because the gateway is not there, bring up the gateway, see a 503 because the engine is still loading the model. With dependsOn + wait, the order is guaranteed by the reconciler, not by the operator’s memory.

The result: deleting the assistant’s entire namespace and letting Flux rebuild it from git produces exactly the same stack, in the same order, with the same GPU configuration. The 4×H100 cluster stops having a configuration known only to whoever built it, and becomes a reproducible projection of a repo that anyone can audit.

Conclusion

GitOps is not a tool, it is an inversion of the direction of control: instead of pushing changes to the cluster, you declare the state in git and let an agent pull the cluster towards it. For an LLM assistant (six services, several namespaces, start-up dependencies, delicate GPU configuration) that inversion turns a fragile and opaque system into a reproducible and auditable one. Flux is the foreman who keeps the site identical to the plan and undoes any change that has not gone through the plan first. The price is a real learning curve and a new debugging skill. The prize is that the cluster stops keeping secrets about itself: everything it is, is written down, signed and reproducible in git.

References

OpenGitOps — GitOps principles — opengitops.dev
Flux installation — fluxcd.io/flux/installation
Flux Kustomization — fluxcd.io/flux/components/kustomize/kustomizations
helm-controller — github.com/fluxcd/helm-controller
Flux Image Policies — fluxcd.io/flux/components/image/imagepolicies
Flux Image Update Automations — fluxcd.io/flux/components/image/imageupdateautomations
Automate image updates to Git — fluxcd.io/flux/guides/image-update
SOPS — github.com/getsops/sops

Serving embeddings and rerankers with TEI in production

Thu, 11 Jun 2026 09:40:00 +0000

Sixth piece in an operational series about squeezing a generic on-premise LLM cluster of 4×H100 SXM 80 GB. If RAG on the CPU argued where the data plane runs and the embedder zoo decided which model you serve, this post looks at the engine that serves them: the embeddings and rerankers server. The sisters in this batch, document ingestion from PDF to indexed chunk, GitOps for the inference stack with Flux and hardening and secrets for the sovereign stack, build the rest of the system around this piece. The closer of the series, the end-to-end sovereign assistant with LibreChat + LiteLLM + RAG, is still in draft.

TL;DR

Serving embeddings is not model.encode(text) scattered through the ingestion code. Just as you do not put a transformers.generate() inside your API and call it vLLM, you do not put the embedder inline either: you put it behind a dedicated server that does batching, controls concurrency, emits metrics and exposes a single HTTP contract that both ingestion and query-time reuse. In the 2026 open source ecosystem that server is TEI, Text Embeddings Inference from Hugging Face: a Rust binary that serves embedding, reranker (cross-encoder) and sequence classification models, with token-based dynamic batching, and that exposes OpenAI-compatible endpoints (/v1/embeddings) on top of the native /embed, /embed_sparse, /rerank and /predict (TEI repo, docs). It runs on CPU (ONNX Runtime, Intel MKL) and on CUDA (with FlashAttention and cuda graphs), with the same contract on both sides, so the rest of the system does not know, or care, what silicon sits behind it. The technical piece that gives almost all the performance is dynamic batching: TEI groups requests that arrive separately into a batch that shares the fixed cost of the forward, controlled by --max-batch-tokens (how many tokens fit per batch) and --max-concurrent-requests (backpressure). Grouping raises throughput, going from serving 1 sequence to 32 multiplies tokens/s at almost constant cost until the hardware saturates, in exchange for some latency while the tray fills. The numbers: a 568M encoder (bge-m3, bge-reranker-v2-m3) takes ~1.1 GB in fp16 and ~0.57 GB in int8, so several replicas fit even in a small H100 slice; embedding a short query online costs tens of ms, but reranking the top-50 costs linearly in candidates because a cross-encoder evaluates each (query, doc) pair separately. The split across the 4×H100 cluster: TEI-CPU on the fleet for batch ingestion (no SLA, throughput/€), TEI-GPU in a MIG slice of one H100 for high-QPS online embeddings and rerank, talking to ingestion and to the LiteLLM gateway through the OpenAI contract.

The analogy: the stamping press that only stamps vectors

Picture a workshop with a specialised stamping press. It does not make varied parts; it does one single thing: it takes flat material and, in one stroke, stamps a shape into it. In our case it takes text and stamps a vector. And like every industrial press, it has one property that defines all its behaviour: the cost of the stroke is almost fixed, no matter how much material you put on the tray.

Lowering the press, heating the die, aligning, pressing and lifting costs, say, one second. If you put a single sheet in, you spend that whole second stamping one part: a waste. If you fill the tray with thirty sheets and lower the press once, you spend nearly the same second and come out with thirty stamped parts. The cost per part collapses. The press performs far better when it fills the tray before pressing.

This is exactly the dynamic batching of an embeddings server. The “stroke” is the model’s forward pass on the GPU or the CPU: an operation that loads the weights, multiplies them by the activations and emits the vectors, with a large fixed cost that does not scale with how many sequences you process at once, until you saturate memory bandwidth or the compute units. Serving requests one at a time is stamping one sheet per stroke. Grouping them into a batch is filling the tray.

But the press has an operational dilemma, and here is the heart of the post. If you wait until the tray is completely full before pressing, the first sheets that arrived sit waiting for the rest: latency. If you press as soon as one sheet drops so nobody has to wait, you go back to the waste of one stroke per part: low throughput. The embeddings server solves this with a bounded wait window: it gathers whatever arrives within a window of microseconds to milliseconds, or until the batch token budget is full, and then it presses. That is the policy that turns a pile of independent requests into a few efficient strokes without anyone waiting too long.

The rest of the post is, at bottom, about configuring that press properly: how many sheets fit on the tray (max-batch-tokens), how much queue is tolerated before work is rejected (max-concurrent-requests), whether the press lives on the cheap CPU fleet (nightly ingestion, pure throughput) or in a slice of the expensive GPU (online, bounded latency), and why the reranker is a different press whose cost does grow with the number of parts.

What TEI is, exactly

Text Embeddings Inference (TEI) is Hugging Face’s inference server for the family of models that does not generate tokens: embedders, cross-encoder rerankers and sequence classifiers. It is to encoders what vLLM is to generative LLMs: a high-performance server, written in Rust, that handles tokenisation, batching, concurrency, metrics and the HTTP contract, leaving the application code to only make calls (TEI repo).

Three model types, three jobs:

Embedding (bi-encoder): turns a text into a reusable vector. bge-m3, Snowflake Arctic Embed, multilingual-e5. The vector is stored in the index and reused on every search.
Reranker (cross-encoder): takes a pair (query, document) and emits a relevance score, not a vector. bge-reranker-v2-m3. It produces nothing reusable: every pair is evaluated again.
Sequence classification: emits labels/probabilities over a text. Useful for lightweight guardrails, routing, language or toxicity detection.

The endpoints

TEI exposes both an OpenAI-compatible endpoint and its native endpoints, verified against the current documentation (Quick Tour, DeepWiki):

Endpoint	What for	Model type
`/v1/embeddings`	Embeddings with the OpenAI contract (drop-in for clients that already speak OpenAI)	embedder
`/embed`	Dense embeddings, TEI’s native API	embedder
`/embed_sparse`	Sparse embeddings (lexical head of models like `bge-m3`/SPLADE)	embedder
`/rerank`	Orders a list of texts by relevance against a query	reranker (cross-encoder)
`/predict`	Sequence classification (labels/scores)	classifier
`/embed_all`	Per-token embeddings (no pooling)	embedder
`/similarity`	Direct similarity between texts	embedder
`/metrics`, `/health`, `/info`, `/tokenize`, `/decode`	Operation, observability and utilities	all

The operational key is in the first row: TEI speaking the OpenAI contract on /v1/embeddings means the embedder is swappable without touching client code. The same code that called OpenAI’s text-embedding-3-large points at your on-prem TEI by changing the base_url, and the gateway (LiteLLM) routes it as one more provider. /rerank, by contrast, is not part of the OpenAI standard, no such endpoint exists in their API, so the gateway treats it as a reranking-specific endpoint (LiteLLM TEI rerank).

The backends

TEI runs on CPU and on GPU with the same HTTP contract, which is exactly the property the CPU/GPU split of RAG needs (TEI docs, hardware):

CPU: cpu-* image. ONNX Runtime (recommended) or Intel MKL backend. It exploits the integer compute paths (AVX-512+VNNI, AMX on 4th-gen Xeon) when the model is in int8. It is the batch ingestion engine.
CUDA: architecture-specific images (cuda-*, with variants for different compute capabilities). It uses FlashAttention and cuda graphs to squeeze the GPU. It requires compute capability ≥ 7.5 (Volta is out) and CUDA 12.2+ compatible drivers. It is the high-QPS online engine.
There is also Metal support (Apple Silicon) and experimental ROCm (AMD Instinct), less relevant to the on-prem case of this post.

The golden rule: the same bge-m3 served by TEI-CPU and by TEI-GPU exposes an identical /v1/embeddings. The client does not notice the silicon. That is what lets you put ingestion on CPU and online on GPU without rewriting anything.

Why a dedicated server and not calling the model inline

The temptation, especially in the prototype, is to load the embedder into the application process and call model.encode() directly. It works in the demo and breaks in production for five reasons, all solved by the dedicated server:

Batching. Your ingestion code processes documents; your query code embeds one query at a time. Inline, each encode() is an independent forward, one press stroke per sheet. A dedicated server joins requests from different origins (ingestion + several concurrent queries) into a single batch and amortises the fixed cost. This is what really multiplies throughput, and you cannot do it if the model lives inside each isolated process.
Concurrency and backpressure. Under load, what happens when 10,000 requests arrive at once? Inline, you run out of memory or queue without control. TEI has --max-concurrent-requests: above that limit it rejects instead of degrading everyone, which is the correct way to manage overload (TEI CLI args).
Metrics. TEI exposes /metrics in Prometheus format: latencies, batch size, tokens/s, queued requests. Inline you have none of this without hand-instrumenting every encode(). To observe the data plane you need that endpoint.
A reusable HTTP contract. The same server is consumed by batch ingestion, by online query-time and by any other service that needs vectors. One copy of the model in memory, one version point, one place to change the model. Inline, each service loads its own copy and the version drifts on its own.
Lifecycle separation. The embedder is updated, restarted or scaled without touching the application. If tomorrow you swap bge-m3 for Snowflake Arctic, you change the server container, not the code of N services. (Remember the embedder drift trap: changing embedder forces a reindex; having a single server makes that change atomic and auditable.)

It is the same argument for why nobody serves an LLM with transformers.generate() in production: you serve it with a dedicated engine. The embedder is no different.

Dynamic batching: the mechanics of the press

Here is the performance engine. TEI does token-based dynamic batching: instead of processing requests one at a time or grouping them by a fixed number of requests, it groups them by token budget (TEI README, Discussion #367).

Two knobs rule:

--max-batch-tokens (the default depends on the build, typically 16384): the total tokens that fit in a batch. With max-batch-tokens=1000 you get 10 requests of 100 tokens, or one of 1000. The docs say it clearly: this number should be as large as possible until the model becomes compute-bound. It is the size of the press tray, measured in tokens, not in parts, because the real cost scales with tokens, not with number of texts.
--max-concurrent-requests (default 512): how many requests it admits in flight before rejecting. It is the backpressure control, not the batching one: it protects against surges by returning an error instead of queuing indefinitely.

The policy: TEI keeps an internal queue. When there is work, it fills a batch up to the max-batch-tokens budget (or until there are no more requests waiting) and sends it to the model in one stroke. Requests that arrive while the previous batch is being processed wait in the queue and go into the next one. Watch out for a real operational detail: if max-batch-tokens is smaller than the model’s maximum input length, you can trigger a loop where a long request never fits into a batch (Issue #723); the token budget has to be at least as large as the longest sequence you accept.

The number: throughput with batching vs without batching

Let us model the press. Let $C$ be the fixed cost of a forward pass (loading weights, launching kernels) and $c$ the marginal cost per sequence inside the batch. For a batch of size $B$, the stroke time is roughly:

$$t(B) \approx C + c \cdot B$$

as long as the batch fits in the hardware (memory-bandwidth-bound, which is the normal regime of a small encoder on GPU). The throughput, sequences per second, is:

$$\text{throughput}(B) = \frac{B}{t(B)} = \frac{B}{C + c \cdot B}$$

Let us put in concrete numbers, indicative but of the right order for bge-m3 (568M) on a GPU serving ~256-token sequences. Say $C = 4$ ms of fixed cost per stroke and $c = 0.5$ ms per marginal sequence.

Without batching ($B=1$):

$$t(1) = 4 + 0.5 = 4.5 \text{ ms} \quad\Rightarrow\quad \text{throughput} = \frac{1}{4.5\text{ms}} \approx 222 \text{ sec/s}$$

With batching ($B=32$):

$$t(32) = 4 + 0.5 \times 32 = 20 \text{ ms} \quad\Rightarrow\quad \text{throughput} = \frac{32}{20\text{ms}} = 1600 \text{ sec/s}$$

The batch of 32 multiplies throughput by ~7.2× compared with serving one at a time, because the fixed cost $C$, which at $B=1$ was 89% of the time, is spread across 32. And the asymptotic limit, as $B \to \infty$, is $1/c = 2000$ sec/s: the press cannot go faster than its marginal cost, and we are already at 80% of that ceiling with $B=32$. Going up to $B=128$ would give ~1969 sec/s, a marginal improvement in exchange for a good deal more latency and memory. Batching has diminishing returns: most of the gain is in going from 1 to a few dozen.

The latency trade-off is the other face. The request that arrived first in the batch of 32 does not see its response until the whole stroke finishes: it waits for the tray to fill plus the 20 ms of the forward. If the 32 requests arrived within a window of, say, 5 ms, the first request suffers ~5 ms of batching wait + 20 ms of compute = 25 ms of latency, against the 4.5 ms it would have had being served alone. The p99 latency gets worse, that is the price of the full tray, but in exchange the system absorbs 7× more load with the same silicon. That is why the correct configuration depends on the use case: in batch ingestion (no user waiting) you maximise max-batch-tokens and latency does not matter; online (user waiting) you bound the wait window to keep p99 under control even at the cost of some throughput.

Deployment: compose for embedder and reranker

TEI is deployed as a container. The key arguments: --model-id (the model from the Hub), --pooling (how it aggregates tokens into the vector: cls, mean, splade, last-token), --dtype (precision), and the batching knobs already seen. Two separate services, since the embedder and the reranker are different models and different endpoints, each with its own container.

Embedder (`bge-m3`) on GPU

# docker-compose: TEI serving bge-m3 as an embedder, on GPU
services:
 tei-embed:
 image: ghcr.io/huggingface/text-embeddings-inference:cuda-1.9
 command:
 - --model-id=BAAI/bge-m3
 - --pooling=cls
 - --dtype=float16
 - --max-batch-tokens=16384 # big tray: the press performs
 - --max-concurrent-requests=512 # backpressure: reject above this
 ports: ["8081:80"]
 volumes: ["./hf-cache:/data"]
 deploy:
 resources:
 reservations:
 devices: [{driver: nvidia, count: 1, capabilities: [gpu]}]
 # exposes /v1/embeddings (OpenAI), /embed, /embed_sparse, /metrics

Reranker (`bge-reranker-v2-m3`) on GPU

 tei-rerank:
 image: ghcr.io/huggingface/text-embeddings-inference:cuda-1.9
 command:
 - --model-id=BAAI/bge-reranker-v2-m3  # cross-encoder, ~568M
 - --dtype=float16
 - --max-batch-tokens=16384
 - --max-concurrent-requests=256
 ports: ["8082:80"]
 volumes: ["./hf-cache:/data"]
 deploy:
 resources:
 reservations:
 devices: [{driver: nvidia, count: 1, capabilities: [gpu]}]
 # exposes /rerank — invoked ONLY over top-k (20/50), not over hundreds

bge-reranker-v2-m3 is a ~568M cross-encoder built on bge-m3, multilingual, that takes (query, document) and emits a relevance score directly, without producing a reusable vector (model card, BGE docs). It carries no --pooling because it emits no embedding: it emits a scalar.

CPU for batch ingestion

For the CPU fleet, it is enough to change the image and the dtype; the HTTP contract is identical:

 tei-embed-cpu:
 image: ghcr.io/huggingface/text-embeddings-inference:cpu-1.6
 command:
 - --model-id=BAAI/bge-m3
 - --pooling=cls
 - --dtype=float16  # on CPU, ONNX int8 if the export supports it
 - --max-batch-tokens=8192 # smaller tray: the CPU saturates sooner
 ports: ["8083:80"]
 volumes: ["./hf-cache:/data"]
 # ONNX/MKL backend: exploits AVX-512+VNNI / AMX on 4th-gen Xeon

Any OpenAI client embeds against the endpoint without knowing what is behind it:

from openai import OpenAI
client = OpenAI(base_url="http://tei-embed:80/v1", api_key="-") # api_key ignored
r = client.embeddings.create(model="BAAI/bge-m3", input=["user query"])
vec = r.data[0].embedding # same contract as OpenAI; the silicon is invisible

And the rerank, through its native (non-OpenAI) endpoint:

import requests
docs = ["retrieved chunk 1", "retrieved chunk 2", "...top-20 from retrieval..."]
r = requests.post("http://tei-rerank:80/rerank",
 json={"query": "user query", "texts": docs})
ranked = r.json() # [{index, score}, ...] ordered by relevance

Integration: how ingestion and the gateway consume it

The TEI server has two clients with opposite profiles, and that is precisely the reason to centralise it:

Ingestion (batch, no SLA) calls /v1/embeddings or /embed with large lots of chunks. Here throughput is what matters: ingestion pushes thousands of texts and TEI groups them into large batches against the high max-batch-tokens. It is the case of the press with a full tray. Ingestion then writes the vectors (dense + sparse) into the vector DB; the specific PDF-to-indexed-chunk pipeline is the sister piece of this series in preparation, and meanwhile the PostgreSQL + Qdrant schema in microservices covers the structure of the pipeline.

The gateway (query-time, online) consumes TEI through two different routes:

Embedding the query: a single call to /v1/embeddings with a short text. The gateway, LiteLLM or another L7 router, treats it as one more OpenAI embeddings provider, routed by model. That embedded query goes to the vector DB for the hybrid search.
Reranking the top-k: after retrieving 20-50 candidates from the retriever, the gateway calls /rerank on the reranker server with the query and the candidate list, and keeps the best ones. Since /rerank is not an OpenAI standard, LiteLLM exposes it through its own reranking contract (LiteLLM rerank with TEI).

The full query-time pattern: embed query (TEI dense) → search (vector DB, hybrid retrieval) → rerank top-k (TEI rerank) → send query + top-5 to the LLM. Two different TEI endpoints, one for dense and another for rerank, both behind the gateway, both swappable.

Sizing: memory, replicas and latency budget

Footprint of a 568M encoder

The weight footprint of bge-m3 (568M params) depends only on precision:

$$\text{fp16: } 568 \times 10^6 \times 2 \text{ B} \approx 1.14 \text{ GB} \qquad \text{int8: } 568 \times 10^6 \times 1 \text{ B} \approx 0.57 \text{ GB}$$

On top of this comes the space for the batch’s temporary activations (proportional to max-batch-tokens × hidden dimension) and, in TEI, the FlashAttention/cuda graphs structures. In practice, a bge-m3 in fp16 is served comfortably in ~6 GB of VRAM even at high batch, and in int8 on CPU the whole model (0.57 GB) fits in the cache and RAM of any server without blinking, activating the AMX/VNNI paths that make it viable, which is exactly the argument of RAG on the CPU.

How many replicas fit? If we reserve a 10 GB slice of an H100 for serving embeddings online, with an effective footprint of ~2.5 GB per replica (fp16 weights + activations + overhead), around 3-4 replicas of bge-m3 fit in that slice, splitting the QPS between them. In int8 the footprint drops and more fit, at the cost of a full point of quality. For the reranker, the weight footprint is the same (568M), but its batch pattern is different (query-doc pairs), so it is worth sizing it separately.

Latency budget: short query vs top-50 rerank

Here is the fundamental asymmetry between embedder and reranker, and it is what decides the sizing.

Embedding a short query (online): it is one forward over a text of tens of tokens. On GPU, inside a dynamic batch, it is tens of ms including the batching wait, the order of the earlier numerical example, ~5-25 ms. It is cheap and constant: a query is a query, no matter how many documents are in the corpus. That is why embedding online fits easily into an interactive latency budget.

Reranking the top-50 (online): a cross-encoder does not produce a reusable vector; it evaluates every pair (query, document) in a forward. The cost is linear in the number of candidates:

$$\text{cost}_{\text{rerank}} \propto k \times \text{forward}(\text{query} + \text{doc})$$

Reranking a top-50 is 50 forwards of (query + doc) pairs, where each pair is longer than the query alone (query + ~256-token chunk). TEI groups them into batches, and batching helps again there, but the total work is ~50× that of embedding one query. If embedding a query costs of the order of 10-20 ms, reranking 50 candidates costs of the order of hundreds of ms depending on the batch and the hardware. Hence the operational rule repeated throughout the series: broad, cheap recall in the retriever, precision rerank over FEW candidates. Reranking a top-20/50 fits in the online budget; reranking hundreds at high QPS does not, and that is where the reranker demands a dedicated GPU or a larger slice. The detail of why the cross-encoder is expensive but precise is in reranker and hybrid retrieval.

The query-time latency budget, added up:

Stage	Typical cost (order)	Scales with
Embed query (TEI dense)	~10-20 ms	constant (1 short text)
Hybrid search (vector DB, HNSW+sparse)	single-digit ms	index size (sublinear)
Rerank top-k (TEI cross-encoder)	~100-300 ms	linear in k
Generation (LLM, outside TEI)	seconds	output tokens

The reranker is the most expensive stage of the data plane, and the only one that scales badly with the width of retrieval. Size it as the critical component.

Applied to the generic 4×H100 cluster

Let us come down to the cluster of the series: 4×H100 SXM 80 GB plus a generic CPU fleet (Xeon with AMX, NUCs). TEI lives in two places depending on the load profile, and the identical OpenAI contract is what makes each place swappable.

TEI-CPU on the fleet → batch ingestion. Re-embedding the corpus is throughput-bound work with no latency SLA: its place is the CPU fleet with bge-m3 in int8 (ONNX/AMX). Here you maximise max-batch-tokens because nobody is waiting, and the press performs with the tray brimming. No H100 should spend a cycle reindexing a corpus that changes once a day, which is exactly the bad split that RAG on the CPU takes apart. The CPU fleet scales horizontally and cheaply; the corpus is chopped up and spread across nodes.

TEI-GPU in a MIG slice → high-QPS online embeddings and rerank. Query-time has a latency SLA and, if the system serves many requests per second, it needs the GPU’s throughput and low latency. But a 568M encoder does not deserve a whole H100: it takes ~2.5 GB effective, and leaving it 80 GB wastes 97% of the card. The right answer is a MIG slice: partition an H100 into hardware-isolated instances and give TEI one of the small slices (a 1g.10gb, for example), leaving the large slices for generation or for other tenants. The online embedder and reranker live there, with memory and compute isolation guaranteed by the partitioning. The how of the partitioning, MIG, MPS, time-slicing and when to use each, is in sharing a GPU; the reading for TEI is direct: MIG gives the hard isolation that an online service with bounded latency wants, while batch ingestion on CPU does not even touch the GPU.

The split, in one sentence: ingestion squeezes throughput/€ on CPU with a full tray; online squeezes latency/QPS in a MIG slice of an H100; both speak the same /v1/embeddings and /rerank, so moving load from one side to the other is changing a URL in the gateway.

If at some point you need a 7B embedder (gte-Qwen2, NV-Embed) whose quality bge-m3 cannot reach, that is no longer small-encoder work: it drags an LLM’s cost profile with it and is served where 7Bs are served, with vLLM --task embed in a large GPU slice, not with TEI-CPU. But it is the occasional exception, not the base load.

Conclusion

TEI is the stamping press of the RAG data plane: a specialised server that only stamps vectors and scores, and that performs far better when it fills the tray before pressing. The technical piece that gives the performance is token-based dynamic batching, controlled by max-batch-tokens and max-concurrent-requests, with a clear trade-off between throughput and p99 latency that is resolved differently in ingestion (tray brimming, no SLA) and online (bounded window, p99 protected). The architectural virtue is the OpenAI-compatible contract on /v1/embeddings, which makes the embedder swappable without touching code, plus the native /embed_sparse and /rerank endpoints for the sparse head of bge-m3 and for the bge-reranker-v2-m3 cross-encoder. The numbers drive the sizing: a 568M encoder fits in gigabytes, not tens of gigabytes, so several replicas live in a MIG slice; embedding a query is cheap and constant, but reranking is linear in candidates and is the critical stage of the latency budget. On the 4×H100 cluster, the right split is TEI-CPU for batch ingestion and TEI-GPU in a MIG slice for high-QPS online: two presses, two silicons, one contract.

References

Hugging Face — Text Embeddings Inference (TEI), repo y docs (Rust, backends CPU ONNX/MKL y CUDA, endpoints OpenAI-compatibles). https://github.com/huggingface/text-embeddings-inference · https://huggingface.co/docs/text-embeddings-inference
Hugging Face — TEI Quick Tour (endpoints /embed, /embed_sparse, /rerank, /predict, /v1/embeddings). https://huggingface.co/docs/text-embeddings-inference/quick_tour
Hugging Face — TEI CLI arguments (--model-id, --pooling, --dtype, --max-batch-tokens, --max-concurrent-requests). https://github.com/huggingface/text-embeddings-inference/blob/main/docs/source/en/cli_arguments.md
Hugging Face — TEI README / Discussion #367 (token-based dynamic batching; semántica de max-batch-tokens). https://github.com/huggingface/text-embeddings-inference/blob/main/README.md · https://github.com/huggingface/text-embeddings-inference/discussions/367
BAAI — BGE-M3 model card (568M, XLM-RoBERTa, dense+sparse+colbert, MIT). https://huggingface.co/BAAI/bge-m3
BAAI — bge-reranker-v2-m3 model card (cross-encoder ~568M sobre bge-m3, multilingüe). https://huggingface.co/BAAI/bge-reranker-v2-m3 · https://bge-model.com/bge/bge_reranker_v2.html
Chen, J., et al. BGE M3-Embedding. arXiv:2402.03216. https://arxiv.org/abs/2402.03216
BerriAI — LiteLLM: proveedor TEI para /rerank. https://github.com/BerriAI/litellm/issues/8372

End-to-end document ingestion: from PDF to indexed chunk

Thu, 11 Jun 2026 09:00:00 +0000

Fourth piece in an operational series about squeezing a generic on-premise LLM cluster of 4×H100 SXM 80 GB. Its siblings in this batch: serving embeddings and rerankers with TEI in production details the inference piece that this ingestion feeds; GitOps for the inference stack with Flux versions and deploys this whole pipeline; and hardening and secrets for the sovereign stack protects the corpus and the credentials this ingestion touches. The final consumer of what we build here, an end-to-end sovereign assistant, is assembled in a later instalment.

TL;DR

A RAG system inherits the quality of its corpus, and the corpus inherits the quality of the ingestion that manufactured it. This is the garbage-in/garbage-out of retrieval: a chunk badly extracted from a table, a page header repeated a thousand times, a scanned PDF from which you only got OCR noise; all of it gets embedded, indexed and reappears as poisoned context in the model’s answer. Document ingestion is not an afternoon’s script, it is a six-stage pipeline with engineering decisions at every one: (1) extract/parse, where almost everything is decided: layout-aware (Docling, which with Granite-Docling-258M preserves tables, formulas and structure, and which according to IBM avoids classic OCR and is up to ~30× faster) against plain text extraction (PyMuPDF), with OCR or a VLM for scans; (2) clean/normalise, removing boilerplate, normalising Unicode, rebuilding broken paragraphs; (3) chunk, and here 2026 has shifted the consensus: a February 2026 benchmark put recursive at 512 tokens in the lead (69% accuracy) ahead of semantic chunking (54%), and late chunking adds global context at no extra storage cost; (4) enrich metadata, with source, page, section, timestamp, ACL/tenant: for filtered retrieval, citation and auditability; (5) embed, through an embeddings server such as TEI; (6) index, in pgvector or Qdrant, with payload, dense + sparse. In between, exact deduplication (hash) and near-duplicate detection (MinHash/LSH or cosine with a threshold), because near-duplicates degrade retrieval recall and diversity. The numbers matter: a corpus of $N_{docs}\times$ pages $\times$ tokens/page gives the total tokens and, at a given CPU throughput, the ingestion time; and $N_{chunks}\times d\times\text{bytes}$ gives the index size, which in int8 falls 4× against fp32. The reading for the 4×H100: ingestion is CPU work, which links to the RAG-on-CPU piece; the GPU only comes in if you parse with a VLM (Granite-Docling) or use a 7B embedder.

The analogy: an archive’s cataloguing chain

Picture the cataloguing department of a large document archive. It is not one person putting papers into boxes; it is a chain of workstations, each with a different craft and its own quality standard. A document goes in raw at one end and comes out the other turned into a record you can find in seconds. If one station does its job badly, the ones downstream inherit the error and amplify it.

The first station is intake and reading. A heterogeneous box arrives: typed reports, crooked photocopies, folded tables, microfiche. An expert archivist really reads each item: distinguishes the body of the text from the margins, rebuilds a table that spans two pages, transcribes by hand what the scanner could not read. A novice archivist, by contrast, photocopies everything flat and hands over a slab of text where the table columns are interleaved into nonsense. That is exactly the difference between layout-aware parsing and plain text extraction, and it is where almost all of the quality is won or lost.

The second station is weeding out. Before filing anything, someone removes the duplicates (three copies of the same memo, two near-identical versions of a report) and cleans off the useless marks: “COPY” stamps, footers repeated on every sheet, coffee stains. If you do not weed, the archive fills up with copies that, when someone searches, return the same document six times and bury everything else. This is deduplication and cleaning.

The third station cuts it into records. A 400-page book is not catalogued as one giant record; it is broken down into manageable entries, by chapter, by section, of a size a reader can take in at a glance. Too big and the record mixes topics; too small and it loses context. This is chunking, and the size of the record is the decision that most shapes retrieval.

The fourth station labels. Each record carries a shelf mark, a date, its provenance, an access level (can anyone see this, or only the legal department?). Without those labels you cannot filter a search or say which document a statement came from. These are the metadata: source, page, section, timestamp, ACL.

The fifth and sixth stations place the record on the indexed shelf: they translate it into the catalogue’s language, a vector, and put it in the right drawer of the filing cabinet, so that a query finds the related records without walking through the whole thing. This is embedding and indexing in the vector store.

The moral runs through the whole post: RAG cannot retrieve better than what the cataloguing chain filed. If the first station chopped up a table badly, no reranker will fix it afterwards. Ingestion is the quality station for the entire system, and almost all of it, like the archive full of patient people working with nobody timing them, is background work that fits on CPU.

The six-stage pipeline

The temptation to treat ingestion as “read the PDF and chop it up” is the source of 80% of mediocre RAG systems. Each stage has a quality standard and a characteristic failure, and failures cascade downstream: if you parse badly, cleaning does not recover what was lost; if you chunk badly, embedding freezes the error into the vector; if you do not label, you will not be able to filter or cite. The rest of the post walks through the six stations and their decisions.

Stage 1 — Parsing: layout-aware vs plain text

Almost everything is won or lost here. A PDF is not text: it is a set of instructions for drawing glyphs on a canvas. “Extracting the text” from a PDF means reconstructing a reading order that the format does not guarantee, and tables, columns and figures break it systematically.

There are two philosophies, and the choice shapes the rest of the pipeline.

Plain text extraction. Tools such as PyMuPDF (fast, robust, no heavy dependencies) read the PDF’s text stream and dump it out. For single-column documents, running prose and no tables, it is perfect: extremely fast, faithful and cheap on CPU. Its limit shows up with structure: a two-column table comes out with the cells interleaved, a double-column document mixes the end of one with the start of the other, and a scanned invoice does not come out at all because there is no text layer. PyMuPDF can also segment by the table of contents (TOC) when the PDF carries one, which helps with chunking by section (Omdena, Document Parsing for RAG: A Complete Guide for 2026).

Layout-aware parsing. Tools such as Docling (an open source project driven by IBM Research) and unstructured.io first understand the layout, identifying titles, paragraphs, tables, figures, lists and formulas, and only then extract the content while respecting that structure. Docling captures table structure (rows, columns, multi-level headers) and, in its 2026 evolution, does so with a VLM: Granite-Docling-258M, released by IBM in January 2026 under Apache 2.0, a compact vision-language model (~258M parameters, Granite 3 backbone + SigLIP2 visual encoder) that converts pages, whether PDF, slides or scans, directly into a structured format called DocTags, preserving tables, code, inline and block mathematics, and the document’s hierarchy (IBM, Granite-Docling: End-to-end document understanding; model card on Hugging Face; Docling repo). IBM states that the VLM route avoids classic OCR and that this “reduces errors and speeds up the solution by up to 30×” against a traditional OCR pipeline. That is an IBM Research figure; I quote it and treat it as indicative, not as an independent benchmark reproduced here.

unstructured.io offers partitioning strategies graded by document complexity: fast (plain text, quick), hi_res (identifies the layout, recommended when classifying tables and elements properly matters), VLM and auto, balancing speed, cost and accuracy (Unstructured, PDF Parsing Strategies for RAG). The practical rule: fast for prose, hi_res or VLM for documents with tables and structure.

OCR for scans

When the document has no text layer (a scan, a photo, a digitised microfiche), you have to recognise the characters. Three routes:

Classic OCR (Tesseract, PaddleOCR, EasyOCR, which is what Docling integrates with when explicit OCR is needed). Mature, CPU-friendly, good with clean text; it struggles with tables, handwriting and complex layouts.
End-to-end VLM (Granite-Docling and similar). The model “looks” at the page and emits structure directly, with no separate OCR stage. Better with complex layouts; this is where the GPU does come in if the VLM is large or the volume is high.
Hybrid: OCR for character transcription, a layout model for the structure.

The honest criterion: for a corpus of native PDFs with text, PyMuPDF or unstructured fast handle it on CPU and cheaply. For a corpus with dense tables, forms or scans, layout-aware Docling/Granite-Docling earns its cost in chunk quality, and it is the only stage of the pipeline where the GPU can be justified.

Document case	Recommended tool	Silicon
Native PDF, single column, prose	PyMuPDF / unstructured `fast`	CPU
PDF with tables, double column, hierarchy	Docling / unstructured `hi_res`	CPU (layout models)
Scan, form, handwriting, complex layout	Granite-Docling (VLM) or OCR+layout	GPU if large VLM / high volume
HTML, DOCX, PPTX	Docling (multi-format) / native parsers	CPU

Stage 2 — Cleaning, normalising and deduplicating

Freshly parsed text comes in dirty. Cleaning means removing what adds nothing and normalising what is represented in a thousand ways:

Boilerplate: headers and footers repeated on every sheet, page numbers, watermarks, HTML navigation menus, cookie banners. If you do not remove them, they get embedded a thousand times over and contaminate both the index and the answers.
Unicode normalisation (NFC/NFKC), whitespace and dashes: the same character represented in several ways breaks exact matching and dirties the embeddings.
Paragraph reconstruction: undoing the hard line breaks the PDF inserted mid-sentence, without merging paragraphs that were meant to stay separate.

Deduplication: why it matters

RAG suffers two ills from duplicates. The exact kind, the same document uploaded three times, inflates the index and makes a search return the same answer repeatedly, wasting the top-k context slots. The near-duplicate kind, two near-identical versions of a report, a document and its draft, is worse: they look different to a hash but say the same thing, and they degrade retrieval recall and diversity. This is not theory: in the MS MARCO V2 collection a substantial overlap of near-duplicates has been documented which, left untreated, degrades retrieval precision and reduces the diversity of retrieved documents (Ragnarök / TREC RAG 2024).

Two levels of dedup, complementary:

Exact (hash). Compute a sha256 of the normalised content of each document (or chunk) and discard the matches. Cost $O(N)$, trivial. It catches byte-for-byte duplicates.
Near-dup (MinHash + LSH, or cosine of embeddings with a threshold). For those that differ slightly but mean the same thing. MinHash compresses each document into a compact signature such that the probability of two signatures matching at a position equals the Jaccard similarity of the original shingle sets; combined with Locality-Sensitive Hashing (LSH) it finds every near-duplicate pair without comparing all against all, reducing a quadratic problem to almost linear (Brenndoerfer, MinHash, Jaccard, LSH). It is the dominant technique in cleaning LLM training corpora (C4 and RefinedWeb use it) and applies just the same to a RAG corpus (Zilliz, Data Deduplication at Trillion Scale). The alternative, cosine of embeddings with a threshold (say > 0.97), catches semantic duplicates that MinHash cannot see (paraphrasing), but it requires you to already have the embeddings and is more expensive.

The practical rule: exact hash always (it is free); MinHash/LSH for large corpora with versions; cosine with a threshold if paraphrasing is a real problem. And deduplicate before embedding: re-embedding a duplicate means spending compute on rubbish you will then have to filter out.

Stage 3 — Chunking

Chunking is the decision that most shapes retrieval, and the one that carries the most myths. The trade-off is threefold: chunk size ↔ retrieval granularity ↔ context cost.

Large chunks: each one holds more context and there is less risk of cutting an idea in half, but the search is less precise (one vector represents too many topics) and, on retrieval, you put more tokens into the LLM’s prompt, which means more cost and more risk of diluting what is relevant.
Small chunks: very granular, precise retrieval, but each chunk loses context (a 43-token fragment may mean nothing outside its section) and you need to retrieve more of them to cover an answer.

The strategies, from least to most sophisticated:

Fixed size + overlap. Cut every $N$ tokens with an overlap of $k$ tokens between consecutive chunks so as not to slice a sentence dead. Simple, predictable, a reasonable baseline. The overlap is the insurance against cutting an idea right at the boundary.
Recursive (LangChain’s RecursiveCharacterTextSplitter). It tries to cut on separators in priority order (paragraph, then sentence, then word) to respect the natural structure as far as possible before falling back to a hard cut. It is the workhorse.
Semantic. Embed sentences and cut where the similarity between consecutive sentences falls below a threshold, grouping by coherence of meaning. It sounds better on paper; in 2026 practice it has disappointed (see below).
Structure/layout-aware (by headings). It exploits the hierarchy the layout-aware parser has already extracted: one chunk per section or subsection. unstructured offers the by_title strategy, which opens a new chunk when a title element appears, avoiding mixing text from different sections (Unstructured docs). If you parsed with Docling/hi_res, this strategy is almost free and is usually the best one for well-structured documents.
Late chunking. The twist of 2024–2026: instead of chunking and then embedding each chunk separately, embed the whole document first (with a long-context encoder) and then apply the chunk boundaries by mean-pooling the token embeddings inside each span. The result: every chunk keeps the document’s global context (a pronoun or a reference that only makes sense given the previous paragraph is encoded in the vector) and all of it with no extra storage cost, because you end up with one vector per chunk just as before (Jina AI, Late Chunking; arXiv:2409.04701).

What the 2026 benchmarks say (and why semantic disappoints)

It is worth being sceptical about fashion. A Vecta benchmark from February 2026 covering 7 strategies over 50 academic papers put recursive at 512 tokens in first place with 69% accuracy, while semantic chunking came in at 54%, partly because it produced tiny fragments, averaging 43 tokens, too small to mean anything (Firecrawl, Best Chunking Strategies for RAG in 2026). A systematic analysis from January 2026 also identified a “context cliff” around 2,500 tokens, where answer quality drops once you feed in contexts that are too long, an extra argument against giant chunks ([ibid.]). The honest reading: recursive at a moderate size with overlap remains the baseline that is hard to beat; late chunking is the improvement with the best cost/benefit ratio when the model supports it; semantic promises more than it delivers.

A worked numerical example of chunking

Take a technical document of 30 pages, with ~500 tokens of useful prose per page after cleaning (tables and figures are chunked separately). That is $30 \times 500 = 15{,}000$ tokens of text. We chunk with recursive at 512 tokens and 20% overlap ($0.20 \times 512 \approx 102$ tokens). The effective “step” between the start of one chunk and the next is:

$$\text{step} = \text{size} - \text{overlap} = 512 - 102 = 410 \text{ tokens}$$

The number of chunks in the document is then, approximately:

$$N_{chunks} \approx \left\lceil \frac{T_{doc} - \text{overlap}}{\text{step}} \right\rceil = \left\lceil \frac{15{,}000 - 102}{410} \right\rceil \approx \lceil 36.3 \rceil = 37 \text{ chunks}$$

Without overlap it would have been $\lceil 15{,}000 / 512 \rceil = 30$ chunks; the 20% overlap costs us 7 extra chunks (~23% more) in exchange for not splitting ideas at the boundaries. That is the concrete price of overlap: more vectors to embed, index and store, in exchange for robustness in retrieval. We will use this factor when sizing the full corpus.

Stage 4 — Enriching with metadata

A chunk with no metadata is a record with no shelf mark: it exists but it is no use. Each chunk gets a payload attached with, at a minimum:

Source and location: document_id, document name/URI, page number, section/heading (which the layout-aware parser already gave you). Essential for citation: being able to say “this comes from document X, page 12, section 3.2” is what separates an auditable RAG from one that hallucinates with no traceability.
Timestamp: when it was ingested and the document’s date. It lets you filter by freshness and detect stale content.
ACL / tenant: who is allowed to see this chunk. It is critical and it is applied as a filter at retrieval time: a user from department A must not retrieve chunks marked for B only. Without this, the RAG system is a data leak waiting to happen.
Embedding model version (model_version): so you know which embedder generated each vector and can migrate without mixing incompatible spaces.

These metadata are not decoration: they enable filtered retrieval (searching only in what the user can see, or only in documents after a given date), citation (reconstructing the origin of every statement) and auditability (knowing what was retrieved, from where and when). All of it lives in the point’s payload in the vector store.

Stages 5 and 6 — Embedding and indexing

The last two stations translate the chunk into a vector and place it on the shelf.

Embedding. The chunks are sent in batches to an embeddings server, typically Hugging Face’s TEI (Text Embeddings Inference), which exposes the OpenAI /v1/embeddings contract and runs on CPU or GPU. It is throughput-bound work with no latency SLA: its natural home is the CPU with a small encoder in int8 (the sibling piece in this batch, serving embeddings and rerankers with TEI, details the how). It is worth emitting dense + sparse at the same time: the dense vector captures semantics, the sparse one (SPLADE/BM25-like) exact lexical overlap, and together they make hybrid retrieval more robust.

Indexing. The vectors, with their payload, are upserted into the vector store. Two reference options, both current in 2026:

pgvector (a PostgreSQL extension). Its great virtue is living inside Postgres: ACID transactions, joins with the relational metadata, a single database to operate. Version 0.8 added halfvec (half-precision vectors, 2× less storage) and 0.9 (early 2026) added support for sparse vectors and speed improvements. Its known limit: it has no native int8 quantisation, so high-dimensional embeddings consume RAM linearly (Encore, pgvector vs Qdrant 2026; Katz, Scalar and binary quantization for pgvector).
Qdrant (a dedicated vector engine). It supports int8 scalar quantisation (float32 → int8, 4× less memory) and product quantisation, native sparse vectors and RRF fusion for hybrid search. It is more efficient in memory and in quantisation; the cost is operating one more system alongside Postgres (Markaicode, pgvector vs Qdrant 2026).

The practical rule: pgvector if you already have Postgres and the corpus fits comfortably in RAM (a single database to operate and back up); Qdrant if memory efficiency and int8 quantisation are critical because of the corpus size. Synchronisation between the relational truth (Postgres) and the index (Qdrant) when both are used is detailed in PostgreSQL + Qdrant in microservices.

A cross-cutting stage — Incremental ingestion and idempotency

A living corpus changes: documents are added, edited, deleted. Re-indexing everything each night is expensive and causes windows of unavailability. The alternative is incremental ingestion with two pillars:

Idempotency by doc-id + hash. Each chunk is identified deterministically ({doc_id}_{chunk_index}) and each document carries a content hash. On reprocessing, if the hash has not changed, nothing is re-embedded: the compute is saved. If it changed, the old chunks for that doc_id are deleted and the new ones are upserted. An upsert with a deterministic id is idempotent: reprocessing an event twice does not create duplicates.
CDC (Change Data Capture). Instead of polling, Debezium reads the PostgreSQL WAL and propagates additions, edits and deletions to the index in near real time. A deletion in Postgres triggers the deletion of that document’s chunks in the vector store, avoiding the “ghost documents” that contaminate retrieval. The deep dive is in PostgreSQL + Qdrant in microservices and in the one on Debezium and CDC.

The arithmetic: sizing the corpus and the index

Two calculations you need to be able to do before provisioning anything.

Corpus sizing and ingestion time

Suppose a corporate corpus of $N_{docs} = 50{,}000$ documents, averaging 20 pages and 500 useful tokens per page after cleaning. The total tokens in the corpus:

$$T_{corpus} = N_{docs} \times \text{pages} \times \text{tokens/page} = 50{,}000 \times 20 \times 500 = 5 \times 10^{8} \text{ tokens}$$

Five hundred million tokens. Applying the overlap factor from the chunking example (~1.23×, the 23% extra chunks from the 20% overlap) and an effective size of 512 tokens/chunk, the number of chunks is:

$$N_{chunks} \approx \frac{T_{corpus}}{\text{step}} = \frac{5 \times 10^{8}}{410} \approx 1.22 \times 10^{6} \text{ chunks}$$

That is, ~1.22 million chunks to embed. At a conservative CPU embedding throughput of 3,000 tok/s per Xeon server in int8, the same figure we use in the RAG-on-CPU piece, the ingestion time for the first full corpus on one box is:

$$t_{ingesta} = \frac{T_{corpus}}{\text{throughput}} = \frac{5 \times 10^{8}}{3{,}000} \approx 1.67 \times 10^{5} \text{ s} \approx 46 \text{ hours}$$

46 hours on a single box sounds bad, but ingestion is embarrassingly parallel: the corpus is shared out. With 8 CPU servers it drops to ~6 hours, comfortably within a weekend window for the initial load; and subsequent incremental ingestions (only what changed) take minutes. Layout-aware parsing adds its own cost, since Docling with a VLM is slower than PyMuPDF, but it is also batch and parallelises the same way.

Vector index size

Each vector has dimension $d = 1024$ (that of bge-m3). In float32 (4 bytes/dimension), each vector takes:

$$\text{bytes}_{fp32} = d \times 4 = 1024 \times 4 = 4096 \text{ B} = 4 \text{ KB}$$

For the 1.22 M chunks, the dense vectors alone in fp32:

$$\text{size}_{fp32} = N_{chunks} \times d \times 4 = 1.22 \times 10^{6} \times 4096 \text{ B} \approx 5.0 \text{ GB}$$

In int8 (1 byte/dimension), each vector takes 1 KB and the total falls 4×:

$$\text{size}_{int8} = N_{chunks} \times d \times 1 = 1.22 \times 10^{6} \times 1024 \text{ B} \approx 1.25 \text{ GB}$$

To this you must add the HNSW index (~1.2× the size of the vectors for $m=16$) and the payload (metadata + chunk text, ~500 B/chunk → ~0.6 GB). In round numbers:

Configuration	Vectors	HNSW (~1.2×)	Payload	Total
fp32	~5.0 GB	~6.0 GB	~0.6 GB	~11.6 GB
int8	~1.25 GB	~1.5 GB	~0.6 GB	~3.4 GB

The reading: a corpus of 50,000 documents fits in the RAM of a single node even in fp32, and in int8 (Qdrant) it fits with room to spare, which keeps search latency in single-digit milliseconds. int8 quantisation is almost always the balance point, saving 4× with a recall loss typically below 1%. (These numbers are order-of-magnitude, with the constants and assumptions stated; they are for sizing, not for pinning down an invoice.)

Applied to the generic 4×H100 cluster

Let us bring this down to the series’ cluster: 4×H100 SXM 80 GB plus a generic CPU fleet (Xeon with AMX, NUCs). The correct split for ingestion is almost all CPU, with the GPU as an occasional exception:

Cleaning, dedup, chunking, metadata, ingestion embedding and indexing → the CPU fleet. All of this is batch work, throughput-bound, with no latency SLA. It is exactly the “data plane” the RAG on CPU piece talks about: no H100 should spend a cycle chunking documents or building an HNSW index (which was always CPU by design). The nightly incremental re-indexing of a corpus that changes on the scale of hours is the textbook case of “CPU work in no hurry”.
The GPU only comes in at two points. First, in VLM parsing: if the corpus has scans, dense tables or forms and you choose Granite-Docling-258M or another vision-language model, that parsing can be accelerated on GPU, although at 258M parameters it is light and, at moderate volumes, runs on CPU without drama. Second, in the large embedder: if retrieval quality demands a 7B embedder (gte-Qwen2, NV-Embed) instead of bge-m3 (568M), that embedder is once again an LLM-class model and lives where the 7B models live, on the GPU.
The 4×H100 are reserved for generation. As throughout the series, the expensive and scarce silicon is kept for what is latency-bound, the LLM producing the answer, and at most for the peaks of VLM parsing or 7B embedding the CPU cannot absorb. To give a sense of the ceiling: a 4×H100 node serving bge-m3 through TEI runs at around ~2,000 chunks/s, against the thousands of tok/s of a Xeon in int8; but using the H100s for daily ingestion means spending the resource the whole organisation fights over on a job the CPU fleet does overnight without anyone missing it.

The sentence that sums up the split: ingestion is the archive’s cataloguing chain, and almost all of it is done by patient, cheap staff (CPU); the star writer (GPU) is only disturbed when there is a page no classic OCR can decipher.

Closing: quality is decided upstream

The recurring error in mediocre RAG is not in the reranker or the prompt: it is in an ingestion that parsed a table badly, did not weed out the duplicates, or chunked at a size that destroys context. Garbage-in, garbage-out: no downstream component fixes what ingestion spoiled upstream. Investing in the cataloguing chain, with layout-aware parsing where it is needed, real dedup, measured chunking, complete metadata, is what moves the needle on system quality the most, and almost all of it fits on the CPU fleet. The GPU, like the star writer, should only touch the corpus when its brain is genuinely required.

References

IBM — Granite-Docling: End-to-end document understanding (Granite-Docling-258M, Apache 2.0, DocTags, enero 2026). https://www.ibm.com/new/announcements/granite-docling-end-to-end-document-conversion
ibm-granite — granite-docling-258M model card (VLM ~258M, Granite 3 + SigLIP2). https://huggingface.co/ibm-granite/granite-docling-258M
Docling project — Docling: Get your documents ready for gen AI (layout, tablas, multi-formato). https://github.com/docling-project/docling
Unstructured — PDF Parsing Strategies for RAG (fast / hi_res / VLM / auto; chunking by_title). https://unstructured.io/blog/mastering-pdf-transformation-strategies-with-unstructured-part-2 · https://docs.unstructured.io/
Omdena — Document Parsing for RAG: A Complete Guide for 2026 (PyMuPDF, TOC, pipeline). https://www.omdena.com/blog/document-parsing-for-rag
Firecrawl — Best Chunking Strategies for RAG (and LLMs) in 2026 (benchmark Vecta feb-2026: recursive 512 → 69%, semántico 54%; context cliff ~2.500 tok). https://www.firecrawl.dev/blog/best-chunking-strategies-rag
Günther, M., et al. — Late Chunking: Contextual Chunk Embeddings Using Long-Context Embedding Models. arXiv 2409.04701. https://arxiv.org/abs/2409.04701 · Jina AI: https://jina.ai/news/late-chunking-in-long-context-embedding-models/
Brenndoerfer, M. — MinHash: Jaccard Similarity, LSH, and Near-Duplicate Detection. https://mbrenndoerfer.com/writing/minhash-algorithm-jaccard-similarity-lsh-deduplication
Zilliz — Data Deduplication at Trillion Scale (MinHash LSH dominante en limpieza de corpus). https://zilliz.com/blog/data-deduplication-at-trillion-scale-solve-the-biggest-bottleneck-of-llm-training
Ragnarök / TREC RAG 2024 — near-duplicados en MS MARCO V2 degradan recuperación y diversidad. https://arxiv.org/pdf/2406.16828
Encore — pgvector vs Qdrant in 2026 (halfvec, sparse, límites de cuantización de pgvector). https://encore.dev/articles/pgvector-vs-qdrant
Markaicode — pgvector vs Qdrant (2026 Benchmarks) (Qdrant int8 scalar quantization 4×). https://markaicode.com/vs/pgvector-vs-qdrant/
Katz, J. — Scalar and binary quantization for pgvector. https://jkatz05.com/post/postgres/pgvector-scalar-binary-quantization/
Hugging Face — Text Embeddings Inference (TEI), backends CPU/GPU, endpoints OpenAI-compatibles. https://github.com/huggingface/text-embeddings-inference

Taking RAG to the CPU: separating the data plane from the generation plane

Thu, 11 Jun 2026 03:20:00 +0000

Third piece in an operational series about squeezing a generic on-premise LLM cluster of 4×H100 SXM 80 GB. Its siblings: sharing one GPU between workloads (time-slicing, MPS, MIG) and serving several models on one GPU (swap + sleep) attack the split inside the GPU. This one attacks the split outside: which parts of RAG never need to touch the GPU at all. The closing piece of the series, the end-to-end sovereign assistant (fourth instalment, in preparation), assembles the complete system where these pieces fit together.

TL;DR

A RAG system is not one thing, it is three phases with opposing compute profiles, and putting them all on the GPU “because it is AI” is an allocation mistake. (1) Build/ingest, embedding the corpus and building the index, is batch, throughput-bound work with no latency SLA: its natural home is the CPU. (2) Query-time retrieval, embedding the query, HNSW search, RRF fusion, lightweight rerank, is mostly CPU, with caveats only for heavy reranking; vector search always ran on CPU, even in stacks sold as “GPU”. (3) Generation, the LLM producing the answer, is latency-bound, and there the GPU is irreplaceable: a 7B on CPU gives a time to first token of seconds, unacceptable for chat. The technical key to why (1) and (2) fit on CPU: the embedder is not an LLM. bge-m3 is around 568M parameters (an XLM-RoBERTa encoder), not 7B+; in int8 it occupies about 580 MB and activates fast integer compute paths (Intel AVX-512 + VNNI + AMX on 4th-gen Xeon and later; NEON SDOT/UDOT on ARM). The runtimes are ready: TEI with a CPU backend (same OpenAI API /v1/embeddings and /rerank), Qdrant’s fastembed (ONNX-CPU), bge-m3 in ONNX int8 with its three heads (dense/sparse/ColBERT). The Intel + Hugging Face blog post with Optimum Intel and fastRAG reports up to around 10× on indexing for BGE-large int8 on a 4th-gen Xeon (their benchmark figure, encoding-only; I quote it and qualify it below). The operational conclusion: separate the data plane (CPU) from the generation plane (GPU). In the 4×H100 cluster, no H100 should be spent re-indexing a corpus that changes once a day. That goes to the generic CPU fleet (Xeon AMX, NUCs), and the H100s are reserved for generation and, at most, for rerank peaks or large 7B embedders. What does not go down to CPU: interactive generation, massive reranking at high QPS, re-indexing with a strict real-time SLA, and 7B embedders (gte-Qwen2, NV-Embed).

The analogy: the library and the librarian

Picture a serious research library. There are three distinct jobs, done by different people, on different clocks.

The first is cataloguing. Boxes of new books arrive; someone opens them, classifies them, assigns a shelf mark, indexes them in the catalogue and puts them on the right shelf. It is patient background work, done at night or between other tasks. Nobody is standing over you with a stopwatch waiting for today’s batch to be catalogued: what matters is that tomorrow it is done and done well. It is pure throughput: how many books you catalogue per hour, not how long you take on one specific book. This is ingestion.

The second is handling an enquiry at the desk. A reader turns up and asks about a topic. The librarian goes to the catalogue, which already exists, finds half a dozen relevant shelf marks, fetches them and puts the books on the desk. It is fast, light, and consists of searching an index that already exists, not building it. This is retrieval.

The third is writing a reasoned report from those books. The reader, or an expert you commission, reads the six books, compares them, synthesises, and writes an argued answer with citations. This is slow, demands a very well-trained mind, and the reader is waiting: here there is a human stopwatch. This is generation, the LLM.

The moral is about staff allocation. You do not put your star writer, expensive, scarce, with a queue of people waiting for reports, to catalogue boxes of books at dawn. Cataloguing is done by a large, cheap team that works through the night without rushing. The star writer touches only what genuinely needs their mind: writing. In our system, the star writer is the H100, and cataloguing at dawn is corpus ingestion. Spending the H100 on re-indexing is exactly the error of putting the writer to work labelling boxes.

The rest of the post is, essentially, which parts of the library work the cheap CPU team can do (almost all of them) and which is irreducibly the GPU writer’s job (only the last one).

The three phases and their compute profiles

The confusion that GPU over-provisioning feeds on is treating “RAG” as a monolithic block that “uses AI, therefore goes on the GPU”. No. RAG is a data pipeline with a generative model plugged in at the end. The correct architectural boundary does not separate “what uses models” from “what does not”, because both sides use models, but throughput-bound from latency-bound, which is the same as separating the data plane from the generation plane.

Why ingestion fits on CPU: the embedder is not an LLM

The whole argument rests on a size asymmetry that gets overlooked. People hear “embeddings” and “generation” and put them in the same bag of “big models that need a GPU”. But the embedding encoder and the generative LLM are two orders of magnitude apart in parameters.

bge-m3, the reference multilingual embedder, is an XLM-RoBERTa of around 568M parameters (model card, paper arXiv:2402.03216). Its sibling reranker, bge-reranker-v2-m3, is built on the same base and lands at roughly the same 568M parameters (model card). Compare that with an entry-level generative LLM: a Llama 3.1 8B has around 14× more parameters, and the big production models sit at 70B+. A 568M encoder is, in compute budget terms, a different animal.

Two structural differences make that encoder comfortable on CPU:

It is an encoder, not an autoregressive decoder. It processes the whole sequence in a single forward pass and emits the vector. There is no token-by-token decode, no growing KV cache, none of the memory-bound generation phase that kills the CPU. It is a dense matrix pass and that is it.
It quantises to int8 with almost no loss. In int8, bge-m3 occupies on the order of 580 MB and, above all, activates the integer compute paths that a modern CPU executes quickly: matrix instructions such as Intel AMX (Advanced Matrix Extensions, 4th-generation Xeon and later), AVX-512 with VNNI (Vector Neural Network Instructions) on earlier Xeons, and NEON SDOT/UDOT on ARM. The quality loss from going FP32 to int8 on these models usually stays below 1% of retrieval recall, practically invisible (Intel + Hugging Face, CPU Optimized Embeddings).

Let us quantify the int8 size. For $P = 568 \times 10^6$ parameters at 1 byte each:

$$\text{size}_{\text{int8}} \approx 568 \times 10^6 \text{ params} \times 1 \text{ byte/param} \approx 568 \text{ MB}$$

In other words, the model fits in the cache and RAM of any server or NUC without blinking, and the bottleneck is integer compute, exactly what AMX/VNNI accelerate. There is nothing in this profile that asks for a GPU.

Runtimes that already do this effortlessly

Nothing needs inventing. The CPU ecosystem for the data plane is mature:

Text Embeddings Inference (TEI) from Hugging Face: a Rust server with CPU backends via ONNX Runtime (recommended) or Intel MKL, and OpenAI-compatible endpoints (/v1/embeddings) plus /rerank (TEI repo). That is, the data plane on CPU exposes exactly the same HTTP contract as a GPU server; the rest of the system never learns which silicon is behind it.
fastembed from Qdrant: a lightweight library that loads embedders on ONNX-CPU and generates dense, sparse and ColBERT vectors (fastembed repo). Designed from the start to run without a GPU.
bge-m3 in ONNX int8 with its three heads (dense / sparse-lexical / ColBERT multivector) exported and quantised, ready for ONNX Runtime CPU.

The Intel and Hugging Face figure that anchors viability: in their benchmark with Optimum Intel + fastRAG on a 4th-generation Xeon (8480+, 56 cores, 1 socket), the int8 variant of BGE-large reaches up to around 10× indexing throughput against FP32 (HF blog, Haystack/deepset). The small print needs reading, so I read it: that ~10× is encoding-only (tokenisation excluded), at sequence length 256, comparing int8 against FP32 on the same CPU. It is not “CPU 10× faster than GPU”, it is “int8 10× faster than FP32 on CPU”. It is still the relevant figure: it tells you that with quantisation the CPU goes from unworkable to perfectly useful for batch ingestion.

Viability table: CPU for each component?

This is the operational table. The column that matters is the caveat, because a bare “yes” or “no” lies.

Component	CPU viable?	Caveat
Chunking (slicing the corpus)	Yes, always	It is regex, parsing and windows; it never had anything to do with the GPU.
Ingest embedding `bge-m3` dense	Yes, its best case	Overnight batch, int8 + AMX/VNNI. This is exactly what the CPU shines at.
Sparse / SPLADE / BM25 head	Yes, CPU-native	Lexical work is pure inverted index; the GPU adds nothing here.
Building the HNSW index (Qdrant, pgvector)	Yes, always CPU	The HNSW graph build is CPU by design in these engines.
Query embedding (online)	Yes	A single short text; tens of ms on CPU, more than enough for chat.
Dense + sparse search + RRF	Yes	Vector search always ran on CPU, even in “GPU” stacks. RRF is sorting lists.
Cross-encoder reranker `bge-reranker-v2-m3` top-20/50	Yes, with care	A cross-encoder evaluates $k$ query-doc pairs: cost $\propto k$. Over 20-50 candidates it works; over hundreds at high QPS, no.
ColBERT late interaction	Marginal on CPU	The token-by-token matrix product of late interaction is heavy; viable at low volumes, suffers with QPS.
LLM generation	No, in practice	A 7B on CPU gives TTFT in seconds. Interactive latency = GPU.

Two rows deserve underlining because they dismantle myths.

“Vector search needs a GPU.” False from the start. The HNSW index, the navigable small-world graph used by Qdrant, pgvector with vector/halfvec, Milvus in CPU mode and almost everything else, was always built and traversed on CPU. Even the stacks advertised as “GPU-accelerated RAG” do the embedding on GPU but the ANN search stays on CPU in the vast majority of deployments; the GPU variants of the index (CAGRA and similar) are the expensive exception, not the norm, and are justified only with billions of vectors and extreme QPS. For a corporate corpus of millions of chunks, HNSW on CPU resolves in single-digit milliseconds.

“The reranker is a model, therefore GPU.” The bge-reranker-v2-m3 reranker is a cross-encoder of around 568M: it runs on CPU. The caveat is the number of pairs. A cross-encoder does not produce a reusable vector; it evaluates the (query, document) pair jointly, so its cost grows linearly with the number of candidates $k$:

$$\text{cost}_{\text{rerank}} \propto k \times \text{forward}(\text{query} + \text{doc})$$

Reranking the top-20 or top-50 coming out of hybrid retrieval is perfectly affordable on CPU. Reranking hundreds of candidates at high QPS is not: there the linear cost explodes and the GPU wins. The rule of thumb: broad cheap recall in the retriever, precision rerank over few candidates. (The detail of hybrid retrieval and reranking is in the fundamentals piece linked below.)

The numbers, with honest methodology

Here comes the part where a lot of people lie by omission. I am going to give throughput ranges, but they are literature ranges and orders of magnitude, not my own measurements on this hardware. Take them as such: the right decision does not depend on nailing the number, it depends on understanding the allocation.

For bge-m3 dense, sequence ≈256 tokens, embedding throughput moves roughly like this:

Platform	Dense throughput (order of magnitude)	Reading
High-end GPU (5090 fp16, TEI)	~12k tok/s+ (indicative)	The ceiling; expensive and scarce.
Large server CPU (Xeon ~56 cores, int8 ONNX)	low band of thousands tok/s	~1/5–1/10 of the GPU, but horizontally scalable and cheap.
Edge CPU / NUC (4-8 cores, int8)	tens to low hundreds tok/s	Enough for overnight ingestion of a local corpus.

The temptation is to read the second row as “CPU is 5-10× slower, discarded”. That is the wrong reading for ingestion. For batch work with no SLA, what rules is not absolute tok/s but throughput per euro and throughput per watt, and there the arithmetic changes sign.

Take a numerical example of allocation. Suppose a corpus of 2 million chunks of around 256 tokens that has to be re-indexed once a day (the corpus changes, the embeddings have to be redone). That is:

$$2 \times 10^6 \text{ chunks} \times 256 \text{ tok/chunk} \approx 5.1 \times 10^8 \text{ tokens}$$

At a conservative CPU throughput of, say, 3,000 tok/s per int8 Xeon server:

$$t_{\text{ingest}} \approx \frac{5.1 \times 10^8 \text{ tok}}{3000 \text{ tok/s}} \approx 1.7 \times 10^5 \text{ s} \approx 47 \text{ hours on a single server}$$

47 hours on one box sounds bad until you remember two things. First, this is embarrassingly parallel: the corpus is sliced and distributed; with 8 CPU servers it drops to around 6 hours, with 16 to around 3 hours, comfortably inside the overnight window. Second, and more importantly: that same work on the GPU blocks the GPU. If the H100 does 12k tok/s, it takes around 12 hours, but they are 12 hours of the H100, the resource the whole organisation fights over in order to generate. Spending the scarce, expensive resource on re-indexing a corpus that changes once a day is bad allocation, even if it is “faster”: you are optimising the wrong tok/s.

The mental rule: for ingestion, optimise throughput/€ and throughput/W; absolute tok/s belongs to the generation plane, where the human stopwatch really is running.

Decision tree: CPU or GPU for this piece?

Reference architecture (a): CPU-only

The first case is a node with no GPU: a NUC, an office Xeon, a sovereign edge server in a branch or in an air-gapped environment. The whole data plane lives there; generation is delegated to a remote GPU endpoint or done in batch with an SLM when latency is not pressing.

The stack:

TEI-CPU serving bge-m3 int8 with dense + sparse (same OpenAI contract /v1/embeddings, plus /rerank for the reranker).
Qdrant with a dense HNSW index plus sparse vectors, native RRF fusion.
Reranker bge-reranker-v2-m3 over the top-k (via TEI’s /rerank).
Gateway that orchestrates and, for generation, calls an external endpoint.

# docker-compose: full RAG data plane on CPU (no GPU)
services:
 tei-embed:
 image: ghcr.io/huggingface/text-embeddings-inference:cpu-latest
 command: ["--model-id", "BAAI/bge-m3", "--pooling", "cls", "--dtype", "int8"]
 ports: ["8081:80"]
 # ONNX/MKL backend: exploits AVX-512+VNNI / AMX if the Xeon supports it

 tei-rerank:
 image: ghcr.io/huggingface/text-embeddings-inference:cpu-latest
 command: ["--model-id", "BAAI/bge-reranker-v2-m3", "--dtype", "int8"]
 ports: ["8082:80"]
 # exposes /rerank — invoked ONLY over top-20/50, never over hundreds

 qdrant:
 image: qdrant/qdrant:latest
 ports: ["6333:6333"]
 volumes: ["./qdrant_storage:/qdrant/storage"]
 # HNSW dense + sparse vectors + RRF, all CPU

Hybrid search with RRF fusion in Qdrant (dense + sparse in a single query):

from qdrant_client import QdrantClient, models

client = QdrantClient(url="http://qdrant:6333")

# query embedding: dense and sparse from TEI-CPU (HTTP wiring omitted)
hits = client.query_points(
 collection_name="corpus",
 prefetch=[
 models.Prefetch(query=dense_vec, using="dense", limit=50),
 models.Prefetch(query=sparse_vec, using="sparse", limit=50),
 ],
 query=models.FusionQuery(fusion=models.Fusion.RRF), # native RRF
 limit=20,
).points
# -> then: POST /rerank (TEI) over these 20, keep the top-5
# -> then: the gateway sends query + top-5 to the GENERATION endpoint (GPU)

Generation, on this CPU-only node, leaves the node: the gateway builds the augmented prompt and sends it to a vLLM endpoint on the GPU cluster (or, if there is no interactive SLA, to an SLM on CPU in batch mode, accepting a TTFT of seconds). The entire data plane, everything above, runs without a single GPU.

Reference architecture (b): the recommended hybrid

This is the one I recommend for the general case with a GPU cluster available: data plane on CPU, generation plane on GPU, connected by OpenAI-compatible HTTP contracts so that each side is replaceable.

Generation server, minimal, on GPU:

# vLLM on the GPU cluster — generation ONLY
services:
 vllm-gen:
 image: vllm/vllm-openai:latest
 command: >
 --model meta-llama/Llama-3.1-8B-Instruct
 --dtype bfloat16 --max-model-len 8192
 --gpu-memory-utilization 0.85
 # exposes /v1/chat/completions — the gateway sends it query + the top-5 already retrieved
 deploy:
 resources:
 reservations:
 devices: [{driver: nvidia, count: 1, capabilities: [gpu]}]

The virtue of the design: because both sides speak the OpenAI contract over HTTP, the CPU data plane and the GPU generation plane scale separately and are replaceable. If tomorrow you want to move the rerank to GPU because QPS went up, you change one URL. If you want to add more CPU ingest nodes, you add them without touching generation. The whole stack is OSS and licence-clean: bge-m3 and bge-reranker-v2-m3 are MIT (bge-m3, reranker), Qdrant is Apache-2.0, TEI and vLLM are OSS.

Applied to the generic 4×H100 cluster

Let us bring this down to the cluster in the series: 4×H100 SXM 80 GB plus a generic CPU fleet (Xeon with AMX, NUCs). The correct allocation:

Build and indexing → CPU fleet. No H100 should spend a cycle re-embedding the corpus. That goes to the Xeon AMX machines (large servers, throughput in the thousands of tok/s in int8) or, for small local corpora, to the NUCs overnight. Overnight re-indexing of a corpus that changes once a day is the textbook case of “unhurried CPU work”.
The H100s → generation. All four cards are reserved for what only they do well: producing tokens at interactive latency. This is what the sibling pieces in the series, sharing a GPU and several models on one GPU, help to squeeze: once ingestion stops competing for the GPU, all the expensive silicon is free to generate and is shared better between models and tenants.
The H100s, at most, → rerank peaks or large embedders. If at some point you need a 7B embedder (gte-Qwen2, NV-Embed) for a domain where bge-m3 does not reach, or a massive rerank at a QPS the CPU cannot absorb, those peaks can indeed visit the GPU. But they are the occasional exception, not the base load.

The auditability angle: ENS / NIS2

There is a compliance argument that rarely gets mentioned and that the CPU/GPU split gives you almost for free.

A CPU-only node with no proprietary driver is easier to audit. There is no closed NVIDIA kernel stack, no CUDA and firmware versions to reconcile with the supply chain, no proprietary driver surface to document for an ENS or a NIS2 assessment. The whole data plane, chunking, embeddings, index, search, runs on OSS software on generic CPU with standard instructions. For a sovereign or classified environment, being able to say “the plane that touches the corpus depends on no proprietary binary” is a real argument, not marketing.

And there is a second auditability angle intrinsic to RAG done well: source traceability. A RAG system that retrieves identifiable chunks and cites them is auditable, since you can reconstruct which document each claim came from, as opposed to context-stuffing or the model’s opaque parametric knowledge, where there is no way to know where a fact came from. That traceability lives in the data plane (what was retrieved, from which source, with what score), precisely the plane we are putting on auditable CPU. The two arguments reinforce each other: the auditable silicon and the auditable chain of evidence are the same plane.

When NOT to take it to CPU

For honesty, and to avoid falling into the mirror image of the hype, the cases where CPU is not the answer:

Generation at interactive latency. The obvious case. A 7B on CPU gives TTFT in seconds: unacceptable for chat. If the user is waiting, generation goes to GPU. With no practical exceptions today.
Massive reranking at high QPS. A cross-encoder or ColBERT over hundreds of candidates, multiplied by many requests per second, saturates the CPU. The cost $\propto k \times \text{QPS}$ crosses the threshold where the GPU pays off. Keep CPU rerank bounded to top-20/50; if you need more breadth at higher QPS, move up to GPU.
Real-time re-indexing with a strict SLA. If the corpus changes continuously and freshness is measured in seconds (not hours), CPU throughput may not fit the window. There, ingest embedding may need a GPU, but note that this is rare: most corporate corpora change at the pace of hours or days, not seconds.
Large embedders (7B). bge-m3 (568M) is comfortable on CPU; a gte-Qwen2 or NV-Embed at 7B is LLM-class again and drags the same cost profile as generation. If your retrieval quality demands a 7B embedder, that embedder lives where 7Bs live: on the GPU.

The sentence that sums it all up: the CPU is the default home of the data plane; the GPU is the justified exception for what is latency-bound and massively online. Start by putting everything on CPU and move up to GPU only what proves it does not fit, not the other way round.

References

Chen, J., et al. BGE M3-Embedding: Multi-Lingual, Multi-Functionality, Multi-Granularity Text Embeddings. arXiv 2402.03216. https://arxiv.org/abs/2402.03216
BAAI — BGE-M3 model card (568M params, XLM-RoBERTa, 8192 tokens, MIT). https://huggingface.co/BAAI/bge-m3
BAAI — bge-reranker-v2-m3 model card (cross-encoder on top of bge-m3, ~568M). https://huggingface.co/BAAI/bge-reranker-v2-m3
Intel + Hugging Face — CPU Optimized Embeddings with Optimum Intel and fastRAG (~10× indexing for BGE-large int8, 4th-gen Xeon). https://huggingface.co/blog/intel-fast-embedding
deepset / Haystack — CPU-Optimized Embedding Models with fastRAG and Haystack. https://haystack.deepset.ai/blog/cpu-optimized-models-with-fastrag
Hugging Face — Text Embeddings Inference (TEI), CPU ONNX/MKL backends, OpenAI-compatible endpoints. https://github.com/huggingface/text-embeddings-inference
Qdrant — fastembed (ONNX-CPU, dense/sparse/ColBERT) and Hybrid Search with RRF. https://github.com/qdrant/fastembed · https://qdrant.tech/documentation/beginner-tutorials/hybrid-search-fastembed/

Serving several models on a single GPU: co-residency, model-swapping and sleep mode

Thu, 11 Jun 2026 02:40:00 +0000

This is the second piece of an operational series on squeezing a generic on-premise LLM cluster of 4×H100 SXM 80 GB with NVLink. Its sibling, Sharing a GPU: time-slicing, MPS and MIG, splits the GPU so that several processes can use it at once; this one splits the complementary problem: you have more models than VRAM and you need them to coexist in time, not just in space. The third, The RAG data plane on CPU, moves off the GPU everything that does not need to be there; and the end-to-end sovereign assistant (fourth instalment, in preparation) assembles all of this behind LibreChat and LiteLLM. Here we assume that VRAM is the scarce resource and that keeping everything loaded at once is not an option.

TL;DR

You have an agent LLM serving almost all the traffic, one or two rerankers for RAG, an alternate model (another family, another language, a fine-tune) and, now and then, a large model for hard tasks. Added up, they do not fit in the 80 GB of an H100. There are three ways to make them coexist, and the key to not confusing them is to understand that they are three different states of where the weights live:

Co-residency: several models loaded at the same time in HBM, if the sum of their VRAM budgets fits in the physical memory. It is the ideal when they fit: zero switching latency. The catch: each engine must cap its own VRAM; if two engines both believe they own the whole 80 GB, the second one to ask for memory dies with OOM. On a GPU shared by time-slicing, sharing time does not protect memory (the processes still compete for the same HBM) and this problem gets worse, as we see in the sibling piece.
Model-swapping (with llama-swap): only one (or very few) resident at a time. The proxy looks at the request’s model field and, if another one is needed, unloads the current model and loads the requested one. Since there is only one inside, each model can use almost the whole GPU. The cost is a full cold start: seconds from local NVMe, minutes from network storage. It mixes engines: it exposes OpenAI-compatible endpoints including /rerank with llama-server on GGUF.
vLLM sleep mode (--enable-sleep-mode, endpoints /sleep and /wake_up): the model is not unloaded, it is put to sleep. The weights are parked in host RAM (level 1) or discarded while the process stays alive (level 2). The wake is 18–200× faster than a cold start because the process is still alive and keeps the CUDA allocator, the CUDA graphs and the compiled JIT kernels. The only thing rebuilt is the KV cache, which is discarded on sleep. It needs enough RAM for the sleeping weights.

The decision tree is short. Do they all fit? Co-residency. They do not fit, switching is infrequent and you mix engines? llama-swap. Everything is vLLM, there is RAM to spare and wake latency matters? Sleep mode. Sustained load on all of them at once? That is not swapping, it is more GPUs or replicas (see one big vs N small). This post puts numbers on each of them on an 80 GB H100.

The analogy: one stage, one spotlight

Picture a theatre with a single stage and a single spotlight, the GPU. You have more actors (models) in the company than fit under the light at once. There are three ways to stage the show:

Co-residency: several small actors on stage. If the cast for this scene is three slight actors, the main LLM, one reranker, another reranker, they all fit under the spotlight at the same time. Switching lines between them is instantaneous: they are already on stage. But the stage has a fixed size; add a fourth, burly actor (the large model) and they do not fit, so someone falls off the edge. That is the OOM.
Model-swapping: the actor goes home. When an actor finishes their part, they go home (the disk). If the next scene needs them again, they have to come back from home: get dressed, put the make-up on, travel to the theatre. That takes minutes. In exchange, while they are on stage they have the whole stage to themselves. That is the llama-swap model: maximum VRAM per actor, but re-entry costs a full round trip.
Sleep mode: the actor waits in the wings. Instead of going home, the actor steps out of the spotlight but stays backstage (host RAM), already dressed and made up. When their turn comes, they step in in two strides. No journey, no wardrobe: just crossing the curtain. That is the vLLM wake: the weights come back not from disk but from RAM, and everything that goes into “preparing the actor”, the allocator, the CUDA graphs, the kernels, is already done because they never left the building.

The operational moral is the same as in the theatre: the cost of an actor is not only their talent, it is how long they take to walk on when you need them. Co-residency pays for the permanent spot; swapping pays for the journey; sleep pays for the RAM of keeping them waiting nearby. The rest of this post is about choosing which, with a VRAM budget and a stopwatch in hand.

The three states: where the weights live

Co-residency: the VRAM budget

Co-residing is the default option when they fit: zero switching latency because every model is already in HBM. The whole question is do they fit?, and it is answered with a VRAM budget. The VRAM of an inference engine splits, roughly, into three items:

$$\text{VRAM}_{\text{model}} = \underbrace{P \cdot b}_{\text{weights}} + \underbrace{\text{KV}}_{\text{cache}} + \underbrace{A}_{\text{activations + overhead}}$$

where $P$ is the number of parameters, $b$ the bytes per parameter given the quantisation, KV the KV-cache budget (which scales with concurrency and context length) and $A$ the overhead of activations, CUDA buffers and fragmentation. The hard rule of co-residency is:

$$\sum_i \text{VRAM}_{\text{model } i} + \text{headroom} \;<\; \text{VRAM}_{\text{physical}}$$

If the sum overflows, no sharing scheme saves you: the first engine to ask for memory beyond the free gap dies with OOM. And here is the critical nuance that connects to the sibling piece: splitting the GPU by time does not split the memory. Under time-slicing, two processes take turns on compute but share the whole HBM with no isolation; if both assume they own the 80 GB, they collide. Only MIG gives memory partitions with a real boundary, as we see in Sharing a GPU. To co-reside properly, each engine must cap itself: in vLLM, --gpu-memory-utilization 0.45 tells it “do not use more than 45 % of the GPU”; in llama-server, you control the layers on GPU and the KV size. If you do not set those limits, vLLM claims 90 % of the GPU for itself by default, and there is no room left for anyone else.

A worked example: what fits in 80 GB

Take the realistic scenario. A 32B agent LLM in FP8 ($b = 1$ byte/param):

$$P \cdot b = 32 \times 10^9 \cdot 1 \text{ B} = 32 \text{ GB of weights}$$

On top of that comes the KV cache. To serve with decent concurrency and agent contexts (which are long: history, tools, documents), a KV budget of ~12 GB is reasonable, plus some ~2 GB of activations and overhead. Total for the main LLM: ~46 GB. That leaves ~34 GB of the 80. Let us see what fits there:

Service	Size (FP8/INT8)	KV + overhead	Total VRAM
32B agent LLM	32 GB	~14 GB	~46 GB
Reranker A (cross-encoder ~0.5B)	~0.5 GB	~0.5 GB	~1 GB
Reranker B (cross-encoder ~0.5B)	~0.5 GB	~0.5 GB	~1 GB
8B alternate model	~8 GB	~4 GB	~12 GB
Sum			~60 GB
Free headroom			~20 GB

With this they co-reside comfortably: the 32B LLM, the two rerankers (which are featherweights, a few hundred MB to 1 GB each) and even an 8B alternate, leaving 20 GB of cushion. Rerankers are the textbook case for co-residency: so small that they always fit alongside the main LLM, and rotating them would be absurd. Switching between them is instantaneous because they are all loaded.

When does it break? When the occasional large model arrives, a 70B in FP8:

$$70 \times 10^9 \cdot 1 \text{ B} = 70 \text{ GB of weights}$$

The 70B’s weights alone already eat 70 of the 80 GB. There is no way to co-reside it with the 32B, not remotely. That is where co-residency ends and rotation begins: the 70B can only come in if we evict the 32B from the GPU. That is the exact boundary: co-reside while the sum of budgets fits with headroom; rotate as soon as one model needs, on its own, more VRAM than is left over.

Model-swapping with llama-swap: the actor goes home

When they do not fit at once, the first answer is to rotate: keep one resident model and change it on demand. The canonical tool for this in the on-premise world is llama-swap, a Go proxy that sits in front of your inference servers (llama.cpp, vLLM, TabbyAPI and so on) and starts and stops them as needed.

The mechanism is elegant in its simplicity. Every OpenAI-compatible request carries a model field. llama-swap reads that field, looks at which upstream server it has configured for that model, and:

If the requested model is already loaded, it routes the request straight through.
If another one is loaded, it stops it (freeing its VRAM) and starts the right one.
When the new server reports ready, it forwards the request.

The cost of a swap is exactly a full cold start of the new model: unloading the current one (fast, just freeing memory) plus loading the new one (slow, moving the weights from disk to HBM). That second term is the painful one, and its magnitude is decided by where the weights are: from local NVMe it is seconds; from network storage (Ceph RGW, NFS) it can be minutes. The whole analysis of the load path, and why the default loader makes it slow, is in From disk to HBM; here the consequence is enough: a swap is only viable if it is infrequent, because every switch pays that toll in full.

The advantage in exchange: since there is only one resident, that model can use almost the whole GPU. The 70B that co-resides with nobody fits comfortably if it is the only one inside. And llama-swap mixes engines: you can have a vLLM configured for the big LLM, a llama-server with GGUF for the alternate, and two llama-server instances for the rerankers, which expose /rerank, /v1/rerank and /v1/reranking natively. That heterogeneity (GGUF + rerank + vLLM behind a single OpenAI endpoint) is exactly what vLLM alone does not give you, and the main reason to choose llama-swap.

Example ConfigMap: two rerankers rotating on the same port

A concrete and useful case: you have two rerankers, one multilingual and one specialised in code, that you do not need at the same time and would rather not keep both resident. With llama-swap they rotate on the same endpoint, triggered by the model field. The ConfigMap (mounted as the proxy’s config.yaml in a Kubernetes deployment) would be:

apiVersion: v1
kind: ConfigMap
metadata:
 name: llama-swap-rerankers
data:
 config.yaml: |
 # How long a model stays loaded after the last request
 # before llama-swap unloads it to free VRAM
 healthCheckTimeout: 60

 models:
 # Multilingual reranker (GGUF, via llama-server)
 "reranker-multilang":
 cmd: >
 /usr/bin/llama-server
 --model /models/bge-reranker-v2-m3.Q8_0.gguf
 --reranking
 --port ${PORT}
 --n-gpu-layers 99
 --ctx-size 8192
 # ttl: after 300 s idle, it is unloaded and the GPU is freed
 ttl: 300

 # Code reranker (GGUF, via llama-server)
 "reranker-code":
 cmd: >
 /usr/bin/llama-server
 --model /models/codereranker.Q8_0.gguf
 --reranking
 --port ${PORT}
 --n-gpu-layers 99
 --ctx-size 8192
 ttl: 300

A request to /v1/rerank with "model": "reranker-multilang" starts that server; the next one with "model": "reranker-code" stops the multilingual one and starts the code one on the same ${PORT} that llama-swap manages. Since both are small, the swap between them takes one or two seconds: quantised GGUFs weigh a few hundred MB and come from local NVMe. The ttl controls how long a model stays loaded after its last request: raising it avoids swaps if requests arrive in bursts; lowering it frees the GPU sooner for other uses.

An honest caveat: this pattern of two rerankers that rotate makes sense when they do not fit alongside the rest, or when you want to reserve the VRAM for something else. If they do fit (and a couple of 0.5 GB rerankers almost always fit, as we saw above), co-residing them is strictly better: zero swap latency. llama-swap shines when you rotate large models or mix engines that do not coexist well, not for juggling tiny models that would fit together.

vLLM sleep mode: the actor in the wings

Swapping has a problem: every switch pays the entire cold start, and a cold start is not just moving weights. As we saw in From disk to HBM, starting an inference engine includes initialising the Python process and the CUDA context, setting up the memory allocator, capturing the CUDA graphs and compiling the JIT kernels (DeepGEMM, FlashInfer, TorchInductor). Moving the weights is only one of five items, and often not the largest.

vLLM’s sleep mode (--enable-sleep-mode) attacks exactly that: instead of killing the process when a model stops being used, it puts it to sleep, leaving the process alive. There are two levels, and the difference is where the weights go:

Level 1: offloads the weights to host RAM (CPU) and discards the KV cache. The process stays alive. The wake copies the weights from RAM to VRAM, not from disk. Typical wake: ~0.1–0.8 s for small models, ~3–6 s for large ones. It needs enough RAM for the sleeping weights (on the order of the model’s size in GB).
Level 2: discards the weights entirely, keeping only small buffers (rope scaling tensors and the like). The wake does reload the weights from disk, but everything else, process, allocator, CUDA graphs, JIT kernels, is already done. Typical wake: ~0.8–2.6 s for small models. RAM use is almost nil (megabytes).

The key that explains the numbers: at both levels, keeping the process alive preserves the expensive infrastructure. That is why the benchmark in the vLLM blog (Oct 2025) reports that a wake is 18–200× faster than a full reload, and, most counter-intuitively, level 2 is still 23–45× faster than a cold start despite reloading the weights from the same disk, because it skips the other four items. In their measurements on an A100, a full cycle of 5 model switches goes from 357 s without sleep (≈48 s per switch) to 112 s with level 1 (wake of 0.26 s / 0.82 s) or 125 s with level 2 (0.85 s / 2.58 s).

The KV cache is always discarded on sleep. That is not a minor detail: it means the first response after the wake rebuilds the KV from scratch, paying a full prefill. That is why the wake is not “entirely free”: the model is available in sub-second time, but the first request is a little slower until the KV is repopulated. A one-request warm-up after the wake hides that cost.

# Start vLLM with sleep mode (admin endpoints, trusted networks only)
export VLLM_SERVER_DEV_MODE=1
vllm serve <model> --enable-sleep-mode --port 8001

# Sleep (level 1: weights to host RAM)
curl -X POST 'localhost:8001/sleep?level=1'

# Wake up
curl -X POST 'localhost:8001/wake_up'

Security warning (from vLLM’s own blog): the /sleep, /wake_up, /collective_rpc and /reset_prefix_cache endpoints require VLLM_SERVER_DEV_MODE=1 and should only be exposed on trusted networks, since they can take the service down. They are for internal orchestration (a controller that sleeps and wakes models according to the queue), not for the public plane.

The latency maths: why the wake wins

Let us put numbers on “the wake comes from RAM, not from disk”. The cost of having a model available is, in each strategy, the time to move its weights from wherever they are to HBM, plus, in the swap case, the other four cold-start costs. Take the 34 GB of weights of the 32B LLM in FP8 and compare the three paths.

Cold start from NVMe (swap). A reasonable Gen4/Gen5 NVMe gives on the order of ~5 GB/s effective per stream with the default loader (the disk’s theoretical floor is higher, but single-threaded deserialisation does not saturate it, see From disk to HBM). For 34 GB:

$$t_{\text{NVMe→HBM}} \approx \frac{34 \text{ GB}}{5 \text{ GB/s}} \approx 6.8 \text{ s just moving bytes}$$

And that is before adding CUDA graph capture and kernel compilation, which add several more seconds. The real cold start of a 32B is around 15–40 s depending on loader and storage. From the network (Ceph RGW), multiply.

Level 1 wake from RAM. The weights come not from disk but from host RAM, and travel over PCIe Gen5 x16, whose practical host→GPU bandwidth is ~50–64 GB/s. For the same 34 GB:

$$t_{\text{RAM→HBM}} \approx \frac{34 \text{ GB}}{55 \text{ GB/s}} \approx 0.62 \text{ s}$$

And there is nothing else to pay: the allocator, the graphs and the kernels are already there. The real wake of a model this size lands in the sub-second to a few seconds range that vLLM reports. The speed-up against an NVMe cold start is on the order of:

$$\frac{t_{\text{cold start}}}{t_{\text{wake}}} \approx \frac{15\text{–}40 \text{ s}}{0.6\text{–}3 \text{ s}} \approx 10\text{–}60\times$$

consistent with the 18–200× from the blog, whose measurements include smaller models, where the relative weight of the preserved CUDA graphs is even larger and the factor goes up.

Why the bandwidth difference explains almost everything. The key jump is not 5 vs 55 GB/s (a ~11× in transport). It is that the cold start also pays for rebuilding infrastructure, which the wake skips entirely. The table:

Path	Weight source	Bandwidth	34 GB bytes only	+ CUDA graphs / JIT	Realistic total
Cold start (swap) NVMe	disk	~5 GB/s	~6.8 s	yes (several s)	15–40 s
Cold start (swap) network	network	~1–2 GB/s	17–34 s	yes	30 s – min
Level 1 wake	host RAM	~50–64 GB/s	~0.6 s	no (preserved)	0.6–3 s
Level 2 wake	disk	~5 GB/s	~6.8 s	no (preserved)	7–10 s

Look at the level 2 row: it reloads the weights from the same disk as the swap (~6.8 s of bytes), but since it does not rebuild graphs or kernels, its total (~7–10 s) still beats a full cold start (15–40 s). It is the proof that moving bytes is only part of the cost, and the part sleep mode exploits.

Assumptions, honestly: the bandwidths are indicative. NVMe “5 GB/s effective” assumes the default loader; with a concurrent streamer it goes up. PCIe “55 GB/s” assumes Gen5 x16 with a pinned buffer and NUMA locality; if the buffer lands on the wrong socket, it drops. And the “15–40 s” cold-start range depends on the model, the quantisation and whether the files are in page cache (the “the second time it loaded fast” trap). The numbers are there to reason about orders of magnitude, not to size anything without measuring on your hardware.

The decision tree

The three strategies do not compete: each wins in a different regime. The tree, in order:

The node most often ignored is the one at the top right: “sustained load on all of them at once?”. If your four models get constant and simultaneous traffic, neither swapping nor sleep helps, since both assume the models take turns in time. Rotating under sustained load only adds switching latency without solving the underlying problem: there is not enough compute. The answer then is to scale horizontally (more replicas) or spread across more GPUs, a capacity decision analysed in One big vs N small. Swapping and sleep are tools for temporally unbalanced workloads: many models, but rarely active at the same time.

Applied to the generic 4×H100 cluster

Let us bring this down to the 4 H100 SXM 80 GB with NVLink. The winning strategy is not to pick one of the three, but to spread the models across GPUs according to their usage pattern and apply to each GPU the strategy it calls for. A reasonable split:

H100 #0, the workhorse (co-residency). The 32B agent LLM (main service, constant traffic) co-resides with the two rerankers and, if they fit, the 8B alternate. It is the GPU that never rotates: everything living here is used continuously and fits comfortably in 80 GB (the ~60 GB of the example above). Zero switching latency between the LLM and its rerankers, which is exactly what RAG needs, since a fast reranker is useless if you have to wait for a swap every time.

H100 #1, light services with MIG. If you have many small heterogeneous services, an embedding model, a classifier, a guardrail, an STT/TTS, splitting this GPU with MIG into isolated instances (each with its slice of HBM with a real boundary) gives co-residency with memory isolation, preventing a service that inflates its KV from taking the others down. The detail of when MIG beats time-slicing is in Sharing a GPU; the rule here: co-residing light services on one GPU makes sense when they fit and when isolating them is worthwhile, and MIG is the tool for the second part.

H100 #2, the occasional large model (sleep mode or swap). The 70B that is only invoked for hard tasks does not deserve a dedicated GPU kept awake, since it would sit idle most of the time, burning 700 W for nothing. Two options:

If this GPU also serves a medium model routinely and you only occasionally need the 70B, use sleep mode: put the medium one to sleep (level 1, weights to RAM), wake the 70B. Note that level 1 sleep frees the VRAM, the weights go to RAM, so yes, it fits even though the 70B would not fit alongside an awake medium model. Waking the medium one again is sub-second.
If the 70B comes as GGUF or you mix engines, llama-swap rotates between the medium model and the 70B by the model field. Each 70B invocation pays its cold start (seconds from local NVMe), acceptable if it is occasional.

H100 #3, replica / overflow. The fourth GPU absorbs peaks: a replica of the main LLM for when #0’s queue grows, or reserve capacity. There is no swapping here: it is pure capacity, the answer to the tree’s “sustained load” node.

The cross-cutting principle: co-reside what is used together and continuously (LLM + rerankers); isolate the light and heterogeneous with MIG; sleep or rotate the large and occasional; replicate what saturates. The four GPUs do not all do the same thing, each runs the strategy its load pattern calls for. And the NVLink between them matters for something else (tensor parallelism for the 70B if it would not fit even on one; see One big vs N small), but for this post’s problem, many models, one GPU, the lever is when each model needs to be awake.

Traps and things that are not what they seem

“Co-residing is always better if they fit.” Almost, but watch the KV cache: co-residing two models means splitting the KV budget between them. If the main LLM needs a large KV for high concurrency and long contexts, adding a flatmate cuts that KV and lowers its throughput. Sometimes it is better to give the whole GPU to the main model and rotate the secondary one. Co-residency is not free: the tenant takes room away from the cache of the one that matters.

“Sleep mode is like swapping but faster.” Not exactly. Swapping releases the process; you can have N models configured and only pay RAM/disk for the resident one. Sleep mode keeps one live process per sleeping model, so each sleeping vLLM still occupies its process slot, its RAM (level 1) and its management footprint. Sleep scales well to a few rotating models; for many (10+), level 2 (minimal RAM) or plain swapping fit better. Do not put 15 models into level 1 sleep and expect the RAM to hold.

“The wake is instantaneous, I lose nothing.” The model’s wake is sub-second, but the KV cache was discarded on sleep. The first request after the wake pays a full prefill to repopulate the KV, slower than normal. If your SLA is strict on the first response after an idle period, add an automatic warm-up after the wake. Prefix caching helps make that reprefill cheaper if there are stable prefixes.

“llama-swap with a low ttl saves me VRAM for free.” It saves you VRAM while nobody uses that model, but every time it comes back it pays the cold start. An aggressive ttl on a model with bursty traffic turns every burst into a load wait. The right ttl depends on the temporal pattern of the requests, not on how much VRAM you want to free. Measure it.

“Time-slicing lets me co-reside more models.” False and dangerous. Time-slicing shares out compute time, not memory, and all the processes still compete for the same HBM with no isolation. Co-residing via time-slicing does not give you more effective VRAM; it gives you more processes fighting over the same VRAM, and an OOM when the sum overflows. For real memory partitioning, MIG. The detail is in the sibling piece.

Conclusion

Having more models than VRAM is not a problem of insufficient hardware: it is a problem of managing a scarce resource over time. The intuition of “I need one GPU per model” is expensive and almost always wrong, because models are rarely used all at once. The three strategies are three answers to the same question, where do the weights of the models you are not currently using live: in HBM if they fit (co-residency, zero latency but they cost space), on disk if switching is rare (swapping, maximum VRAM per model but a return trip of seconds to minutes), or in RAM if switching latency matters (sleep mode, sub-second wake at the cost of tied-up RAM). Sleep mode is the most interesting addition of 2025 because it breaks the false dilemma of “everything loaded vs reload every time”: by keeping the process alive and preserving the allocator, the CUDA graphs and the kernels, it turns a 30–100 s cold start into a sub-second wake, and it does so even when it reloads the weights from the same disk (level 2), because moving bytes was never the whole cost. On the four-H100 cluster, the play is not to pick one strategy but to spread: co-reside what goes together, isolate the light stuff with MIG, sleep or rotate the large and occasional, replicate what saturates. The GPU is the stage with a single spotlight; the art is knowing which actor walks on, which goes home and which waits in the wings.

References

vLLM Blog (Embedded LLM), Zero-Reload Model Switching with vLLM Sleep Mode, 26 oct 2025: https://blog.vllm.ai/2025/10/26/sleep-mode.html
vLLM Docs, Sleep Mode: https://docs.vllm.ai/en/latest/features/sleep_mode/
mostlygeek, llama-swap (OpenAI/Anthropic-compatible model-swapping proxy): https://github.com/mostlygeek/llama-swap
llama-swap, Configuration: https://github.com/mostlygeek/llama-swap/blob/main/docs/configuration.md
NVIDIA, H100 Tensor Core GPU (HBM3 specs, 80 GB, ~3.35 TB/s): https://www.nvidia.com/en-us/data-center/h100/
NVIDIA, Reducing Cold Start Latency for LLM Inference with NVIDIA Run:ai Model Streamer: https://developer.nvidia.com/blog/reducing-cold-start-latency-for-llm-inference-with-nvidia-runai-model-streamer/

Sharing one GPU between several workloads: time-slicing, MPS and MIG

Thu, 11 Jun 2026 02:00:00 +0000

This post opens an operational series on how to squeeze a generic on-premise 4×H100 SXM LLM cluster. The sibling pieces: Serving several models on one GPU: swap and sleep (what to do when the models do not fit at the same time and have to take turns in memory), RAG on CPU: separating the data plane from generation (moving retrieval off the GPU to free it up) and Sovereign end-to-end assistant with LibreChat, LiteLLM and RAG, the final assembly, a fourth instalment in preparation. Here we start with the most basic question: you have one GPU and you want to put several workloads on it. How do you split it?

TL;DR

You have one GPU, or a few, and several workloads that want to run on it: a chat model, an embeddings service, a reranker, a queue of dev jobs. The GPU is underused if it only runs one thing, but piling several on carelessly causes contention, OOM or cascading failures. There are three mechanisms and they share out different things. Time-slicing (replicas from the NVIDIA k8s device-plugin) multiplexes in time: it advertises the GPU as “N GPUs” and the processes take turns at the compute, but they share the full physical VRAM, with no memory isolation, no fault isolation and no QoS. Its trap is an OOM that does not show up in the Kubernetes scheduler but at run time, when the sum of VRAM allocations exceeds the real memory. MPS (Multi-Process Service) multiplexes in space: it shares out the SMs between processes that execute kernels concurrently, reduces context-switch overhead and allows SM and memory limits per process. It raises throughput when there are many small kernels, but fault isolation is still weak. MIG (Multi-Instance GPU) partitions in hardware: it cuts the Hopper GPU into up to seven instances with dedicated SMs, L2, memory and bandwidth, with real memory, fault and performance isolation; datacenter only (A100/H100/H200/B200), never on an RTX 5090. The rule: real isolation / multi-tenant / compliance → MIG (if it is Hopper); many small concurrent kernels and trust between workloads → MPS; dev, bursts, consumer GPU or no need to isolate → time-slicing. This post works it through with numbers: the VRAM budget of four vLLM instances on an H100 advertised as four replicas, and what fits into a 10 GB MIG instance.

The analogy: a shared hob, one kitchen with several cooks, several kitchens

Imagine you have a single professional hob and three orders to cook at the same time. There are three ways to organise it, and they are exactly the three mechanisms.

Time-slicing is one hob taken in turns, with no larder of your own. Each cook comes in, cooks their dish, leaves, and the next one comes in. The split is temporal: nobody cooks at the same time, they take turns. The problem is not the hob, which gets shared out fine, but the common larder: the ingredients sit in a single shared cupboard and nobody has their own. If the three cooks reserve more flour than there is in total, it is not that they wait their turn: it is that there is no flour. The service goes down for everyone. And if one cook leaves a pan burning and starts a fire, it burns the whole kitchen, not just their corner.

MPS is several cooks coordinated on the same worktop. Now they do cook at the same time, sharing out the space on the worktop (the SMs). A head chef (the MPS daemon) coordinates so they do not collide and so the worktop does not sit empty while one of them waits for water to boil. You can assign each cook a percentage of the worktop and a larder limit. They work faster as a group because the worktop does not sit idle between small tasks. But they still share the kitchen: if one starts a serious fire, the others notice.

MIG is several independent kitchens in the same building. A concrete wall separates each kitchen: its own hob, its own larder, its own door and its own electrical panel. What happens in kitchen 3, a fire, an empty larder, a slow cook, does not touch kitchen 1. It is the only split with genuine isolation. The price: you have to decide in advance how many kitchens and of what size, the walls are fixed, and only expensive buildings (datacenter) come ready to put them up.

The rest of the post is, essentially, when you want cheap turns, when you want coordinated cooks and when you need concrete walls.

An H100 SXM 80 GB does not fill up with just any workload. A bge-reranker-v2-m3 reranker takes a few hundred MB and saturates a handful of SMs; a bge-m3 embeddings service is just as small; a 1B guardrail model in INT4 fits in a couple of GB. Dedicating 80 GB of HBM3 and 132 SMs to serving embeddings is using a hydraulic press to hammer in a drawing pin, the same argument as in mixed environments, but now inside the GPU instead of moving the workload to different silicon.

The aim of sharing is to raise the useful utilisation of fixed capital. But sharing badly introduces three pathologies:

Compute contention: two workloads fight over the same SMs and both run slowly, with unpredictable latency jitter.
Memory contention: the sum of requested VRAM exceeds the physical amount and something dies with a CUDA out of memory.
Cascading failure: a workload that blows up (an illegal kernel, an OOM) can drag its neighbours down with it if they share a context.

The three mechanisms attack these pathologies to different depths. None of them solves all three except MIG, and MIG costs specific hardware. Let us look at them one by one.

Time-slicing: compute turns, shared larder

Time-slicing is temporal multiplexing in software. On Kubernetes, the NVIDIA GPU Operator configures the device-plugin to advertise N replicas of each physical GPU. An H100 declared with replicas: 4 appears to the scheduler as four nvidia.com/gpu resources, and Kubernetes can place four pods on it. Internally, the GPU scheduler hands out compute turns to each process: it runs a bit of process A, switches to B, to C, to D, then back to A. It is the same time-sharing an operating system does with the CPU.

The key idea, and the one that causes most confusion, is this: a replica is NOT a fraction of the GPU. It is a compute turn. NVIDIA’s documentation is explicit: unlike MIG, there is no memory or fault isolation between replicas. The four replicas of the H100 see the full 80 GB of VRAM, unpartitioned. There is no 20 GB per replica. There are 80 GB for all four, handed out in cudaMalloc arrival order.

This has three consequences you need to internalise:

It does not isolate memory. If the sum of what the four processes reserve exceeds 80 GB, the fourth cudaMalloc fails with OOM. The Kubernetes scheduler does not see it coming: it counted four available nvidia.com/gpu resources and happily placed four pods. The OOM appears at run time, not at scheduling time. This is trap number one of time-slicing.
It does not isolate faults. A process that triggers an unrecoverable CUDA error can leave the GPU context in a state that affects its neighbours. They share the same device with no barriers.
It gives no compute QoS. Under contention, the turn-taking does not guarantee a minimum fraction to anyone. Each workload’s latency suffers jitter proportional to how many active replicas are fighting over the GPU at that instant. A latency-sensitive workload (an interactive chat) can see its TTFT dance around depending on what its neighbours are doing.

What is it good for, then? For dev, bursts and low utilisation. If you have four developers who touch the GPU sporadically, advertising four replicas lets all four have access without fighting almost ever (they rarely overlap while active). For batch workloads that tolerate jitter. And, a decisive advantage, it works on consumer GPUs: an RTX 5090 32 GB does not support MIG, but it does support time-slicing. It is the only “Kubernetes-native” way to share a 5090 between several pods.

The VRAM budget in time-slicing (the calculation that avoids the OOM)

Here is the maths you have to do before deploying, because Kubernetes will not do it for you. Suppose an H100 80 GB advertised as 4 replicas and we want to run four vLLM instances on it, one per replica.

vLLM reserves memory with the --gpu-memory-utilization parameter, which is the fraction of the total physical VRAM that each instance keeps (for model weights plus KV-cache). The detail that kills: that fraction is computed over the 80 physical GB, not over a supposed “20 GB of my replica”, because the replica does not have 20 GB. Remember, there is no memory partitioning. Each vLLM sees the 80 GB and reserves its fraction of them.

The no-OOM constraint is therefore that the sum of fractions be less than 1:

$$\sum_{i=1}^{N} g_i < 1 \quad\Longleftrightarrow\quad \sum_{i=1}^{N} g_i \cdot V_{\text{HBM}} < V_{\text{HBM}}$$

where $g_i$ is the --gpu-memory-utilization of instance $i$ and $V_{\text{HBM}} = 80$ GB. It is wise to leave headroom (runtime overhead, fragmentation, CUDA context), so in practice you aim for the sum to stay comfortably below 1, say $\le 0.9$.

A case that works. Four vLLM instances at $g_i = 0.20$:

$$\sum_{i=1}^{4} 0.20 = 0.80 \quad\Rightarrow\quad 0.80 \times 80\ \text{GB} = 64\ \text{GB} < 80\ \text{GB} \quad\checkmark$$

Each instance reserves $0.20 \times 80 = 16$ GB. Four instances add up to 64 GB, leaving a 16 GB cushion. There is no OOM. Each vLLM has 16 GB for weights plus KV-cache: enough for a 7B–8B model in FP8/INT4 with a modest KV-cache.

A case that blows up. The same four instances, but somebody raises $g_i = 0.30$ thinking “I have four replicas, I can give each one more”:

$$\sum_{i=1}^{4} 0.30 = 1.20 \quad\Rightarrow\quad 1.20 \times 80\ \text{GB} = 96\ \text{GB} > 80\ \text{GB} \quad\times$$

The first instances start up and reserve $0.30 \times 80 = 24$ GB each. Three instances are already at $72$ GB. The fourth tries to reserve another 24 GB, there is nothing left, and it dies with CUDA out of memory. And worse: Kubernetes will reschedule it on the same GPU (it still sees four replicas), where it will die again, in a CrashLoopBackOff that makes no sense if you only look at the pod manifest.

The operational rule is brutally simple: in time-slicing, you manage the VRAM budget by hand, adding up the --gpu-memory-utilization values. The replica count controls how many pods fit in compute turns, but it does not reserve a single byte of memory. Confusing the two is the recurring mistake.

MPS: coordinated cooks on the same worktop

The Multi-Process Service (MPS) attacks a different problem. By default, when several processes use the same GPU without MPS, each one has its own CUDA context, and the GPU alternates between contexts (time-slicing at driver level): they do not execute kernels at the same time, they take turns, with context-switch overhead. If your kernels are small and do not fill the GPU on their own, this leaves SMs idle: process A uses 30 % of the SMs during its turn and the other 70 % is wasted.

MPS introduces a daemon that shares a single CUDA context between processes, so that their kernels can execute concurrently occupying different SMs at the same time. It is spatial sharing of compute: instead of taking turns at the whole worktop, each cook occupies a part and they work in parallel. This reduces context-switch overhead and raises throughput when there are many small concurrent kernels that individually do not saturate the GPU.

And, unlike pure time-slicing, MPS allows per-process limits, which gives you a form of QoS:

CUDA_MPS_ACTIVE_THREAD_PERCENTAGE limits the percentage of SMs an MPS client can use. By default each client gets $100 / \text{MaxSharedClients}$. Pinning it to, say, 40 % caps that process’s compute ceiling (MPS docs).
CUDA_MPS_PINNED_DEVICE_MEM_LIMIT imposes a memory cap per client (valid from CUDA 11.5). This is what time-slicing does not have: a per-process VRAM limit that the runtime enforces.

These two limits turn MPS into a resource provisioning mechanism that mitigates the noisy neighbour: you can guarantee that a process does not eat more than X % of the SMs or more than Y GB. The combination gives reasonable QoS, not perfect but real.

The limitation MPS does not solve: fault isolation is weak. Since clients share the daemon’s CUDA context, a fatal error in one client can affect the daemon and therefore the other clients (historically, a client dying dirtily could require restarting the daemon). It is better than time-slicing in this respect, but a long way from hardware isolation. That is why MPS fits when there is trust between the workloads, processes from your own team, not third-party tenants.

The canonical use case: many small concurrent inference requests that individually leave the GPU half empty. MPS overlaps them and raises aggregate throughput. Serving several small models, or several light replicas of the same model, on a datacenter GPU where you trust all the workloads, is MPS territory.

MIG: concrete walls

The Multi-Instance GPU (MIG) is the only one of the three that gives genuine isolation, because it cuts the GPU in hardware. Available on modern datacenter GPUs, A100 (Ampere), H100/H200 (Hopper), B200 (Blackwell), and never on consumer ones: an RTX 5090 (consumer Blackwell) does not support MIG, nor do GeForce cards in general.

MIG divides the GPU into up to seven instances (GPU Instances), and each instance gets a dedicated portion of:

SMs (compute slices): compute is split into 7 slices, each one roughly 1/7 of the SMs.
L2 cache and memory: each instance has its slab of HBM and its portion of L2 cache.
Memory bandwidth: dedicated, not shared.
Data paths and engines: with fault barriers between instances.

The result is that a MIG instance behaves like a smaller, independent GPU: whatever happens in one, an OOM, a kernel blowing up, a workload saturating its compute, does not affect its neighbours. Memory, fault and performance (QoS) isolation, the three things time-slicing does not give and MPS only half gives.

The H100 80GB profiles

MIG does not allow arbitrary sizes: it has fixed profiles. On the H100 80GB, the profile catalogue (notation <compute>g.<memory>gb) is:

Profile	Compute (slices)	Memory	Max. instances
`1g.10gb`	1/7	10 GB	7
`1g.20gb`	1/7	20 GB	4
`2g.20gb`	2/7	20 GB	3
`3g.40gb`	3/7	40 GB	2
`4g.40gb`	4/7	40 GB	1
`7g.80gb`	7/7	80 GB	1 (whole GPU)

(There is also 1g.10gb+me, a variant with media engines for video encoding.) The base memory unit on the H100 80GB is 10 GB per slice (80 GB / 8, with one slice reserved), and the compute unit is 1/7 of the SMs. The profiles combine these units. Note 1g.20gb: the same compute fraction as 1g.10gb (1/7 of the SMs) but twice the memory, useful when a workload needs more VRAM than compute.

One important detail: MIG partitions do not mix freely. The GPU is divided following a valid geometry (the profiles fit together like pieces in a grid), and the profiles are fixed when you configure the GPU; changing them requires draining and repartitioning. They are concrete walls: solid, but they do not move while hot.

The calculation: 7×1g.10gb against 1×7g.80gb

Let us compare the two extremes. On the left, seven 1g.10gb instances: seven isolated 10 GB GPUs. On the right, a single 7g.80gb: the whole H100, unpartitioned.

The operational question is what fits in 10 GB. An instance’s VRAM budget is split between model weights and KV-cache:

$$V_{\text{inst}} = V_{\text{weights}} + V_{\text{KV}} + V_{\text{overhead}}$$

Take a model of 7B parameters in FP8 (1 byte/parameter):

$$V_{\text{weights}} \approx 7 \times 10^9 \times 1\ \text{byte} = 7\ \text{GB}$$

On a 1g.10gb instance (10 GB), after the 7 GB of weights and subtracting roughly 0.5–1 GB of runtime overhead, about 2 GB are left for KV-cache. That is enough for a modest context window and low concurrency, fine for a guardrail service, a classifier or an extraction model that handles short prompts one at a time. A 7B in INT4 (~3.5 GB of weights) leaves ~5.5 GB of KV-cache, much roomier. But a 13B model in FP8 (~13 GB of weights) does not fit in a 10 GB instance: not even the weights get in. For that you need 1g.20gb, 2g.20gb or larger.

Against this, the 7g.80gb (whole GPU) gives you the 80 GB for one large model: a 70B in FP8 (~70 GB of weights) fits with a tight KV-cache, or a 70B with more room spread across several whole GPUs in tensor-parallel (see TP against replicas: one big one versus N small ones).

The reading is clear: fine partitioning (7×1g.10gb) maximises the number of small isolated workloads; not partitioning (1×7g.80gb) maximises the model size that fits. The KV-cache available per instance shrinks proportionally as you partition, so fine-grained MIG serves many light isolated services, not one large model chopped up. If your workload is a single large model, MIG is not for you: use the whole GPU or several in TP.

The decision tree

The three questions, in order:

Do you need REAL isolation?
(multi-tenant, compliance, one workload failing must not touch another)
│
┌────┴────┐
YES NO
│ │
Is it Hopper/ Many SMALL concurrent kernels
Ampere/ AND you trust every workload?
Blackwell? │
│ ┌────┴────┐
┌──┴──┐ YES NO
YES NO │ │
│ │ MPS Dev / bursts / consumer GPU
MIG │ (spatial, / no need to isolate?
│ │ per-process │
│ no real QoS) YES
│ isolation: │
│ rethink TIME-SLICING
│ (move to (temporal, cheap,
│ CPU, another works on a 5090)
│ GPU, or accept
│ the risk with
│ time-slicing)

And each branch in one sentence:

MIG when isolation is a requirement (compliance, multi-tenant, hard SLA) and you have datacenter hardware that supports it. Concrete walls cost money, but if you need them there is no substitute.
MPS when you have many small concurrent workloads that leave the GPU half empty and you trust all of them (same team, not third-party tenants). You raise throughput with reasonable QoS, accepting imperfect fault isolation.
Time-slicing when it is dev, bursts, low utilisation, a consumer GPU, or you simply do not need to isolate anything. Cheap and universal, but manage the VRAM budget by hand.

One nuance the recent documentation records: they can be combined. You can do time-slicing on top of a MIG instance (hardware isolation at the instance boundary, software turns inside), or use MPS inside a MIG instance. The layers are not mutually exclusive; the tree picks the primary strategy.

Applied to the generic 4×H100 cluster

Let us get down to numbers with a generic on-premise cluster of 4×H100 SXM 80 GB with NVLink. It is common to have a heterogeneous menu of workloads: a large chat model, light services (embeddings, reranker, guardrails) and a dev/experimentation queue. Each type calls for a different mechanism. A reasoned split:

GPU 0 and GPU 1, large model in tensor-parallel (not shared). A 70B model in FP8 takes ~70 GB of weights; served comfortably with a generous KV-cache it needs more than one H100. We spread it in tensor-parallel over two whole H100s joined by NVLink (intra-node bandwidth is what makes TP viable; the detail is in TP against replicas). Here we do not share: these two GPUs belong to the large model, full stop. Total isolation by dedication.

$$V_{\text{available}} = 2 \times 80 = 160\ \text{GB};\quad V_{\text{weights}} \approx 70\ \text{GB};\quad V_{\text{KV}} \approx 80\ \text{GB of KV-cache}$$

There is memory to spare for a long request queue and high concurrency.

GPU 2, split with MIG into small instances for light services. Embeddings (bge-m3), reranker (bge-reranker-v2-m3) and a couple of guardrail models (1B–3B) are different workloads, potentially from different teams, and you want a failure or a spike in one not to touch the others. Light multi-tenancy with isolation → MIG. A reasonable partitioning of the H100:

$$\underbrace{3 \times \texttt{1g.10gb}}_{\text{30 GB, 3/7 SMs}} \;+\; \underbrace{1 \times \texttt{4g.40gb}}_{\text{40 GB, 4/7 SMs}}$$

The three 1g.10gb instances (10 GB, 1/7 of the SMs each) host embeddings, reranker and a 1B INT4 guardrail, each isolated, with no noisy neighbour. The 4g.40gb (40 GB, 4/7 of the SMs) hosts an intermediate 7B–13B model with a decent KV-cache for a support service. Each service has its larder and its wall; if the reranker blows up, the chat never finds out.

GPU 3, time-slicing for dev and bursts. The developers touch the GPU sporadically: experiments, short fine-tunes, model trials. They do not need isolation (it is the same team) and they rarely overlap while active. We advertise it as 4 replicas via the device-plugin. Four dev pods fit in turns. VRAM budget with the formula above: if each dev brings up a vLLM at --gpu-memory-utilization 0.20, they add up to $4 \times 16 = 64$ GB < 80 GB, with no OOM. If somebody needs more, lower the replica count or coordinate with the team. The cost of flexibility is manual discipline.

Summary of the split:

Resource	Mechanism	Workload	Isolation
GPU 0 + GPU 1	Dedication (TP)	70B chat in tensor-parallel	total (dedicated)
GPU 2	MIG (3×1g.10gb + 1×4g.40gb)	embeddings, reranker, guardrails, 7B–13B	real hardware
GPU 3	Time-slicing (4 replicas)	dev, bursts, experiments	none (trust)

The logic is always the same: spend isolation (MIG) where you need it, spend cheap concurrency (time-slicing) where you do not, and reserve whole GPUs for what genuinely fills them. An H100 serving embeddings on a 7g.80gb would be as absurd as an RTX 5090 trying MIG: the tool does not match the workload.

What we have not covered

What happens when they do not even fit at the same time: if you have more models than VRAM and they have to take turns in memory (loading/unloading weights, not just taking turns at compute), you enter swap and sleep territory, the sibling piece Serving several models on one GPU.
NUMA-aware scheduling: on multi-socket nodes, which GPU goes with which CPU/memory matters for latency; see Kubelet resource managers on RKE2.
Autoscaling the replicas: how many instances to bring up according to real load, with KEDA and queue metrics; see Autoscaling LLMs on Kubernetes with KEDA.
A jitter benchmark under contention: how much TTFT really wobbles in time-slicing with 4 active replicas compared with MIG, material that deserves its own measurement, not an estimate.

References

NVIDIA — Time-Slicing GPUs in Kubernetes (GPU Operator). https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-sharing.html
NVIDIA — Multi-Process Service (MPS) Overview. https://docs.nvidia.com/deploy/mps/index.html
NVIDIA — MPS: Tools and Interface Reference (CUDA_MPS_ACTIVE_THREAD_PERCENTAGE, CUDA_MPS_PINNED_DEVICE_MEM_LIMIT). https://docs.nvidia.com/deploy/mps/appendix-tools-and-interface-reference.html
NVIDIA — Multi-Instance GPU (MIG) User Guide. https://docs.nvidia.com/datacenter/tesla/mig-user-guide/
NVIDIA — Supported MIG Profiles (H100 80GB catalogue). https://docs.nvidia.com/datacenter/tesla/mig-user-guide/supported-mig-profiles.html
NVIDIA — k8s-device-plugin (time-slicing replicas). https://github.com/NVIDIA/k8s-device-plugin
vLLM — Engine Arguments (--gpu-memory-utilization). https://docs.vllm.ai/en/latest/serving/engine_args.html

The contractor with the master key: isolating AI agents from workstation to cluster

Tue, 09 Jun 2026 16:00:00 +0200

First post of a pair on AI agent isolation. This one fixes the what and the where: the complete map of isolation primitives and which domain each belongs to. The sibling runbook fixes the how, with commands: ai-jail and bubblewrap on the client, Tetragon TracingPolicy and RuntimeClass on the cluster. If you are only going to read one, this one gives you the mental model; the other one gives you the files to copy and paste.

TL;DR

An AI agent that runs code needs access to your filesystem and your tools: compiler, linter, grep, make, cargo, npm. That is the minimum to be useful. The problem is that alongside that access travels the ability to read ~/.aws/credentials, exfiltrate your SSH keys or fire an rm -rf outside the project directory. And no malicious model is needed: a compromised dependency in an npm install is enough, because the well-meaning agent and the poisoned post-install script run with the same permissions. The answer is not to trust the LLM’s good intentions; it is to isolate in order to bound the blast radius. This post walks the five isolation families of 2026, from the process sandbox to the full VM, and sorts them into two columns: what applies on the client (the developer’s workstation: bubblewrap, ai-jail, sandbox-exec, Landlock, the native Claude Code and Codex sandboxes) and what applies on the cluster (where the agent or the inference runs on Kubernetes: namespaces+seccomp, gVisor, Firecracker/Kata microVMs and eBPF/Tetragon as the layer for live observation and enforcement). The thesis: the threat model is the same in both places; the tools are not. Policy extrapolates; the primitive gets rewritten.

The analogy: the contractor with the master key

You hire a builder for a renovation. He is competent and acts in good faith. But two things happen that you do not control. The first: he may misread the instruction and knock down the wrong partition wall. The second, worse: his toolbox may have been tampered with before he came through your door — somebody put something inside — and when he opens it in your living room, that something activates.

Nobody sensible gives him the master key to the whole building. You open the room he works in, you leave him the tools he needs, and you keep the study with the safe and the server room locked. If the renovation goes wrong, by error or by sabotage, the damage stays in that room.

An AI agent is that contractor. The sandbox is the key policy: you give it the project room and the tools, not the master key to the system. And here is the twist that justifies two posts: the builder works in two different buildings. One is your flat, the developer’s workstation, with your credentials, your ~/.ssh, the browser’s password vault. The other is the datacenter, the cluster where inference and autonomous agents serve customers, with data from several tenants at once. The key policy is identical in both: least privilege, bound the radius. But the lock on your flat’s door is not the same as the one at the datacenter. In the flat you fit a cylinder lock (bubblewrap). At the datacenter you put a guard watching every door and a separate wing of the building (Tetragon + microVM). Same principle, different hardware. That is extrapolating the technology, not copying it.

The threat model: what a runaway agent can do

Before choosing a lock it helps to enumerate the burglar. The attack surface of an agent that executes arbitrary bash breaks down into five concrete threats. Not all are defended by the same layer and, an uncomfortable fact that the security documentation itself acknowledges, no single layer covers them all.

Threat	What the agent does	Minimum layer that cuts it
Filesystem out of scope	Reads `.env`, `~/.ssh/id_rsa`, system secrets; modifies sources outside the project	Process sandbox (path allowlist)
Arbitrary network egress	Exfiltrates data, receives instructions from a remote C2, calls unauthorised APIs	Network blocking / NetworkPolicy / microVM
Kernel syscall surface	A kernel exploit from inside the container escalates to the host (shared kernel)	gVisor or microVM (dedicated kernel)
Cross-tenant leakage	One customer’s workload reads another’s data on a multi-tenant platform	microVM (de facto standard)
Secret exfiltration	Extracts tokens and environment variables via `/proc` or the environment	`--clearenv` / tmpfs `$HOME` / secrets outside the pod

There is a sixth threat that no sandbox solves: prompt injection. If an attacker manages to slip instructions into the agent’s context — a poisoned comment in the code, a malicious file the agent reads, an adversarial response from a tool — the agent will execute those instructions with whatever permissions the sandbox grants it. Isolation shrinks the impact radius of a successful injection; it does not prevent the injection. That is why the sandbox is a layer, not the solution: on top go input validation, tool-call allowlists and output auditing. The sentence to internalise: isolation does not make the agent trustworthy; it bounds what an untrustworthy agent can break.

Two domains, one policy

The builder works in two buildings. The distribution of tools reads better in two columns:

The client: isolating the agent on the workstation

Here the agent is a coding assistant — Claude Code, Codex, OpenCode, Cursor — that a developer launches on their machine. One process, one user, and right next to it the juiciest assets that exist: ~/.aws/credentials, ~/.ssh, ~/.gnupg, the browser’s password store. The matching tier is the lightest one: the process sandbox.

bubblewrap (Linux) and sandbox-exec (macOS). bubblewrap (bwrap) is the same sandbox Flatpak uses to isolate each desktop app: a ~50 KB binary, ~4,000 lines of C, maintained by the GNOME team and, the key property, it runs without root via CLONE_NEWUSER, creating namespaces with no elevated privileges. It mounts $HOME as an ephemeral tmpfs and only exposes the project directory, writable; the rest of the system becomes invisible. On macOS the equivalent is sandbox-exec with SBPL profiles: a legacy Apple API, officially deprecated and with no public replacement, but working today. Parity between the two is not exact — on macOS the GPU (Metal) and the display (Cocoa) are system services and sandbox-exec does not restrict them — but both protect what matters: access to the sensitive areas of the filesystem.

Landlock as a second barrier. bubblewrap isolates via namespaces and mounts; Landlock, a Linux Security Module available since kernel 5.13, restricts access at the VFS level, independently of namespaces. It does not replace bwrap: it complements it. It closes vectors that mount-based isolation does not cover on its own (escape routes via /proc, symlink tricks inside allowed mounts) and acts as a safety net should the namespace machinery have a bug. It is defence in depth inside the client itself, and it degrades cleanly to a no-op on kernels that do not support it.

Dev containers, when reproducibility is needed. A dev container (devcontainer.json, what Codespaces and Cursor use) is a Docker container with a configuration layer on top. It gives reasonable filesystem isolation and easy reset (destroy and recreate), but it shares the host kernel, the same limitation as any Docker, and it tends to be long-lived, accumulating state. For an agent running your own team’s code on your own machine it is a good repeatability story; it is not, on its own, the isolation layer for untrusted code.

What wraps all of this. A hand-written bash script works, but it does not scale to a team. The tools that package it:

ai-jail (Rust, GPL-3.0): wraps bwrap/sandbox-exec with per-project config in a .ai-jail file (TOML, committable to the repo, so the whole team inherits the same policy), auto-detection of GPU/Docker/display, a --lockdown mode (project read-only, network cut with --unshare-net, --clearenv), --dry-run for auditing, and --bootstrap to generate each agent’s permission allowlists. It is tool-agnostic: the same binary serves Claude, Codex, OpenCode or Crush. It also applies Landlock automatically on 5.13+ kernels as defence in depth.
The Claude Code /sandbox: since October 2025 Claude Code ships its own sandbox that uses, exactly, bubblewrap on Linux and sandbox-exec on macOS. Its Sandboxed Bash isolates shell commands, but not the file tools, the MCP servers or the hooks, which run with the process’s full permissions unless you enable the beta sandbox-runtime package, which wraps the whole process. There is one nuance worth knowing: if a command fails because of a restriction, the agent can retry with dangerouslyDisableSandbox — it is opt-out, not opt-in.
Codex CLI: three modes via --sandbox (read-only, workspace-write, danger-full-access); the recommended default is workspace-write. The philosophy is deliberate: Codex does not provide the isolation, it delegates it to the environment that wraps it. danger-full-access only makes sense inside a microVM.
Cursor: its cloud agents run in isolated VMs; /worktree creates a single-use isolated worktree per task, and /best-of-n launches several attempts in parallel in separate worktrees.

The cluster: isolating the agent in production

The second building is the datacenter. Here the “agent” can be an autonomous agent running with no human in front of it, or the inference service itself executing generated code, or a multi-tenant workload where one customer’s pod must not touch another’s data. The process is no longer one: they are pods in a Kubernetes cluster (RKE2/RKE3 on a typical sovereign platform). The primitives change in nature.

The pod baseline. First of all, the standard kit: Linux namespaces, seccomp (RuntimeDefault) to trim the syscall surface, cgroups for resource limits, an unprivileged securityContext (runAsNonRoot, readOnlyRootFilesystem, dropping all capabilities) and NetworkPolicy to cut egress. It is the cluster equivalent of the process sandbox allowlist. Necessary, but it shares the kernel with the host: insufficient for genuinely untrusted code.

gVisor (runsc). Google’s user-space kernel: it intercepts the workload’s syscalls before they reach the host kernel and serves them inside a Linux kernel reimplemented in Go (the Sentry). The surface exposed to host kernel vulnerabilities drops sharply, while keeping fast startup and a low footprint. It is the middle ground when the risk of a kernel escape is real but the overhead of a microVM is not acceptable.

Firecracker / Kata microVMs. The de facto standard for untrusted code in 2026. Firecracker (AWS’s VMM written in Rust, on top of KVM) gives each sandbox a dedicated Linux kernel: a kernel exploit inside the microVM does not reach the host by construction. It is what sits underneath Vercel Sandbox (GA January 2026) and E2B. On Kubernetes, Kata Containers brings that model to a RuntimeClass: you mark the untrusted agent’s pod with runtimeClassName: kata and it runs in its own microVM instead of sharing the node’s kernel. For multi-tenant with generated code, this is the baseline, not the luxury.

eBPF / Tetragon: the layer we already have. Here is the piece that separates a platform with runtime observability from one that only trusts its configuration. The previous layers are static: they define what the pod can do before it starts. Tetragon, Cilium’s runtime security component, based on eBPF, is dynamic: it observes, in the kernel and at minimal cost, every process execution, every network connection and every file open in every pod, and it can act inline. It does not replace the sandbox; it watches it from inside the kernel. Where bubblewrap on the client blocks curl with a command blocklist, Tetragon on the cluster hooks tcp_connect in the kernel and, if the destination is not allowed, kills the process with Sigkill before the packet leaves. Where the client hides ~/.ssh behind a tmpfs, Tetragon hooks security_file_open and reports, or kills, any attempt to read a sensitive mounted path. It is the guard patrolling the corridors while the microVMs are the walls. And it is exactly the kind of control that materialises the ENS monitoring and traceability measures (op.mon, op.exp) without instrumenting the application: visibility lives in the kernel, not in the agent’s code.

The landscape table

The five families, their relative isolation strength, their startup cost and the domain where they live:

Tier	Primitive	Isolation	Startup	Natural domain
Process sandbox	Seatbelt · bubblewrap	Baseline	~0 ms	Client (Claude Code default)
Dev container	Docker + seccomp	Moderate	seconds	Client / cluster (repeatability)
User-space kernel	gVisor (`runsc`)	Strong	ms	Cluster (medium multi-tenant)
microVM	Firecracker · Kata	Strongest (practical)	<1 s	Cluster (untrusted code)
Full VM	KVM · EC2	Maximum	30 s+	Cluster (external boundary, compliance)
Runtime enforcement	eBPF · Tetragon	Cross-cutting	always on	Cluster (watches+kills on top of any tier)

Tetragon gets its own row on purpose: it is not a rung on the ladder, it is a cross-cutting layer that operates on top of any of the others. It stacks with all of them.

A numerical note on why the “startup” column decides as much as the “isolation” column. A full VM wins on raw isolation but takes tens of seconds to provision; for an agent that needs a fresh environment per request or per session, that cost is prohibitive. A Firecracker microVM starts in under 1 second and a process sandbox in ~0 ms. That is why the dominant pattern in 2026 is not “the most isolated VM”, but a full VM as the external boundary plus a microVM as the per-request execution unit inside, the architecture of Vercel, AWS Lambda and E2B. On the client the arithmetic is the opposite: the developer launches the agent dozens of times a day interactively, and a startup measured in seconds would break the flow; hence the process sandbox, with its microsecond overhead, is the correct default.

Extrapolate, do not copy

The thesis of this pair of posts fits in one sentence: the threat model is invariant across domains; the primitive that implements it is not. The client and the cluster defend exactly the same five vectors — filesystem, network, kernel, multi-tenant, secrets — but with toolboxes that do not overlap. Every control has its twin on the other side:

$HOME as an ephemeral tmpfs (client) ↔ readOnlyRootFilesystem + emptyDir (cluster).
A curl/wget blocklist in bwrap (client) ↔ a TracingPolicy on tcp_connect in Tetragon plus NetworkPolicy (cluster).
--unshare-net in lockdown (client) ↔ default-deny NetworkPolicy (cluster).
No escape hatch, the process lives inside bwrap (client) ↔ no privileged, no hostPath, RuntimeClass kata (cluster).
~/.ssh and ~/.aws never mounted (client) ↔ secrets outside the pod plus Tetragon watching security_file_open (cluster).

The sibling runbook turns each of these equivalences into concrete files. What matters here is the method: when someone shows you an agent sandbox, be it the Claude Code /sandbox on a laptop or a microVM on a PaaS, the useful question is not “which tool does it use?”, but “which of the five vectors does it close, and which does it leave open?”. The tool gets replaced; the threat map stays.

What no sandbox solves

Three limits that Anthropic’s own documentation states, and which are worth keeping in view so as not to sell smoke:

Egress remains a risk in any sandbox that allows outbound connections. If the agent can open a connection, it can exfiltrate. That is why the client lockdown cuts the network and the cluster uses default-deny NetworkPolicy plus Tetragon: you do not trust “filtering well”, you trust “not letting anything out”.
Code modification remains possible in any sandbox with the project directory mounted writable. The remedy is not a sandbox-technical one, it is git: with the remote untouched and no push permission, the worst case is corrupting the local working copy — git checkout . and start again. The damage never reaches the remote.
No sandbox prevents a compromised prompt from reaching the API. Isolation bounds the impact of an injection; it does not prevent it. The complementary defences — input validation, tool-call allowlists, output auditing — are mandatory alongside isolation, not instead of it.

The operational conclusion: isolation shrinks the blast radius; defence in depth is what closes the circle. A process sandbox for trusted code on a known machine is appropriate and practically free. For an agent acting on user prompts, executing generated code or running multi-tenant, the acceptable minimum in 2026 is a microVM, with Tetragon watching from above. Pick the tier that matches your real threat, check which vector it leaves open, and stack complementary controls on top.

References

bubblewrap: https://github.com/containers/bubblewrap
Landlock LSM: https://landlock.io · https://docs.rs/landlock
ai-jail (Fabio Akita): https://github.com/akitaonrails/ai-jail
gVisor: https://gvisor.dev
Firecracker: https://firecracker-microvm.github.io
Kata Containers: https://katacontainers.io
Tetragon (Cilium): https://tetragon.io
Vercel Sandbox — concepts: https://vercel.com/docs/vercel-sandbox/concepts
E2B: https://github.com/e2b-dev/E2B
Claude Code sandboxing: https://docs.claude.com/en/docs/claude-code
Codex CLI: https://github.com/openai/codex

QLoRA and multi-LoRA at the limit on small models

Tue, 09 Jun 2026 02:30:00 +0000

This post is the training-side companion to Multi-LoRA serving. That one takes apart the consumer, how hundreds of concurrent adapters are served with SGMV kernels and unified paging; this one takes apart the producer, how an adapter is trained on top of a quantised base on a single GPU, and why the pattern “one frozen base SLM + N low-rank adapters” is the natural fit for small models. We do not repeat the serving internals here; we assume you have read them.

TL;DR

QLoRA (Dettmers et al., NeurIPS 2023) solves a specific problem: fine-tuning a model without having the VRAM to hold its weights in BF16, its gradients and the optimiser states. The idea has three pieces. One: freeze the base and quantise it to 4-bit with a new format, NF4 (NormalFloat 4-bit), quantile-optimal for weights distributed almost like a Gaussian. Two: do not train the base, not a single one of its weights moves, but rather a pair of small LoRA matrices in BF16 plugged in parallel; the gradient flows only through that adapter. Three: two memory tricks, double quantisation (quantising the quantisation constants themselves) and paged optimizers (optimiser states that page out to RAM when VRAM gets tight). The measurable operational result: a 3-8B SLM is fine-tuned on an RTX 4090 (24 GB, Ada Lovelace), not on a cluster. And because the output of training is an adapter measured in megabytes, not gigabytes, the pattern that emerges is a single frozen base SLM in 4-bit plus N adapters, one per customer, domain or task, served on top of the shared base with the stack we already covered in multi-LoRA serving. Per-customer isolation, minimum footprint, sovereign deployment.

The analogy: the frozen guitar and the swappable pedalboard

Think of a session guitarist recording for very different clients: a jazz record, a corporate jingle, a metal track. He has one single guitar, his trusted instrument, set up, with a base tone he knows by heart. What he does not do is buy a new guitar for every song. What he does is keep a pedalboard: a distortion pedal, a chorus, a delay. For each track he plugs in the pedal that fits, and the same guitar sounds completely different.

The mapping is exact:

The guitar = the base SLM. A single copy, set up at the factory, frozen. In QLoRA it is also kept in a compressed case: quantised to 4-bit. You do not touch it: you do not change its pickups or adjust the neck. It weighs what it weighs and there it stays.
Each pedal = a LoRA adapter. Small, cheap, specific to one sound. You train it for a task and put it in a drawer.
Training QLoRA = designing a new pedal by listening to the (frozen) guitar through it, adjusting only the pedal’s pots until it sounds the way you want. The guitar’s base tone is not modified; you learn the correction the pedal applies on top.
Serving multi-LoRA (Multi-LoRA serving) = having the whole pedalboard set up on stage and choosing the right pedal per note, per request. The guitar is the same; what changes between requests is which pedal is active.

The analogy holds down to the detail that confuses people most: the training gradient only “touches” the pedal. The guitar is frozen in its compressed case; learning does not move it. That is what lets the base live in 4-bit throughout the fine-tuning without the quantisation getting in the way: no gradient is ever computed for it.

The bare mechanism: LoRA, and why you can train on a 4-bit base

A minimal reminder of LoRA (Hu et al., ICLR 2022). An adapter modifies a matrix W of the base by adding a low-rank product to it:

$$W' = W + B A, \qquad A \in \mathbb{R}^{r \times d}, \quad B \in \mathbb{R}^{d \times r}$$

with r the rank, much smaller than d. In the forward pass BA is not materialised; what is computed is:

$$y = W x + B(A x)$$

The base compute (Wx) happens just the same; the adapter adds two cheap matmuls. The key to QLoRA is who receives a gradient. The base W is frozen: ∂L/∂W is neither computed nor stored. Only A and B are trainable. That is why W can live quantised to 4-bit without trouble: in the forward pass the quantisation is undone on the fly to compute Wx (dequant → matmul in BF16), but since W is never updated it does not need the precision of a trainable weight. The adapter A, B is in BF16, and it is the only path the gradient flows through.

This is what breaks the memory wall. In a full fine-tuning you need, per weight: the weight (2 bytes BF16), its gradient (2 bytes), and the two Adam states (momentum and variance, typically 4+4 bytes in FP32), on the order of 12-16 bytes per trainable parameter. With QLoRA, the base weights take 0.5 bytes (4-bit) and have no gradient and no optimiser states. Only the few million adapter parameters pay the 16-byte cost. For an 8B, that is the difference between ~130 GB and fitting in 24 GB.

NF4: why a new format instead of INT4

QLoRA does not use linear INT4 for the base, but NF4 (NormalFloat 4-bit). The intuition: the weights of a trained transformer are distributed, empirically, very close to a zero-centred Gaussian. INT4 spreads its 16 levels uniformly across the range, which wastes levels in the tails (where there are almost no weights) and leaves few in the centre (where they pile up). NF4 spreads the 16 levels according to the quantiles of a normal distribution: more levels where there is more probability mass. It is, by construction, information-theoretically optimal for exactly Gaussian data, each level covers roughly the same number of weights. It is also symmetric about zero and guarantees an exact representation of 0 (important for sparsity and padding). The detail of the quantisation formats is in Quantisation for LLM inference; here the idea that NF4 spends its bits where the weights are is enough.

Double quantisation and paged optimizers

Quantising to 4-bit is not entirely free: for each block of weights (typically 64) you need to store a scale constant in FP32 so the quantisation can be undone. Those constants weigh something. With blocks of 64 and one FP32 scale (32 bits) per block, that is 32/64 = 0.5 bits per parameter in metadata alone, a 12.5 % overhead on top of the 4 useful bits. Double quantisation attacks that: it quantises the scale constants themselves (to 8-bit, in blocks of 256), bringing the overhead down to ~0.127 bits/param. Quantising the quantisation sounds recursive and it is; the saving is small in absolute terms (~0.37 bits/param) but on an 8B that is hundreds of MB, which is exactly the margin separating “it fits” from “it does not fit” on a 4090.

Paged optimizers attack the memory peaks. During training, certain moments, a batch with a very long sequence, a large activation, push VRAM close to the limit and blow up with an OOM. The idea, borrowed from operating system paging, is to allocate the optimiser states in NVIDIA unified memory: when VRAM gets tight, those pages are evicted to host RAM automatically and brought back when needed. It speeds nothing up; it avoids the crash at the peaks. It turns an “intermittent OOM” into “a bit slower at the worst moments”, which for an unattended training run on a single GPU is the difference between finishing and not finishing.

“Aggressive” training: very low rank and QA-LoRA

“Aggressive” in this context means two things, sometimes combined.

Very low rank (r = 4-8). The rank is the bottleneck of the correction: how much “capacity” the adapter has to steer the base away from itself. A high rank (64, 128) brings the adapter closer to a full fine-tuning but weighs more and takes longer to train. For an SLM adapted to a narrow, well-defined task, an output format, a lexical domain, a response style, a rank of 4-8 is usually enough, and the resulting adapter weighs a fraction. The risk of a low rank is underfitting: if the task demands rewriting a lot of the base’s behaviour, r=4 falls short. The honest rule is empirical: raise the rank only if the eval asks for it, not “just in case”. On small SLMs, where the base has less spare capacity, a low rank tends to work proportionally better than on large models, but this depends on the task and has to be measured, not assumed.

QA-LoRA (quantization-aware LoRA, Xu et al., arXiv:2309.14717). There is a subtle friction in standard QLoRA: you train the adapter in BF16 against a 4-bit base, but if you then want to merge the adapter into the base (W' = W + BA) to serve a clean quantised model, the merge reintroduces precision that the 4-bit format cannot represent, and on requantising you lose part of what was learnt. QA-LoRA trains the adapter aware of the target’s quantisation: it balances the degrees of freedom of the quantisation and of the adaptation (with group-wise quantisation) so that, when it finishes, the adapter merges cleanly into a quantised base without a requantisation step that degrades it. The result is a final model that is quantised-plus-adapted, with no separate adapter at runtime, useful when you want a single deployable artefact per task instead of the shared-base + adapters pattern. The choice between “QLoRA + serve multi-adapter” and “QA-LoRA + merge per task” is a deployment architecture decision, not one of pure quality.

The maths that matters

Three calculations drive any QLoRA decision on SLMs.

Adapter parameters. For each target matrix of dimension d with rank r, the adapter contributes A (r×d) plus B (d×r), that is 2·r·d parameters. Summing over the target matrices and multiplying by the number of layers:

$$\text{params}_{\text{adapter}} = L \cdot \sum_{\text{matrices}} 2 \cdot r \cdot d$$

Worked example — Llama-3-8B, attention (q, k, v, o), d = 4096, L = 32 layers, r = 8. Taking the four attention projections with the same d = 4096 (a simplification; in Llama-3 K and V are narrower because of GQA, which gives even fewer params):

$$\text{params} \approx 32 \cdot 4 \cdot (2 \cdot 8 \cdot 4096) = 32 \cdot 4 \cdot 65\,536 \approx 8.4\text{M params}$$

In BF16 (2 bytes/param): 8.4M · 2 ≈ 16.8 MB ≈ ~17 MB. Seventeen megabytes. Compare that with the base: an 8B in NF4 takes 8\text{G} · 0.5\,\text{bytes} ≈ 4\text{ GB} (plus the small constants overhead after double quantisation). The adapter is 0.4 % of the size of the quantised base. This is what makes having hundreds operationally trivial: an adapter is not a model, it is almost a heavy configuration file.

How many adapters fit on a 4090 after the base + KV? Budget for an RTX 4090 (24 GB): 8B NF4 base ~4 GB, leave ~5 GB for KV cache and inference activations at moderate concurrency → that leaves ~15 GB free (being conservative, call it ~12-15 GB). With ~17 MB adapters (r=8, attention-only):

$$\frac{15\,000\ \text{MB}}{17\ \text{MB/adapter}} \approx 880 \text{ adapters}$$

On the order of thousands if you cut the reserved KV cache or use rank 4 (~8.5 MB/adapter → ~1750 in 15 GB). The bottleneck is never the adapters’ space; it is the KV cache and concurrency. For the details of how those thousands are served concurrently, the heterogeneous batching, the unified paging, the SGMV kernels, see Multi-LoRA serving. The relevant summary here: the adapter compute is nearly free (low rank, two thin matmuls); the serving performance challenge is not that compute but the gather/scatter of the right adapters per batch row when a single batch mixes requests from different adapters. That is the consumer’s problem, not the producer’s.

QLoRA training VRAM in 24 GB. The rough budget to fine-tune the 8B on a 4090:

Component	Approx. VRAM
8B base in NF4 (frozen weights)	~4.0 GB
Adapter (BF16 params + gradient + Adam states, ~16 B/param over ~8-40M params)	~0.3-0.7 GB
Activations (depends on batch and sequence length; the variable bulk)	~6-14 GB
Dequant buffers, scales, workspace	~1-2 GB
Total	fits in 24 GB with margin

The large, variable piece is the activations, which scale with batch × sequence length. That is why real QLoRA on a 4090 is done with a small batch + gradient accumulation (simulating a large batch by accumulating gradients over microbatches) + gradient checkpointing (recomputing activations in the backward pass instead of storing them, trading compute for memory) + bounded sequences. The paged optimizers are the airbag for the activation peaks that would otherwise blow up. The claim “QLoRA fine-tunes an 8B on a 4090” is true with that configuration; with long sequences, a large batch or a high rank, it does not fit. As with any number, the methodology matters more than the headline.

The fit with small models and sovereignty

This is where QLoRA + SLM stops being a VRAM trick and becomes an architectural pattern.

An SLM (3-8B) already fits comfortably on a single GPU for inference. If on top of that the base lives in 4-bit (~4 GB for an 8B), you have memory to spare. What QLoRA enables is for that same machine, the 4090, to be both the producer and the consumer: you train a new customer’s adapter in hours, on the same class of hardware where you then serve it. The artefact that travels between “train” and “deploy” is an adapter of MB, not GB: it is versioned, signed, moved across the network, stored in MinIO/S3 without thinking about the cost.

The sovereign pattern falls out on its own:

Per-customer isolation. Each customer has their adapter, trained only on their data. The base is generic and shared; what is specific to the customer lives isolated in their (A, B) pair. Deleting a customer is deleting a file of a few MB, not retraining anything.
Minimum footprint. One base + N adapters fits where N bases would not come close. The economics of “one model per customer” (tens of GB each) are prohibitive; those of “one base + adapters” (MB each) are trivial. It is exactly the difference between the pedalboard and buying a guitar per song.
Sovereign deployment. Everything fits on-premise, on your hardware, without a single data item leaving the perimeter. Training (QLoRA on the 4090) and serving (multi-LoRA on the same base) live inside. There is no dependency on an external API for fine-tuning or for serving.

The choice between adapting by domain (one adapter per knowledge area) and retrieving by context (RAG that injects the knowledge into the prompt) is real and not mutually exclusive: the adapter changes the model’s behaviour and style, RAG changes the facts it has access to. The sibling post on aggressive RAG on small models in this series works through it; the short rule is: adapt what is stable and behavioural, retrieve what is volatile and factual.

Applied to on-premise infrastructure

On an RTX 4090 (24 GB, Ada Lovelace)

This is QLoRA’s natural workbench. The canonical case: a 3-8B base SLM in NF4, fine-tuning an r=8 attention-only adapter, with gradient checkpointing + gradient accumulation + paged optimizer. It trains in hours for narrow-task datasets (thousands to tens of thousands of examples), and the same machine then serves the base + tens or hundreds of adapters for multi-tenant demos and platform prototypes. The 4090 is where QLoRA went from “a paper technique” to “anyone with a consumer GPU can do it”, and that is exactly its value. The honest rule: it fits with the memory configuration described; with long sequences, a large batch or a high rank, move up the hardware.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

Here QLoRA stops being strictly necessary in order to fit, an 8B in BF16 has room to spare, but it remains useful for another reason: parallelising adapter production. With 320 GB and native FP8 you can train several adapters at once (one job per customer, several in parallel), or fine-tune somewhat larger models with QLoRA without TP. The consumer on this cluster is the serious setup from Multi-LoRA serving: an FP8 base + hundreds of concurrent adapters. The rule of thumb: on the 4090, QLoRA is the tool to be able to fine-tune; on the H100 cluster, it is the tool to fine-tune many at once, cheaply, keeping the quantised format consistent between training and serving.

What we have not covered

The internals of heterogeneous serving (SGMV kernels, MBGMM/MBGMV, unified paging, cold start, eviction): they are covered in full in Multi-LoRA serving. This post is deliberately the producer’s side.
DoRA and variants (magnitude-direction decomposition): they close part of the gap with full fine-tuning; different training pattern, identical serving pattern.
Sub-4-bit and ternary quantisation of the base: what happens when the base drops from NF4 to 2-bit or ternary underneath the adapter; the sibling post in the series works through it.
Collecting the fine-tuning dataset: how each adapter’s corpus is built from production feedback is in Retrain: closing the loop.

References

Dettmers, T., Pagnoni, A., Holtzman, A., Zettlemoyer, L. QLoRA: Efficient Finetuning of Quantized LLMs. NeurIPS 2023. https://arxiv.org/abs/2305.14314
Hu, E., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W. LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022. https://arxiv.org/abs/2106.09685
Xu, Y., Xie, L., Gu, X., Chen, X., Chang, H., Zhang, H., Chen, Z., Zhang, X., Tian, Q. QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Models. ICLR 2024. https://arxiv.org/abs/2309.14717
Sheng, Y. et al. S-LoRA: Serving Thousands of Concurrent LoRA Adapters. MLSys 2024. https://arxiv.org/abs/2311.03285
Chen, L. et al. Punica: Multi-Tenant LoRA Serving. MLSys 2024. https://arxiv.org/abs/2310.18547
Official QLoRA / bitsandbytes repo: https://github.com/artidoro/qlora
Hugging Face PEFT (LoRA, QLoRA): https://github.com/huggingface/peft

Aggressive RAG in small models: trading parameters for retrieval

Tue, 09 Jun 2026 02:20:00 +0000

This post belongs to the series on inference performance in small models. Its sibling piece, The roofline inverts in small models, explains why the compute-bound prefill is the bottleneck that shapes the whole discussion here. It is worth reading first: here we assume that adding more context is not free.

TL;DR

An SLM (say 1B–8B parameters) knows fewer facts than a 70B–700B model, simply because it has fewer weights in which to memorise them. But its ability to reason over text in front of it, following instructions, extracting, synthesising, comparing, degrades far less with size than its encyclopaedic knowledge does. The operational consequence is direct: use the SLM as a reasoning engine over curated context, not as a database. Move knowledge from the weights into the context via retrieval. The problem is that “aggressive retrieval” is often read as “put in lots of chunks”, and that collides head-on with three facts about SLMs: shorter context windows, worse use of long context (the lost in the middle effect is more severe the smaller the model) and a compute-bound prefill whose cost grows with the context length $C$, linearly in the projections and quadratically in attention. You cannot simply add tokens. Each retrieved token is paid for twice, in degraded quality and in TTFT, and the SLM is the worst placed to absorb either cost. The way out is not to retrieve less, but to retrieve better: reranking for precision over recall, compressing the context before injecting it, prefix caching the stable documents, semantic caching of answers, and structured output with external tools that stand in for internal knowledge. This post works through the maths and gives a TTFT number before and after compressing a context from 4,000 to 1,000 tokens on an RTX 4090.

The analogy: the open-book exam

Two students sit the same exam. The first has a prodigious memory: they have memorised the entire syllabus, paragraph by paragraph. The second has an ordinary memory, forgetting dates, confusing names, but is allowed to bring in a crib sheet.

If the second student’s crib sheet is a chaos of piled-up photocopies, they lose: they take too long to find what they are after, get distracted by irrelevant pages and run out of time. But if their crib sheet is excellent, trimmed to the essentials, reordered by relevance, with the important things at the top and no filler, then they not only do not lose: they often win, because they reason just as well as the first student and, on top of that, work from verified material instead of from blurry recollections they may be inventing.

The moral has three layers, and each maps to an engineering decision:

Memorising everything is expensive. The first student invested months. A large model invests parameters, and VRAM, and inference FLOPs, in memorising facts.
The crib sheet matters more than its size. A well-made one-page crib sheet beats ten badly organised pages. More retrieved context is not better context: the material’s precision beats volume.
Knowing how to search and synthesise is a different skill from knowing. It is the one the SLM retains. The whole strategy consists of leaning on that skill and subcontracting the memory.

The rest of the post is, essentially, how to build an excellent crib sheet under the constraint that the student (the SLM) reads slowly and gets tired with long texts.

The capacity argument: how many facts fit in the weights

Let us start by justifying the thesis with an order of magnitude, not with faith. How much factual knowledge really fits in a model’s weights?

There is a recurring empirical estimate in the interpretability and memorisation literature: a dense model can store on the order of 2 bits of memorised information per parameter before saturating (the exact figure varies by study and training regime; take it as an order of magnitude, not as a law). An 8B-parameter model then has an information storage ceiling on the order of:

$$8 \times 10^9 \text{ params} \times 2 \text{ bits/param} = 1.6 \times 10^{10} \text{ bits} \approx 2 \text{ GB of information}$$

And that budget is not only for facts: the vast majority goes on grammar, syntax, reasoning ability, code, formatting, and only a fraction is left for encyclopaedic knowledge. Compare it with the other side: a retrievable corpus of several million documents, a corporate wiki, a document repository, a technical knowledge base, easily takes up hundreds of GB to terabytes of text, indexed and queryable with millisecond latency. The asymmetry is two or three orders of magnitude in favour of the external corpus.

The conclusion is not that the weights are useless, they are where reasoning lives, which is the expensive thing to replicate, but that competing with an external index on fact capacity is losing by construction. A 70B model has ~9× more memorisation budget than an 8B one, but it is still negligible against the corpus. That is why the large model also does RAG in production. The difference is that the SLM needs it: without retrieval, its factual knowledge is too sparse and, worse, prone to hallucinating exactly in the gaps it did not memorise.

The central tension: retrieving more is not putting in more

This is where most naive designs break. “Aggressive retrieval” sounds like a large top-k: if retrieval helps, retrieve 20 chunks instead of 5. But in an SLM that fails for two independent reasons, one of quality and one of cost.

(a) SLMs make worse use of long context

The lost in the middle effect (Liu et al., 2023) is well known: LLMs retrieve information placed at the beginning and end of the context better, and the information in the middle worse. What gets less emphasis is that the effect is more severe the smaller the model. An SLM has fewer attention heads, fewer layers and poorer internal representations for “tracking” a relevant fact buried at position 14 out of 20 chunks. Its nominal context window also tends to be shorter (4K–32K against the 128K+ of the large ones), and the effective window, the length beyond which quality collapses, is smaller still. Putting in 20 chunks does not mean the model reads all 20: it means it will probably ignore or misread the ones in the middle, while paying the cost of all of them.

(b) Prefill grows with the context and is compute-bound

This is the blow people underestimate. Prefill, processing the full prompt before emitting the first token, is the compute-bound phase of inference (unlike decode, which is memory-bound; the detail lives in The roofline inverts). Its cost grows with the context length $C$, and it determines TTFT (time to first token). More chunks → more prefill tokens → more TTFT and more compute cost per request. In an SLM, where prefill is proportionally more expensive relative to the model, this hurts especially.

The operational conclusion is uncomfortable but clear: you cannot compensate for fewer parameters simply by putting in more context. Each retrieved token is paid for twice, in degraded quality and in TTFT, and the SLM is the worst placed to absorb both costs. The way out is to retrieve less but better, and to compress what you retrieve.

The maths of prefill

Let us put numbers on “prefill grows with the context”. For a context of $C$ tokens, a transformer layer does two classes of work:

Linear projections (QKV, attention output, FFN): each token is multiplied by fixed-size weight matrices. The cost is $O(C)$ in FLOPs, linear in the number of tokens.
Attention ($QK^\top$ and the multiplication by $V$): each token attends to all the others. The cost is $O(C^2)$, quadratic in the number of tokens.

The total prefill cost per layer has the form:

$$\text{FLOPs}_{\text{prefill}} \approx \underbrace{a \cdot C}_{\text{projections}} + \underbrace{b \cdot C^2}_{\text{attention}}$$

with $a$ and $b$ constants that depend on the model dimension. For moderate contexts (a few thousand tokens) in an SLM, the linear term still dominates or is comparable to the quadratic one; the quadratic term becomes dominant at long contexts. What matters: if you compress the context $C \to C/k$, the linear term falls $\times k$ and the quadratic one falls $\times k^2$. Compression is the only lever that attacks both terms at once, and it attacks the worse one disproportionately.

Numerical example: TTFT before and after compressing, RTX 4090

Let us model TTFT as the time to process the prefill tokens at a given prefill throughput. Take an RTX 4090 (24 GB, Ada Lovelace) serving a quantised SLM, with a prefill throughput of ~5,000 tok/s (an illustrative figure; the real value depends on the model, the quantisation and the batch, so measure it, do not assume it).

Let there be a retrieved context of 4,000 tokens (8 chunks of ~500 tokens). Approximating TTFT as dominated by the context prefill:

$$\text{TTFT}_{\text{before}} \approx \frac{4000 \text{ tok}}{5000 \text{ tok/s}} = 0.80 \text{ s}$$

Now we compress that context to 1,000 tokens ($k = 4$). Prefill throughput is not constant with $C$, it drops a little at long contexts because of the quadratic term, but taking the conservative linear tokens/throughput approximation:

$$\text{TTFT}_{\text{after}} \approx \frac{1000 \text{ tok}}{5000 \text{ tok/s}} = 0.20 \text{ s}$$

TTFT falls from 0.80 s to 0.20 s, a $4\times$ reduction in the linear part. But the FLOPs arithmetic is even more favourable in the attention component: that part of the work falls $\sim k^2 = 16\times$. In practice total TTFT does not fall 16× because the cost is not purely quadratic at this scale, but the real reduction lies between 4× and something larger depending on how much attention weighed, and the aggregate compute saving (what pays the electricity bill and frees the GPU for another request) is substantially bigger than the plain 4× of the token count.

The argument generalises: compressing the context by a factor $k$ reduces TTFT by at least $\sim k\times$ and the attention cost by $\sim k^2\times$. For an SLM, where TTFT is often the SLA that matters, this is the difference between an assistant that answers instantly and one that feels slow.

The five levers to resolve the tension

The strategy is not “retrieve less and settle”. It is retrieve aggressively from the index and then distil aggressively what you retrieved before it reaches the SLM. Five levers, in order of application within the pipeline.

1. Aggressive reranking: precision over recall

The initial retriever (dense, sparse or hybrid) optimises recall: it brings back 50–100 candidates so as not to leave anything out. The reranker, a cross-encoder that sees the query and the document together, optimises precision: it reorders those candidates and you keep the 3–5 best. For an SLM this is not a luxury, it is structural: since the model uses long context badly, every chunk that goes in must earn its place. Better 4 chunks of very high relevance than 15 mediocre ones. The detail of hybrid retrieval and reranking is in Reranking and hybrid retrieval; here the rule is enough: maximise recall in the retriever, maximise precision in the reranker, and inject few.

2. Context compression: distilling the crib sheet

Once you have the best chunks, they still contain filler: padding sentences, redundancy, context irrelevant to the specific query. Context compression trims them before injection:

Extractive compression (LLMLingua / LongLLMLingua style, Jiang et al. 2023): a small model scores the perplexity or relevance of each token/sentence with respect to the query and removes the low-information ones, keeping the densest extractive subset. It reduces tokens without a second large generative model in the loop. LongLLMLingua adds position-aware reordering to mitigate lost in the middle.
Abstractive compression: a model summarises the retrieved chunks into shorter text. More aggressive in token reduction, but it introduces a generative step (cost and possible loss of fidelity).
Soft prompts / context distillation: compressing the retrieved context not into text, but into a handful of embeddings/soft tokens that the model consumes directly. It reduces the number of prefill tokens to a minimum, at the price of a trained, model-specific component.

The key point connects with the maths above: compressing what you retrieved by a factor $k$ reduces prefill tokens $\times k$, and therefore TTFT $\sim\times k$ and the attention cost $\sim\times k^2$. It is the lever with the best return when long context is the bottleneck.

3. Prefix caching of stable context

Not all the context changes between requests. System instructions, definitions, recurring reference documents, schemas: these are stable prefixes. Prefix caching stores the already-computed KV cache of those prefixes and reuses it, so that prefill only processes the new part (the query and the specific chunks). If 60 % of your context is stable, you save 60 % of that segment’s prefill on every hit. For it to work, the stable context must go at the start of the prompt (the KV cache is prefix-dependent) and it pays to maximise the hit rate; the hit-rate engineering detail is in Prefix cache hit rate. It combines especially well with RAG: retrieved documents that repeat across sessions get cached once.

4. Semantic caching of answers

A layer in front of the model: if a query is semantically equivalent to one answered before (embedding similarity above a threshold), return the cached answer and skip the model entirely, retrieval, prefill and decode included. In real workloads with long tails of repeated or near-repeated questions (FAQ, support), the saving is enormous because it eliminates the full cost, not just the prefill one. The trap is the threshold: too lax and you serve wrong answers to similar-but-different questions. The design is in Semantic caching for RAG.

5. Structured output and function calling: lean on tools, not on memory

The last lever changes what the SLM depends on. Instead of asking it to know a fact (its weak point), make it call a tool that knows it: a database query, an API, a calculator, a validator. Structured output (forcing JSON conforming to a schema) and function calling turn the SLM into an orchestrator that extracts arguments from the context and delegates the computation or the query. A reasonably capable SLM emits a well-formed tool call far more reliably than it recalls a specific fact. This reduces the pressure on parametric knowledge and on retrieval: for structured, fresh data (prices, inventory, states), querying beats retrieving text and beats memorising. The fundamentals are in Structured output and Function calling.

The complete pipeline

The five levers are not alternatives: they chain together. The flow, with the token counter falling at each step:

The order matters. Retrieving aggressively (high recall) before filtering guarantees that the right material is among the candidates; reranking and compressing afterwards guarantees that only the dense and relevant material pays the prefill toll; caching wraps everything so as not to repeat work. The SLM only sees the final crib sheet, short and ordered.

Implications for on-premise inference

The mental trap to avoid: treating the SLM as a large model with less quality. It is not. It is a different cost profile that rewards a different design. Three practical consequences:

The token budget is a first-class resource. With a large model with a 128K window, “putting in a bit more” is cheap relative to the model. With an SLM, every context token shows up in TTFT and in quality. Treat context size as a quantity to minimise subject to covering the answer, not to maximise.
The investment is worth it precisely because the model is cheap. Reranker, compressor and caches add complexity, but the model they serve is economical enough to run many replicas. The bottleneck shifts from the model to the data pipeline, which is exactly where you want it.
Retrieving does not replace adapting; they combine. For deep, recurring domain knowledge, adapting the SLM with LoRA (see the sibling QLoRA and aggressive multi-LoRA) can put part of the knowledge “into the weights” cheaply, reducing what has to be retrieved. Aggressive RAG and aggressive adaptation do not compete: the first gives freshness and citability, the second gives fluency and domain formatting. A good design uses both.

On the RTX 4090 (24 GB, Ada Lovelace)

The canonical scenario: a quantised SLM (4B–8B in INT4/FP8) fits with room to spare, leaving VRAM for a generous KV cache, essential for prefix caching, and for the reranker (a cross-encoder of a few hundred MB). The LLMLingua-style extractive compressor runs on a separate small model or on CPU. The TTFT calculation above (0.80 s → 0.20 s compressing 4× at ~5,000 tok/s) is representative of this card. The rule of thumb: if TTFT goes above your SLA, the first adjustment is to compress the context, not to change the model.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

With 320 GB and native FP8 prefill is much faster, so the temptation is to relax the token discipline. That is not quite advisable: the lever changes from TTFT to aggregate throughput. Compressing the context not only speeds up each request but frees prefill compute to serve more requests per GPU; compute-bound prefill is exactly the resource that saturates first under load. Here prefix caching and semantic caching, shared across replicas, are what pay off most: at high QPS, the prefill work you avoid by caching is pure throughput you gain. The SLM is still the cheap reasoning engine; the difference is that you now run many in parallel and the data pipeline is what decides how many requests fit.

What we have not covered

Evaluating compression: how to measure that compressing $k=4$ does not throw away correct answers (faithfulness, answer recall over a set of questions with ground truth).
Query-aware versus query-agnostic compression: compressing before or after knowing the question changes what can be cached and what can be discarded.
Chunking and granularity: chunk size interacts with reranking and compression; that is left for the corpus curation post.
Multi-hop and agents: when a question requires several retrieval rounds, the token budget is split across hops and compression discipline becomes critical.

References

Lewis, P., et al. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks. NeurIPS 2020. https://arxiv.org/abs/2005.11401
Liu, N.F., et al. Lost in the Middle: How Language Models Use Long Contexts. TACL 2024. https://arxiv.org/abs/2307.03172
Jiang, H., et al. LLMLingua: Compressing Prompts for Accelerated Inference of Large Language Models. EMNLP 2023. https://arxiv.org/abs/2310.05736
Jiang, H., et al. LongLLMLingua: Accelerating and Enhancing LLMs in Long Context Scenarios via Prompt Compression. ACL 2024. https://arxiv.org/abs/2310.06839

Aggressive quantisation (state of the art): from 4-bit to ternary

Tue, 09 Jun 2026 02:10:00 +0000

This post is the direct continuation of Quantization for LLM inference, which covers the “solved” regime (FP8, INT4 with GPTQ/AWQ). Read it first: here I assume the scale+zero-point maths, what GPTQ and AWQ do, and the PTQ/QAT distinction. What we add is the sub-4-bit frontier, where scalar post-hoc quantisation stops working and you have to change tools.

TL;DR

There is a sharp dividing line around 4 bits. Above it, quantising is a solved problem: INT8 is indistinguishable from BF16, and INT4 with a good method (AWQ, GPTQ) loses 1-2 MMLU points and little else. The method is still the usual one, take each weight, scale it, round it to a short integer, and it works. Below 4 bits, that method collapses: at 2 bits, naive scalar quantisation can double perplexity. The reason is geometric, each weight has only 4 possible values and the rounding error stops being negligible, and the way out is not “round better”, it is to change the representation. State-of-the-art 2-bit methods (AQLM, QuIP#, QTIP) stop quantising individual weights and quantise vectors of weights against dictionaries (codebooks), and they “whiten” the weight matrix to spread its energy and flatten outliers (incoherence processing). Ternary is something else again: BitNet b1.58, with weights in {-1, 0, +1} (~1.58 bits), is not PTQ, it is a model trained natively under that constraint, and it changes the arithmetic of the matmul from multiplications to additions and subtractions, hitting the compute ceiling and the memory ceiling at once. The mental rule: at ≥4-bit you compress the photo; below 4-bit you have to repaint it.

The analogy: the JPEG that cannot be compressed any further

In the quantization post we used the JPEG with an edge detector to explain INT4. The analogy still holds here, but it has to be pushed to its limit.

A JPEG at quality factor 90 is indistinguishable from the original. At quality 60 you notice it a little, but it is still “the same photo”. At quality 30 the 8×8 blocks appear, the haloes around edges, the banding in gradients. At quality 10 the image is destroyed: you can tell there was a face, but the details have disappeared under the artefacts. And here is the key: no JPEG encoder exists that can compress to quality 10 without those artefacts, because the JPEG algorithm (blockwise DCT + coefficient quantisation) has an information floor below which its own mechanism introduces the noise.

What do you do if you need the photo at that file size and you need it to still look good? You do not compress the original harder. You repaint the photo knowing in advance that it is going to live compressed: an illustrator redraws it with clean lines, a reduced palette, no subtle gradients, an image designed to survive brutal compression. The result at “10 KB” looks infinitely better than the original JPEG crushed to 10 KB, because it is not the same process: one destroys existing information, the other generates new information already adapted to the constraint.

That is exactly the frontier of this post:

Scalar PTQ (≥4-bit) = compressing the JPEG. Up to a certain ratio, it is still the same photo.
State-of-the-art vector PTQ (2-bit: AQLM, QuIP#, QTIP) = a far more sophisticated image codec (dictionaries, decorrelating transforms) that stretches the compressible ratio a bit further before the collapse.
Native ternary (BitNet b1.58) = repainting the photo. You are not compressing an existing BF16 model; you are training a new one that is born ternary.

The map of the frontier, bit by bit

Quantising a model is deciding how many distinct values each weight can take. With b bits per weight there are 2^b possible values. The central question is: past what b is the number of values so small that rounding destroys the model?

Bits	Values/weight	State of the art	Method required	Typical loss vs BF16
8	256	Solved	RTN, SmoothQuant, FP8	~0 (indistinguishable)
4	16	Solved	AWQ, GPTQ	1-2 pp MMLU, +0.1-0.3 PPL
3	8	Small degradation	tuned GPTQ/AWQ, GGUF Q3_K	3-5 pp MMLU
2	4	Serious unless SOTA	AQLM, QuIP#, QTIP (non-scalar)	scalar: collapse; SOTA: 4-8 pp
1.58	3 (ternary)	Native only	BitNet b1.58 (QAT/native training)	n/a (not PTQ)
1	2 (binary)	Research	native, dubious claims	large / no clear methodology

The three transitions that matter:

8 → 4 bits: nothing breaks. With 16 levels per weight and one scale per block of 128, the rounding error is small relative to the dynamics of the weights. GPTQ compensates for the error by propagating it to the neighbouring weights; AWQ protects the ~1 % of salient channels. The model barely notices. This is in the previous post.

4 → 2 bits: the elbow. Two things happen at once here. First, with only 4 levels, the scalar quantiser can no longer represent the weight distribution, which is roughly Gaussian with long tails, without a proportionally enormous rounding error. Second, and more subtly: the quantisation error stops being “small noise that the model absorbs” and becomes structured, systematically biasing the activations. Naive scalar PTQ at 2 bits on a Llama 8B typically doubles perplexity or worse. That is the elbow of the curve.

2 → 1.58 bits: a change of nature. You do not cross this with a better compression method. You cross it by training the model with the constraint from the start. It is a discontinuity: to the left you are doing PTQ, to the right you are doing training.

Why scalar PTQ collapses below 4 bits

The scalar quantiser has a fundamental limitation: it quantises each weight separately, ignoring the fact that the weights of a row or column are correlated and that the error of one could be compensated by another. At 4 bits this matters little; at 2 bits it is lethal. There are three possible attacks, and the state-of-the-art methods use all three.

1. Vector quantisation: dictionaries instead of scales

Instead of mapping each weight to one of 4 values, group the weights into vectors (say, of 8 weights) and map each vector to the nearest entry of a learned dictionary (codebook). If the dictionary has 256 entries, encoding a vector of 8 weights costs 8 bits (the index) → 1 bit/weight, but every “reconstructed value” is a point in an 8-dimensional space chosen to minimise the error over the real distribution of the weights.

The advantage is information-theoretic: a vector dictionary can place its reconstruction points where the weights actually are (in clusters), whereas the scalar quantiser is forced to put its 4 levels on a regular grid, wasting resolution on empty regions. It is the difference between a road map with a uniform grid and one that puts more detail where the cities are.

AQLM (Additive Quantization of Language Models, arXiv:2401.06118) takes this to the extreme with additive quantisation: each weight vector is reconstructed as the sum of several codes from several dictionaries (multi-codebook). It is more expressive than a single dictionary because the number of combinations is the product of the sizes, not the sum. AQLM was one of the first methods to make 2-bit “usable” (not collapsed) on large models, at the cost of an expensive calibration process and specialised inference kernels.

2. Incoherence processing: whitening the matrix

The second attack targets the outliers. The weight matrices of a transformer have a few entries (and a few channels) with a magnitude much larger than the rest. Those outliers dominate the quantiser’s range: if you have to represent a weight of magnitude 8 while the rest are of magnitude 0.5, your scale stretches to cover the 8 and you waste almost all the resolution.

Incoherence processing (the core idea of QuIP and QuIP#) attacks this by multiplying the weight matrix W by random orthogonal matrices on the left and on the right: W' = U W V^T. Because U and V are orthogonal, the operation is invertible and the maths of the product can be undone at inference time by absorbing it into the neighbouring layers (just as AWQ absorbs its scales). But the rotation spreads the energy: an “incoherent” matrix has its values distributed almost uniformly, with no concentrated outliers, because mixing coordinates with a random rotation flattens the distribution (it is, in essence, the central limit theorem acting on linear combinations). A matrix without outliers quantises far better at 2 bits. It is the equivalent of “whitening” a signal before digitising it.

QuIP# (arXiv:2402.04396) combines incoherence processing with E8 lattice codes: rather than an arbitrary dictionary, it uses the E8 lattice (an optimal sphere packing in 8 dimensions, the best known). Quantising vectors of 8 weights against the E8 lattice gives the smallest reconstruction error possible for a given bit density, because E8 is literally the most efficient way to place points in 8D. It is classical coding theory applied to LLM weights.

3. Coding with memory: trellis

QTIP (arXiv:2406.11235) adds the third attack: trellis-coded quantisation. Instead of quantising each vector independently, it models the sequence of weights as a path through a trellis (the same structure as the convolutional codes of telecommunications) and picks the optimal code sequence with the Viterbi algorithm. The intuition: introducing memory between successive quantisations allows correlated errors that cancel out, instead of independent errors that accumulate. QTIP, on top of incoherence processing, improves on QuIP# in quality at 2-3 bits while keeping fast inference kernels.

The idea common to all three: stop quantising scalars and start quantising vectors with dictionaries, and decorrelate the matrix before doing so. None of them is “rounding better”; all three change the representation at the root. That is why, below 4 bits, a flag in vLLM is no longer enough: you need co-design of the quantisation method and the inference kernel.

Native ternary: BitNet b1.58

Here we change continents. Everything above is PTQ: it starts from a trained BF16 model and compresses it. BitNet’s ternary compresses nothing.

BitNet b1.58 (arXiv:2402.17764) trains a transformer from scratch where every weight is constrained to {-1, 0, +1} throughout training. Three values ⇒ log₂(3) ≈ 1.58 bits/weight. Quantisation is not a later step: the linear layers (BitLinear) quantise their weights to ternary in the forward pass of every training step, and gradients flow through a straight-through estimator. The model learns to work with ternary weights. This is QAT taken to the extreme: not a short robustness fine-tune, but the constraint present from the first token of training.

That difference is what dodges the elbow of the curve. PTQ at 2 bits tries to find the best ternary or quaternary approximation of a model that was trained expecting full precision, and that model has “fragile” weights that depend on nuances 2 bits cannot capture. BitNet, by contrast, never had those nuances: its weights were born ternary, so the network distributed its representational capacity in a way compatible with the constraint. It is repainting the photo instead of compressing it.

What changes is not just the memory, it is the arithmetic

The most underrated point about BitNet: with weights in {-1, 0, +1}, multiplication disappears from the matmul. Multiplying an activation x by a ternary weight w is trivial: if w = +1 you add x, if w = -1 you subtract x, if w = 0 you do nothing. The dominant operation of a transformer, the matrix-vector product, goes from a sea of floating point multiply-accumulate (MAC) to integer additions and subtractions.

This matters because it connects to the roofline. As explained in The inverted roofline of small models, LLM inference has two ceilings: memory (HBM bandwidth for loading weights) and compute (tensor core FLOPs). Normal quantisation (INT4, FP8) attacks only the memory ceiling: the weight takes up less space, but to multiply it you dequantise to FP16 and do the same multiplication as always. Ternary attacks both ceilings at once: the weight occupies 1.58 bits (memory) and the operation is an addition instead of a multiplication (compute). That is why BitNet needs its own kernels, bitnet.cpp, that execute the ternary matmul without ever going through FP16; a kernel that dequantised to FP16 in order to multiply would throw away half the advantage.

The honest trade-off: BitNet b1.58 is training from scratch. You cannot “convert your Llama 8B to BitNet”. If you want ternary, you train (or use) a natively ternary model, with everything that implies in pre-training cost and weight availability. Today it is a research line with published models at modest scales, not a drop-in replacement for your current serving.

QAT as the bridge between PTQ and native

Between “compressing post-hoc” (PTQ) and “training natively ternary” (BitNet) there is a middle point: QAT (Quantization-Aware Training). You take an already trained model and do a short fine-tune with the quantisation operations inside the loop, so that it learns to be robust to low bit counts without paying for a full pre-training.

Gemma 3 publishes official QAT variants for exactly this: models that, after the QAT fine-tune, hold up at INT4 with far less quality loss than pure PTQ on the same model. The cost is a training cost (GPU hours to days on an existing model), not an inference cost. For INT4 with QAT you recover almost all the quality; for 2-bit, QAT helps but the terrain is still difficult; for ternary, QAT stops being a “short fine-tune” and becomes native training (BitNet).

The decision hierarchy:

PTQ = default at ≥4 bits. Minutes to hours, without touching training weights. It covers 90 % of production.
QAT = when PTQ loses too much and the difference matters. Low bits (2-3), or sensitive models. You pay for a fine-tune.
Native (ternary) = when you want to go below 2 bits and change the arithmetic. You pay for pre-training. It only makes sense if you control the model from its creation.

The maths that matter: footprint and how many fit

The weight footprint is direct: bytes = (bits/param / 8) × N, with N the number of parameters. For an 8B model:

Level	bits/param	8B footprint	Ratio vs BF16
BF16	16	16.0 GB	1.0×
INT8	8	8.0 GB	2.0×
INT4	4	4.0 GB	4.0×
3-bit	3	3.0 GB	5.3×
2-bit	2	2.0 GB	8.0×
1.58-bit (ternary)	~1.58	~1.6 GB	~10×

(Real ternary takes up slightly more than 1.58 bits/param because you have to pack 5 ternary values into 8 bits, 5 × log₂(3) ≈ 7.92 bits, and because norms and embeddings usually stay at higher precision. The ~1.6 GB figure for 8B is the right order of magnitude.)

How many 8B models fit on an RTX 4090?

An RTX 4090 (24 GB, Ada Lovelace) has 24 GB. Reserving ~4 GB for KV cache and activations leaves 20 GB for weights. How many distinct 8B models fit loaded at the same time:

Level	8B footprint	Models in 20 GB	Comment
BF16	16.0 GB	1	one, with scant headroom left
INT8	8.0 GB	2	two different models
INT4	4.0 GB	5	the solved regime; quality ~lossless with AWQ
3-bit	3.0 GB	6	small degradation already visible
2-bit	2.0 GB	10	only viable with AQLM/QuIP#/QTIP
1.58-bit	~1.6 GB	~12	natively ternary models only

The arithmetic is seductive, from 1 to 12 models on the same card, but it should be read sceptically. Jumping from INT4 (5 models, almost lossless) to 2-bit (10 models) doubles the capacity, but only if you use a state-of-the-art method and accept 4-8 MMLU points. And the jump from 2-bit to ternary (10 → 12) is marginal in memory terms: ternary is not justified by footprint against a state-of-the-art 2-bit, it is justified by the arithmetic (the compute ceiling) and because being native it avoids the quality elbow. If your only metric is “how many GB it takes”, state-of-the-art 2-bit already gives you nearly everything. Ternary is for when you also want the compute saving.

The conceptual curve: perplexity vs bits

Three readings of the curve. One: to the right of 4 bits, the three branches are almost stuck together and almost flat, the solved regime. Two: the red branch (naive scalar PTQ) has a brutal elbow between 3 and 2 bits; that is where it doubles perplexity. The blue branch (state-of-the-art vector PTQ) flattens that elbow, it does not remove it, but it makes it tolerable down to 2 bits. Three: the green point of native ternary is on neither of the two PTQ curves, because it is not obtained by compressing: it is obtained by training, and that is why it can fall below the elbow without paying the quality price that any PTQ pays at that bit density. It is the difference between the crushed JPEG and the repainted photo.

Mandatory scepticism: “lossless” 1-bit and benchmarks without methodology

Three warnings for reading the aggressive-quantisation literature:

“1-bit lossless” almost always comes with small print. Pure binary {-1, +1} (1 bit) loses the ability to represent zero, which matters in transformers (many weights are effectively null). That is why the true state of the art at low density is ternary (1.58 bits), not binary: the zero is worth its extra 0.58 of a bit. When a paper announces “1-bit”, it pays to check whether (a) it really is 1 bit or 1.58 rounded down in the headline, (b) “lossless” is measured in WikiText perplexity (easy) or on reasoning benchmarks (where the collapse shows up), and (c) it compares against a baseline of the same effective size or against a much larger model in order to inflate the advantage.

Flat perplexity ≠ preserved quality. Perplexity on a generic corpus is the metric most forgiving of aggressive quantisation. A 2-bit model can have a perplexity almost identical to BF16 and at the same time drop 10 points on GSM8K or on a code benchmark, because multi-step reasoning amplifies errors that average perplexity does not see. Distrust any sub-4-bit claim that only reports perplexity. As we already said in the quantization post, the loss has to be measured on the target task.

Hardware comparability. The “X times faster” numbers for ternary only apply with the specialised kernels (bitnet.cpp) and on hardware where addition/subtraction arithmetic genuinely wins. On a GPU with tensor cores designed for FP16/FP8, a naive ternary kernel can be slower than a well-optimised INT4, because it wastes the silicon. The ternary advantage is real, but it is a co-design advantage (model + kernel + sometimes hardware), not a flag you switch on over your current stack. Any benchmark that does not specify the exact kernel and hardware is noise.

Implications for on-premise inference

On the RTX 4090 (24 GB, Ada Lovelace): the practical regime today is still INT4 AWQ for 7-14B models, solved, almost lossless, natively supported. State-of-the-art 2-bit (AQLM/QuIP#/QTIP) is viable and lets you load larger models or more models at once, but it demands the specific kernels of each method and an expensive calibration, and it pays in quality. It makes sense when VRAM is the bottleneck and you accept the trade-off; not as a default. Ternary on a 4090 is experimental: without tensor cores designed for ternary addition and subtraction, the compute advantage is diluted, although the memory saving remains.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8): here the default is FP8 (almost indistinguishable quality, native throughput) or INT4 AWQ for models that do not fit in FP8. State-of-the-art sub-4-bit is for serving enormous models (200B+) when neither FP8 nor INT4 fits with the KV cache headroom you want, at the cost of quality and kernel complexity. Native ternary, today, is a research object rather than a production one: its promise, hitting both roofline ceilings, is greater on CPU/edge (where there are no FP8 tensor cores to exploit) than on an H100 cluster, which already has dedicated FP8 hardware.

The rule of thumb, June 2026: ≥4-bit is settled engineering; state-of-the-art 2-bit is a real lever but with a cost in method and in quality; ternary is an architectural bet, not a deployment tweak.

References

Ma, S. et al. The Era of 1-bit LLMs: All Large Language Models are in 1.58 Bits (BitNet b1.58). https://arxiv.org/abs/2402.17764
Egiazarian, V., Panferov, A., Kuznedelev, D. et al. Extreme Compression of Large Language Models via Additive Quantization (AQLM). https://arxiv.org/abs/2401.06118
Tseng, A., Chee, J., Sun, Q., Kuleshov, V., De Sa, C. QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks. https://arxiv.org/abs/2402.04396
Tseng, A., Sun, Q., Hou, D., De Sa, C. QTIP: Quantization with Trellises and Incoherence Processing. https://arxiv.org/abs/2406.11235
Frantar, E., Ashkboos, S., Hoefler, T., Alistarh, D. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers. https://arxiv.org/abs/2210.17323
Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., Han, S. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration. https://arxiv.org/abs/2306.00978
Google DeepMind. Gemma 3 QAT (Quantization-Aware Training) models — official blog: https://developers.googleblog.com/en/gemma-3-quantized-aware-trained-state-of-the-art-ai-to-consumer-gpus/
Microsoft. bitnet.cpp — 1-bit ternary inference kernels: https://github.com/microsoft/BitNet

Test-time quantization: quantising on the fly with no calibration dataset

Tue, 09 Jun 2026 02:00:00 +0000

This post is the natural continuation of Quantization for LLM inference, which is worth reading first: that is where GPTQ, AWQ, scale + zero-point and why activation outliers are the central problem all live. Here we are not discussing how many bits to use, but when and with what information the scales are computed: offline against a corpus (PTQ) or on the fly against real traffic (TTQ).

TL;DR

Activation-aware quantisation (AWQ, SmoothQuant) decides which channels to protect by measuring the magnitude of the activations over a calibration dataset in an offline pass, before deployment. The implicit assumption is that this corpus represents future traffic. But activation outliers, the channels with 10-100× the median magnitude that dominate the quantisation error, depend on the input: they change with the domain, the language and the client’s distribution. When real traffic drifts away from the calibration, the fixed scales stop being optimal and quality drops. Test-time quantization (TTQ) removes the corpus and the offline pass: it derives the activation-aware scales at inference time, from the activations that are actually observed, per token or per batch. The trade-off is honest and not minor: it introduces runtime overhead, computing statistics, detecting outliers, recomputing scales at every step, which competes directly with the saving from quantising. On small models that overhead weighs proportionally more, because the forward is short and the fixed per-step costs dominate (the framing is in the inverted roofline for SLMs). TTQ is orthogonal to the format: it is not a competitor to INT4 or FP8, it is a different way of deriving s. It pays off when there is no calibration pipeline, when the traffic distribution is shifting or unknown, and in multitenant settings where no representative corpus exists.

You are here: DEPLOY

The analogy: the tailor who takes measurements versus off-the-peg sizes

A clothes shop has two ways of dressing a customer.

The first is to sell off-the-peg sizes. At some point the factory measured an “average customer”, a mean mannequin built from a population sample, and cut the garments to those measurements. When a customer walks in, you hand them the size that comes closest. It is extremely fast: the garment is already sewn, it just gets handed over. The problem appears when the customer does not look like the average mannequin: if their shoulders are much broader than average, their particular outlier, the standard size pulls or leaves spare fabric, because it was cut to protect other areas. This is offline calibrated PTQ: AWQ measured the importance of each channel over a corpus and fixed the scales once and for all; fast at inference time, but blind to the specific customer.

The second is the tailor who takes measurements on the spot. When the customer walks in, the tailor gets out the tape, measures that customer, spots where their particular bulk is and adjusts the cut to their real anatomy. The result fits better, especially for customers who fall outside the mould. But every customer costs time: measuring, marking, deciding. This is TTQ: the scales are derived on the fly from the activations that the input actually generates.

The analogy holds up in three details:

The average mannequin = the calibration dataset. If the population walking into the shop resembles the mannequin, the sizes work; if not, they fail at the extremes.
Measuring every customer = computing activation statistics per token/batch. A better fit, but a fixed cost paid on every garment.
The broad shoulders = the outlier activation channels. They are precisely the areas where the fit matters and where the generic size gets it most wrong.

The tailor wins when the customers are varied or unknown. The tailor loses when you have a homogeneous population and a mannequin that represents it well: there, paying for measurement on every customer is a waste of time.

The problem TTQ solves: fixed calibration ages with the traffic

Let us recall from the quantization post what AWQ and SmoothQuant do exactly. They do not quantise all channels equally: they identify the ~1 % of channels whose activations have a large magnitude, the salient channels, and protect them by scaling them before quantising. To measure that importance they need to see activations, and they see them over a calibration dataset (128-512 samples, typically WikiText or a slice of the domain) in an offline pass prior to deployment.

The assumption is strong: that the activation distribution of the calibration corpus represents that of production traffic. Two reasons why that assumption breaks:

Activation outliers depend on the input. They are not a fixed property of the model the way weights are. The channel that is an outlier when processing C++ code may not be one when processing conversational Arabic or log JSON. The magnitude and position of the peaks change with the domain, the language and the input format.
Real traffic is rarely the corpus. You calibrate with English WikiText and the client sends you support tickets in Spanish with tables pasted in. The calibration protected the channels that WikiText activated, not the ones that real traffic activates. The scales are suboptimal exactly where the client lives.

The result is distribution-dependent degradation: the quantised model keeps its quality while the input resembles the calibration and loses it as the input drifts away. The most awkward case is multitenant: if you serve clients with different domains from the same quantised model, no single representative corpus exists; any fixed calibration favours some tenants and penalises others.

The TTQ mechanism: measure the real activations and scale on the fly

TTQ (arXiv:2603.19296, March 2026) proposes deriving activation-aware quantisation at inference time, with no offline pass and no calibration dataset. The idea, in its bare conceptual form:

Step 1 — Observe. When the activation tensor X arrives at a linear layer (per token or per batch), cheap statistics are computed over the channels: a measure of central tendency (median or mean magnitude) and one of dispersion per channel. This is the equivalent of AWQ looking at its corpus, but done over the activations that really are coming in right now.

Step 2 — Detect outliers on the fly. With those statistics, the channels whose magnitude shoots up relative to the tensor median are identified; the typical criterion is a threshold of the form “magnitude > k × median”. These are the channels that, if quantised with the same scale as the rest, blow up the error.

Step 3 — Derive scales and segregate. For normal channels a scale is computed that exploits the range; for the outliers a different treatment is applied, a scale of their own, or keeping them at higher precision, in the style of on-the-fly mixed precision. It is the same philosophy as LLM.int8() (segregating outliers to FP16) or AWQ (scaling salient channels), but with the threshold and the scales recomputed over the current input, not frozen since calibration.

Step 4 — Quantise and multiply. With the fresh scales, the tensor is quantised and the GEMM runs. The activations entering the next layer compensate for the rescaling, just as in AWQ, so that the maths cancels out.

The key difference with AWQ is not in what is done (protecting activation outliers) but in when and against what: AWQ decides once, offline, against a corpus; TTQ decides at every step, on the fly, against real traffic. It is the transfer to inference of the “test-time” idea: adapting the computation to the specific sample in front of you instead of to a precomputed average.

The maths that matter

The error of quantising an outlier with the wrong scale

Let us recall affine uniform quantisation from the base post: an integer code q = round(x/s) - z with scale s and zero-point z, and reconstruction x̂ = s·(q + z). For a b-bit quantiser with a symmetric range, the scale that covers a tensor of maximum magnitude M is approximately s = M / (2^{b-1} - 1). The rounding error of each element is bounded by half a scale: |x - x̂| ≤ s/2.

Here is the outlier problem. The scale s is chosen to cover the largest value in the group. If one channel has 30× the median magnitude and you share a single scale with the rest of the tensor, that magnitude rules: M is the outlier, so s is inflated 30× relative to what the majority would need. The absolute rounding error of the normal values rises proportionally.

A concrete calculation. Take a group where the median magnitude is 1.0 and an outlier channel is 30.0, quantised to INT4 (b = 4, levels ±7):

With a shared scale, s = 30 / 7 ≈ 4.29. The rounding error of a typical value (magnitude ~1) is up to s/2 ≈ 2.14. In other words, the error on the normal values is of the order of their own value: the outlier has destroyed the resolution of everything else. Relative error of a value of magnitude 1: up to ~214 %.
Segregating the outlier (pulling it out to FP16 or giving it its own scale) and quantising the rest with M = 1, s = 1/7 ≈ 0.143. The error of a typical value drops to s/2 ≈ 0.071, ~7 % relative. Thirty times less error on the majority of the group’s weights.

That is the entire reason for the existence of activation-aware quantisation: to detect and treat separately the ~1 % of channels that, if not segregated, hijack the scale. AWQ does it against the corpus; TTQ does it against the real input. And if the channel that is an outlier in production was not an outlier during calibration, AWQ did not protect it: it quantised real traffic with the inflated scale from the case above. That is where TTQ gains precision.

The overhead: the cost of measuring at every step

The price is symmetric. Computing the per-token statistics, per-channel magnitudes, median or percentile, outlier threshold, scales, means reductions over the activation tensor that did not exist in the forward pass with frozen scales. Let us call:

T = forward time per token with fixed scales (static PTQ), in µs.
Δ = extra per-token cost of deriving the statistics and scales on the fly, in µs.

The relative overhead is simply:

$$\text{overhead} = \frac{\Delta}{T}$$

The key point is that Δ is relatively fixed per step (it depends on the number of channels and layers, not on how much “useful” work the model does), while T scales with the size of the model. That is why the ratio behaves very differently depending on the model:

Large model (say, 70B): T is large, every forward moves tens of GB of weights from HBM. If Δ ≈ 8 µs and T ≈ 800 µs, the overhead is 8/800 = 1 %. Negligible against the saving from quantising.
SLM (say, 1B): T is small, the forward per token is short. With the same Δ ≈ 8 µs and T ≈ 60 µs, the overhead is 8/60 ≈ 13 %. No longer negligible: it eats a good part of what you gained by quantising.

This connects directly to the inverted roofline for small models: on an SLM the fixed per-step costs (kernel launches, synchronisations, overheads that do not scale with the model) weigh proportionally more, because there is less useful work to spread them over. TTQ’s Δ is exactly one of those fixed costs. Going per-batch instead of per-token amortises Δ across all the tokens of the batch and lowers the relative overhead, at the cost of coarser scales; it is the first parameter to touch.

The uncomfortable conclusion: TTQ gives away robustness to distribution but spends part of the acceleration budget on measuring, and in the regime where acceleration is scarcest, the SLMs, the ones most often deployed at the edge, is where that spending hurts most. It is not free; it is a change of currency.

A note of methodological scepticism: arXiv:2603.19296 is from March 2026, very recent, and as of the date of this post there are no broad independent reproductions. Whatever speedup and quality figures circulate should be taken with the same caution as any number without published methodology: what hardware, what batch size, what real measured Δ, against what baseline (well-calibrated or badly calibrated PTQ), in what domain? The conceptual argument, robustness to distribution in exchange for per-step overhead, is solid; the concrete multipliers are pending validation.

What TTQ is NOT: marking it off from the rest of the zoo

TTQ is easily confused with neighbouring techniques. The distinction that matters is that TTQ is the how of deriving the scales, not the format nor the moment in training.

Technique	When the scales are fixed	Needs a calibration corpus	Touches training	Is it a format
Static PTQ (GPTQ, AWQ)	Offline, before deployment	Yes	No	No (it uses INT4/INT8)
QAT	During training	No (training data)	Yes (retrains)	No
FP8 end to end	At runtime, but simple per-tensor scales	Minimal / none	No	Yes (E4M3/E5M2)
TTQ	At runtime, activation-aware per token/batch	No	No	No (orthogonal to the format)

The four distinctions, one by one:

Against static PTQ (GPTQ/AWQ). Same goal (protecting outliers), the same possible format (INT4), but PTQ freezes the decisions offline against a corpus and TTQ recomputes them on the fly. TTQ is, in a sense, “AWQ without the calibration phase, paid for at runtime”.
Against QAT. QAT puts quantisation inside the training loop so that the model learns to be robust to it; it costs a retrain. TTQ does not touch training: it operates on an already trained model, at inference. They are attacks at opposite moments of the pipeline.
Against FP8 end to end. FP8 is a format with its own logarithmic range; its “dynamic scaling” computes a simple per-tensor scalar at runtime, but it does not do activation-aware per-channel outlier detection. TTQ could, conceptually, derive scales on the fly for an FP8 or INT4 quantiser: it is orthogonal to the format.
TTQ is orthogonal to the format. It decides how to obtain s, not how many bits you store q in. You can imagine “TTQ over INT4” or “TTQ over FP8”. What defines TTQ is the source of the scale, real activations on the fly, not the width of the code.

When it pays off (and when it does not)

TTQ is not a universal replacement for AWQ. It is a tool for a specific deployment profile. It pays off when:

You have no calibration pipeline. You want to deploy a quantised model now, without assembling the calibration dataset, running the offline pass or validating that the corpus represents the traffic. TTQ cuts out that whole phase: you load the model and serve.
The traffic distribution is shifting or unknown. An assistant that one day receives code and the next legal contracts in another language. No fixed calibration covers both well; on-the-fly adaptation follows the distribution without recalibrating.
Multitenant with no representative corpus. You serve the same model to clients with disparate domains. No single corpus represents all of them; any fixed calibration creates winners and losers among tenants. TTQ adjusts to each input, whichever tenant it comes from.

It does not pay off when:

You have a stable domain and a good calibration corpus. If your traffic is homogeneous and representative, offline AWQ gives you the same quality with zero runtime overhead. Paying Δ on every token to relearn what a corpus already captured is waste.
You serve SLMs with a tight latency SLA. This is exactly the case where Δ/T is high. If the model is small and TPOT matters, the overhead of measuring can wipe out the gain from quantising. Measure your real Δ before assuming it adds up.
The batch is large and compute-bound. With high concurrency the forward is no longer memory-bound and the cost of the extra reductions competes worse; at the very least, amortise Δ per batch.

Implications for on-premise hardware

On an RTX 4090 (24 GB, Ada Lovelace)

The natural case for the 4090 is the SLM, Qwen 3 1.5B, Llama 3 8B AWQ-INT4, serving at low concurrency. That is precisely the regime where TTQ is riskiest: T per token is small and the 4090 has no accelerated native FP8 (we discussed this in the quantization post), so TTQ’s extra reductions run on CUDA cores competing for the same time. Here the question is not “does it improve quality?” but “does the overhead leave me an acceptable TPOT?”. If the traffic is homogeneous, offline AWQ wins on simplicity and latency. TTQ only justifies its Δ if the input distribution is genuinely unpredictable and the degradation from fixed calibration is measurable.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8)

Here the calculation partly flips. With large models T is high and Δ/T drops to the low single-digit percentage range, so TTQ’s overhead is more digestible. The strong use case is multitenant: a cluster serving a large model to clients with heterogeneous domains, where no calibration corpus satisfies everyone. There, TTQ’s robustness to distribution has real value and the overhead is diluted in a big forward. Even so, on an H100 with native FP8, the baseline to beat is demanding: static FP8 barely loses quality (see the table in the quantization post) and costs nothing at runtime. TTQ has to show that its robustness gain on the outlier tenants outweighs what it gives away in overhead. With a paper from March 2026 and no reproductions, that demonstration is pending.

What we have not covered

The memory cost of the on-the-fly statistics: per-channel buffers, their impact on the footprint and on cache pressure.
Interaction with continuous batching: how scales are derived when a batch mixes requests from different domains in the same step.
TTQ + speculative decoding: whether the draft and the target derive scales on the fly separately, and how that affects the acceptance rate.
Numerical stability: what happens when a batch contains a one-off extreme outlier that inflates the scale for every token of that step.

References

TTQ: Activation-Aware Test-Time Quantization to Accelerate LLM Inference On The Fly (March 2026). https://arxiv.org/abs/2603.19296
Lin, J., Tang, J., Tang, H., Yang, S., Dang, X., Han, S. AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration (MLSys 2024). https://arxiv.org/abs/2306.00978
Frantar, E., Ashkboos, S., Hoefler, T., Alistarh, D. GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers (ICLR 2023). https://arxiv.org/abs/2210.17323
Xiao, G., Lin, J., Seznec, M., Wu, H., Demouth, J., Han, S. SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models (ICML 2023). https://arxiv.org/abs/2211.10438
Dettmers, T., Lewis, M., Belkada, Y., Zettlemoyer, L. LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale (NeurIPS 2022). https://arxiv.org/abs/2208.07339

Device-native architectures: fine-grained MoE and pre-attention router

Tue, 09 Jun 2026 01:50:00 +0000

This post belongs to the series on inference performance in small models. It is the architectural face of a problem we have already looked at from the compute-regime side (the SLM’s inverted roofline) and from the weight-loading side in From disk to HBM. Here the question is different: what if, instead of adapting a large model to the device, we design the model for the device from the first commit?

TL;DR

The default move to get an LLM onto a laptop, a phone or an edge box is to take a dense model built for cloud and compress it: distillation, pruning, quantisation. It is a gesture of reduction: you start from something big and take things away. SmallThinker (arXiv:2507.20984, SJTU IPADS + Zenergize AI) argues for the reverse gesture, designing from scratch, and articulates it in three pieces. First: fine-grained MoE, many small experts with very few activated per token, so that total parameters N (the capacity) decouple from activated parameters A (the compute cost per token). Second: sparse FFN, ReLU-style activation sparsity inside each block, which adds a second level of sparsity on top of the first. Third: a pre-attention router that predicts which experts will be needed before running the attention block and launches the prefetch of those weights from SSD/flash in parallel with attention compute, hiding the storage latency, which is the real bottleneck when the model does not fit entirely in RAM. The authors report SmallThinker-4B-A0.6B and SmallThinker-21B-A3B exceeding ~20 tok/s on consumer CPUs with Q4_0, using ~1 GB and ~8 GB of RAM. The numbers are interesting and the direction is right; the evaluation methodology and the quality cost of activating so little deserve scepticism, and that is what the last part is for.

The analogy: the librarian who anticipates your request

Picture a huge library with a small reading room. You are sitting in the room at a single desk: only a few books fit on it at a time (that is RAM). The bulk of the collection is in the back room, on long shelves that are slow to walk (that is the SSD/flash). And there is a librarian.

The naive method: you read, you reach a point where you need a specific book, you request it, and then the librarian gets up, walks to the back room, looks for it and comes back. Meanwhile, you wait with the page open, making no progress. Every time you need a new book, you pay the full trip to the back room. The reading room spends most of its time waiting, not reading.

The SmallThinker method: the librarian is clever and gets ahead of you. While you are still reading the chapter’s table of contents, working out what it is about and connecting ideas, which in the model is the attention block, they have already glanced over your shoulder, predicted which three or four books you are going to ask for and gone to the back room to fetch them. By the time you finish the contents and formulate the request, the books are already on your desk. You have not waited: the trip to the back room happened in parallel with your reading of the contents.

The analogy holds on four details:

The small desk is RAM; the slow back room is the SSD/flash.
The books are the MoE’s experts: only a few are on the desk at any moment.
Reading the contents is the attention block; requesting and using the books is the FFN/expert block.
The librarian who predicts and gets ahead is the pre-attention router: the prediction is made first, and the trip to fetch (the prefetch) overlaps with reading the contents (attention).

The quantitative question running through the whole post is: does the librarian get back in time? The wait is only hidden if the trip to the back room takes less than your reading of the contents. That is the condition t_{\text{attention}} \ge t_{\text{prefetch}}, and we will do it with numbers.

Compressing a dense model vs. designing for device

It is worth laying out the two approaches plainly, because they are not degrees of the same thing: they are different philosophies.

Approach A — compress a dense model built for cloud. You start from, say, a dense 7B–14B model trained to run on an RTX 4090 (24 GB, Ada Lovelace) or on a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8). To fit it into a device you apply three levers, each with its own post: distillation (you train a small student that imitates the teacher), pruning (you remove weights or whole structures) and aggressive quantisation (you go down to 4 bits or fewer). The resulting model is still dense: all its parameters are activated on every token. You have reduced the number of parameters, but the compute pattern is the cloud’s, only smaller.

Approach B — design for device from scratch. Here the device’s constraints enter the architecture, not a later compression phase. The constraints are three and very concrete:

Weak compute. A laptop CPU or a mobile SoC does orders of magnitude fewer FLOPs than a datacenter GPU. This pushes you to minimise the activated parameters per token, not the total ones.
Little RAM. Tens of GB do not fit. This pushes you to keep resident only what is essential and to stream the rest.
Slow storage. The SSD or flash you are forced to stream from has far less bandwidth than a GPU’s HBM. This turns storage I/O into the bottleneck, and pushes you to hide it.

SmallThinker is approach B taken to the detail: each of those three constraints has an architectural answer. Weak compute is attacked with fine-grained MoE + sparse FFN (minimise A). Scarce RAM is attacked with streaming from SSD (resident ≈ A + cache, not N). Slow storage is attacked with the pre-attention router (hide the I/O behind attention). It is no accident that the three pieces fit together: each one solves a constraint, and together they reinforce each other.

An important nuance, so as not to fall into the hype: approach B is neither free nor universally superior. It requires training a new model (you do not reuse existing weights), and the quality ceiling of a model with a very small A is intrinsically bounded, as we will see. The argument is not “B always wins”, but “for the device regime, B attacks the right bottlenecks, and A only attacks them in passing”.

Two levels of sparsity

The central capacity idea is old and well understood in MoE: separate capacity from compute cost. In a MoE, the model has N total parameters spread across experts, but for each token only A parameters are activated (those of the top-k experts the router picks). Compute cost per token scales with A; knowledge capacity scales with N. SmallThinker applies this idea at two superimposed levels.

Level 1 — fine-grained MoE. “Fine-grained” means many small experts instead of few large ones, with very few activated per token. Instead of, say, 8 experts of which you activate 2, you have dozens of experts of which you activate a handful. With smaller experts, the same A is spread across more possible combinations, which gives the router fine granularity and keeps A very low relative to N. The result is an aggressive N/A ratio: a lot of capacity, very little compute per token.

Level 2 — sparse FFN (ReLU-style activation sparsity). This level is orthogonal and operates inside each FFN. With a ReLU-style non-linearity, a large fraction of the intermediate layer’s neurons produce exactly zero for a given token. A neuron that comes out at zero contributes nothing to the output: its matrix-vector multiplication can be skipped. This is activation sparsity: predictable token by token, and exploitable so as not to load or multiply the weight rows/columns corresponding to inactive neurons. It is the same phenomenon exploited by work such as Deja Vu or PowerInfer; SmallThinker builds it in from the factory by choosing activations that favour it.

The combined effect, in one sentence: large N (capacity), tiny A (compute cost per token ≈ proportional to A), and on top of that, inside that A, a fraction of the multiplications is saved through activation sparsity. It is sparsity on top of sparsity.

The pre-attention router: predict and prefetch

Here is the paper’s specific piece, and the one that gives the post its name. The problem it solves is one of I/O scheduling, not of quality.

When the model does not fit entirely in RAM, the expert weights live in SSD/flash and are loaded on demand. The naive flow of a MoE layer is sequential: you run attention, then the router decides which experts are needed, then you load those experts from SSD (waiting), then you run the FFN of those experts. The loading step is pure waiting: the CPU is blocked waiting for bytes from the SSD. In the device regime, where the SSD is slow, that waiting time dominates the decode step.

The pre-attention router breaks the sequentiality by inverting the order of the decision. The observation is that the router does not need this same layer’s attention output to make a reasonable prediction of which experts will be needed: it can predict it from the state it already has before running attention. So:

Before running the layer’s attention block, the router predicts the experts that will be needed.
It launches the prefetch of those experts from SSD/flash asynchronously.
In parallel, the CPU runs the attention block, which is pure compute and does not need the SSD.
When attention finishes, the prefetched experts are (ideally) already in RAM, and the FFN proceeds without waiting.

Storage I/O has been overlapped with attention compute. It is exactly the librarian who goes to the back room while you read the contents.

The hiding condition is the inequality above: the prefetch is completely hidden if and only if

$$t_{\text{attention}} \;\ge\; t_{\text{prefetch}}.$$

If attention takes longer than loading the experts, the load is free (it was already done). If the experts are too large or the SSD too slow, t_prefetch > t_att and a wait bubble equal to t_prefetch − t_att appears. That is why the design needs A to be small (small experts → fewer bytes to prefetch → low t_prefetch) and the grain to be fine: the two things level 1 of sparsity does are not only about saving FLOPs, they are about making the prefetch fit underneath attention.

The maths that matter

Memory footprint: resident N vs. A + cache

The parameter that decides whether the model fits is how much you have to keep resident in RAM at once.

Everything in RAM. If you require all the experts to be loaded, the footprint is \approx N (all parameters, multiplied by bytes/parameter according to the quantisation). For a 21B this is prohibitive on a device.
Streaming from SSD. If you only keep the active experts resident plus a cache of the recent/likely ones, the footprint drops to \approx A + \text{cache}. The weights that are not in RAM live on SSD and get prefetched when needed. This is where the real saving is: the resident set scales with A, not with N.

The non-expert part of the model (embeddings, attention, router, layernorms) is always resident, but in a fine-grained MoE the bulk of N is in the experts, so the approximation resident ≈ A + cache + dense_part is a good one.

The prefetch calculation, with numbers

Let us put numbers on the analogy. Suppose a consumer SSD at 5 GB/s of sequential read and a quantised expert of size X MB. The time to load one expert is

$$t_{\text{1 expert}} = \frac{X \text{ MB}}{5000 \text{ MB/s}} = \frac{X}{5000}\ \text{s} = \frac{X}{5}\ \text{ms}.$$

Let us make X concrete. In SmallThinker-4B-A0.6B with Q4_0 (~0.5 byte/param effective counting block overhead), a small expert of, say, 4M parameters weighs \approx 4\text{M} \times 0.5 = 2 MB. Loading it costs t_{\text{1 expert}} = 2/5 = 0.4 ms.

Now the scheduling question: if the layer’s attention block takes Y ms, how many experts can I prefetch while attention runs? The number is

$$n_{\text{prefetch}} = \left\lfloor \frac{Y}{t_{\text{1 expert}}} \right\rfloor = \left\lfloor \frac{Y \cdot 5}{X} \right\rfloor.$$

With Y = 2 ms of attention and X = 2 MB per expert: n_{\text{prefetch}} = \lfloor 2 \times 5 / 2 \rfloor = 5 experts. That is, within that layer’s attention window the SSD manages to bring in 5 experts. If the layer’s top-k activates ≤ 5 experts, the prefetch hides them all and t_prefetch ≤ t_att: zero load latency. If the layer needed 8 experts, you would bring 5 for free and pay the load of the remaining 3 as a bubble: (8-5) \times 0.4 = 1.2 ms of waiting per layer. Hence why the design wants fine grain with a small top-k: to fit underneath the attention window.

Two critical observations about this calculation:

The 5 GB/s is idealised sequential read. The experts are scattered across the disk; random 4K reads on a consumer SSD are far slower. The effective bandwidth can be a fraction of the nominal one, which reduces n_{\text{prefetch}}. Any methodology reporting tok/s ought to say whether it measures with experts pre-sorted on disk or with realistic access.
The attention window Y shrinks with short context and at the start of generation. With short prompts, attention is cheap and may not cover the prefetch; the advantage of the overlap grows with longer sequences. Another detail an honest benchmark should break down.

Weight footprint: why they report ~1 GB for a 4B

Let us do the 4B calculation in Q4_0. Quantisation to 4 bits ≈ 0.5 byte/param, plus a small overhead of per-block scales (Q4_0 adds one FP16 scale every 32 weights, ~0.56 effective byte/param). So:

$$4\text{B} \times 0.5\ \text{B/param} \approx 2\ \text{GB}.$$

That is, the full model in Q4_0 takes up ~2 GB on disk. But the authors report ~1 GB of RAM. A contradiction? No, and understanding why is understanding the design:

Not all experts are resident. Only the activated ones (A = 0.6B) plus a cache fit in RAM; the rest lives on SSD and is streamed. 0.6\text{B} \times 0.5 \approx 0.3 GB of active experts, plus the dense part (attention, embeddings, router) and a cache of hot experts.
Sparse FFN reduces the work and the useful resident set. The neurons that come out at zero do not need to be materialised for that token.

Adding up active experts + dense part + a reasonable cache, ~1 GB is plausible. But mind the nuance: ~1 GB is the resident set in RAM, not the total footprint in storage, which is still ~2 GB on SSD. Conflating the two, reporting a bare “1 GB”, is misleading if the reader understands “the model takes up 1 GB”. It takes up 2 GB; it keeps 1 GB in RAM. The distinction matters for a device with 2 GB of free storage: there it does not fit.

Analogously, SmallThinker-21B-A3B: 21\text{B} \times 0.5 \approx 10.5 GB on disk; 3\text{B} \times 0.5 \approx 1.5 GB of active experts, and the ~8 GB of RAM reported includes active experts + a generous cache + the dense part. The large cache is what takes it from 1.5 to ~8 GB: you keep many hot experts resident so as not to hit the SSD constantly.

The quality cost: the necessary scepticism

All the machinery above reduces the compute per token to \approx A. But A = 0.6B activated is very little. This is where the enthusiasm needs a brake:

Bounded reasoning capacity. A model that activates 0.6B parameters per token has, per token, the compute power of a 0.6B model, not of a 4B one. The total capacity N=4B helps to store more knowledge (more specialised experts), but the processing of each token is still limited by A. For tasks requiring composition and intensive multi-step reasoning, this is a real ceiling, not a detail.
The router is a single point of failure for quality. If the fine-grained router picks the wrong experts, and with fine grain there are more decisions to take, quality drops without any speed metric reflecting it. The pre-attention router makes this worse: it predicts the experts before seeing attention, with less information than a post-attention router. The authors ought to report how much quality is lost by predicting early (mismatch between the prefetched expert and the expert a post-attention router would have chosen).
The ~20 tok/s needs small print. On which CPU exactly? With what context and generation length (the advantage of the overlap depends on Y)? Cold start included or steady state? Did the SSD have the experts pre-sorted sequentially? An “exceeds 20 tok/s” without those conditions is a marketing number, not a methodological one.
A fair comparison. The right question is not “is it fast?”, but “at equal quality on an independent benchmark, is it faster or smaller than an equivalent compressed dense model?”. That requires evals the reader can reproduce, not just tok/s on the authors’ machine.

None of this invalidates the direction. Designing for device is, conceptually, the right approach: it attacks the real bottlenecks (compute, RAM, I/O) in the architecture instead of patching them afterwards. But “20 tok/s in ~1 GB” is a claim about efficiency, and efficiency only means something anchored to a level of quality measured honestly. Until that anchor is clear, the right amount of scepticism is high.

Implications for on-premise and edge inference

The SSD becomes part of the inference hierarchy. In cloud, the hierarchy is HBM → RAM. On device, the SSD/flash enters as one more level, and its bandwidth and random-access latency become first-order performance parameters. This connects with From disk to HBM: cold start and weight streaming stop being just a startup problem and become part of the steady state.
The heterogeneous edge box starts to make sense. In a mixed environments pattern, a device-native model like SmallThinker runs on the NUC/edge with CPU and SSD, serving locally, while the heavy work stays on the central cluster. The pre-attention router is what makes a GPU-less edge box viable.
Capacity planning changes axes. As discussed in Inference capacity planning, on device the resource to plan is not VRAM but the triad resident-RAM / SSD-bandwidth / CPU-FLOPs. A model with small A and overlapped prefetch moves the bottleneck from “does it fit in RAM?” to “does the SSD feed the prefetch in time?”.

Conclusion

SmallThinker is, above all, a change of question. Not “how do I shrink this cloud model so it fits on the device?” but “what would the model look like if I designed it for the device from the first parameter?”. The answer, fine-grained MoE to decouple N from A, sparse FFN to save inside A, and a pre-attention router that hides storage I/O underneath attention, attacks the device’s three constraints (compute, RAM, I/O) in the architecture, not in a later compression phase. The key condition, t_att ≥ t_prefetch, explains why the pieces fit: fine grain does not only save FLOPs, it makes the prefetch fit underneath attention. The reported numbers (~20 tok/s, ~1 GB / ~8 GB of RAM) are promising and the direction is solid; the cost of activating so little and the lack of methodological detail on quality call for caution. Designing for device is the right bet; measuring it honestly is the unfinished business.

References

Equipo SmallThinker (SJTU IPADS + Zenergize AI). SmallThinker: A Family of Efficient Large Language Models Natively Trained for Local Deployment. arXiv:2507.20984. https://arxiv.org/abs/2507.20984
Repositorio oficial SmallThinker: https://github.com/SJTU-IPADS/SmallThinker
Self-Speculative Decoding for On-device MoE Acceleration. ACM The Web Conference (WWW) 2026. doi:10.1145/3774904.3792218. https://doi.org/10.1145/3774904.3792218
Liu, Z. et al. Deja Vu: Contextual Sparsity for Efficient LLMs at Inference Time. ICML 2023. https://arxiv.org/abs/2310.17157
Song, Y. et al. PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPU (sparse activation + hot/cold experts). SJTU IPADS, 2023. https://arxiv.org/abs/2312.12456

Self-speculative decoding: the model that gets ahead of itself

Tue, 09 Jun 2026 01:40:00 +0000

This post is the direct companion to Speculative decoding: the secretary who types ahead. There, draft and target are two different models; here they are the same model at two depths. Read that one first: we take rejection sampling, the 1/(1-α) ceiling and the speedup formula as known, and all we change here is what the draft is.

TL;DR

Classic speculative decoding demands a pair: a cheap draft model proposes γ tokens and an expensive target verifies them in a single parallel forward pass. On large models the draft can be 1 % of the target and fit comfortably. On small models (SLMs, 1B–8B) that recipe breaks on two fronts: a draft that is 1/10 of a 3B is a 0.3B that barely gets anything right (α collapses), and loading a second model, however small, doubles the pieces to maintain and eats VRAM that is not spare on a 4090 or on device. Self-speculative decoding solves both: the draft is the model itself, run shallowly. A model with L layers produces draft tokens by exiting at an intermediate layer k < L (early-exit) or by skipping a subset of layers (layer-skip), and then verifies those tokens with the full forward pass through the L layers. Because draft and verify share weights and share the KV cache of the common layers, the extra memory cost is zero: there is no second model, no second KV cache, nothing new to load. The price is that the early-exit draft is more expensive than a tiny external draft (it walks k/L of the model instead of 1 %), so the relative cost c goes up. The honest trade-off: with a well-trained dedicated draft (EAGLE-3) that fits in memory, its α is usually higher and it wins; self-spec wins when there is no trained draft, when it does not fit, or when you are on device.

The analogy: the chess player who plays by eye and then calculates

A good chess player does two things with the same brain. First they look at the board and, by eye, in half a second, propose a move that “looks right”: pattern intuition, fast recognition, the shallow layers of judgement. Then, before moving, they calculate properly: three moves ahead, the opponent’s replies, the tactical lines. That deep calculation confirms the intuition or corrects it.

The decisive part is that it is the same person acting as drafter and as reviewer. They do not hire a second, weaker player to guess the move so they can validate it; that would be classic speculative with an external draft. Here the fast draft and the slow verification come out of the same brain, traversed at two depths.

The analogy holds point by point:

The glance by eye is the early-exit forward pass: the model walks only the first k layers and emits a draft token. Fast, approximate.
The deep calculation is the full forward pass through the L layers, which verifies the draft with exact rejection sampling.
That it is the same person is the reuse of weights and KV cache: the k shallow layers of the draft are literally the same as the first k layers of the verify; what has already been computed is not recomputed.
That the final move is identical to the one the player would have chosen by always calculating deeply is the rejection sampling guarantee: output quality does not degrade (the proof is in the speculative post).

Why the external draft does not fit small models

Let us revisit the cost of classic speculative with two numbers. The speedup depends on the acceptance rate α (how often the draft is right) and on the relative cost c = T_draft / T_target. A useful draft needs high α and low c at the same time. On large models that is reachable: a 1B draft for a 70B target has c ≈ 0.015 and, if well distilled (EAGLE), α > 0.8. The product pays off.

On a small model the balance breaks:

The proportional draft is useless. If you want c ≈ 0.1 for a 3B target, your draft is around 0.3B. A generic 0.3B has a distribution so different from the 3B that α falls into the 0.3–0.5 range. And 1/(1-α) with α = 0.4 is a ceiling of 1.67 tokens/step: not even infinite γ gets you more. The prize evaporates.
Loading a second model doubles the pieces. Even if the draft is small in VRAM, it is another checkpoint to version, quantise, validate and serve, and it has its own KV cache. On an RTX 4090 (24 GB, Ada Lovelace) with a quantised 8B and a long context, the KV cache is already tight; adding a second model and its cache can force you to lower concurrency or maximum context. On device (a phone, a NUC, an edge box) there is simply no room.
A trained draft does not always exist for your exotic or fine-tuned model. EAGLE needs the draft trained on-policy against that specific target (see knowledge distillation). If your SLM is your own fine-tune, there is no official published draft.

Self-speculative attacks all three at once with one idea: do not bring a second model; use the first one at half depth.

The mechanism: early-exit as draft, full forward as verify

A transformer with L layers transforms the hidden state layer by layer at each position: h_0 → h_1 → ... → h_L, and the LM head projects h_L to logits. The observation that enables everything: h_k for k < L is already a reasonable hidden state. If you push it through the same LM head (or through a dedicated lightweight head), you get an output distribution that is “premature” but often correct for easy tokens. That is the source of the draft.

The self-speculative iteration has the same structure as classic speculative (draft, verify, accept/reject) but both roles are the same model:

Step 1 — Shallow draft. To produce γ draft tokens, the model walks only the first k layers (or a subset of layers in the layer-skip case) and applies the LM head. Each draft token costs ≈ k/L of a full forward pass. We call c = k/L the relative cost of the draft. The γ drafts are generated autoregressively at this reduced cost.

Step 2 — Full verify. The model runs a single forward pass through the L layers over prompt + x_1...x_γ. Because attention is causal it obtains p(·|prompt, x_<i) for each position, exactly as in classic speculative.

Step 3 — Accept/reject. Rejection sampling identical to the previous post: tokens are accepted from left to right, the first divergence is corrected by sampling from the residual norm(max(0, p−q)), and if all γ are accepted the bonus token is appended. Output quality is exactly that of the full model.

The trick that makes `c` cheaper still: reusing the KV cache of shared layers

Here is the key difference against an external draft. When the model drafts by walking layers 0..k, it computes and stores the KV cache of those k layers for the prompt tokens and the drafts. When the full verify comes along, layers 0..k of the L-layer forward pass are bit for bit the same operations on the same weights that the draft already performed. There is no need to recompute them: the verify reuses the KV cache the draft left for layers 0..k directly, and only genuinely computes the missing layers k..L.

That has two consequences:

Zero extra memory. There is no second KV cache. The KV of the common layers is a single one, shared between draft and verify. Contrast that with vanilla SD, where the draft has its own complete cache (see KV cache).
Partially reused compute. The verify only pays for the “new” layers k..L for tokens that already went through the draft. The full forward pass is not as expensive as L suggests, because the first k layers come from the cache.

The families (state of play 2026)

There is no single way of doing self-speculative. They differ in which layers are skipped and in whether training is needed.

Family	Year / venue	How it picks what to skip	Training?	Distinctive idea
LayerSkip (Elhoushi et al.)	2024, arXiv:2404.16710	Early-exit at a fixed layer `k`; a single LM head serves every exit	Yes — layer dropout + early-exit loss in train/fine-tune	A single model trained to do both draft and verify; reuses partial compute
SWIFT	ICLR 2025 (OpenReview EKJhH5D5wA)	Selects which layers to skip on the fly, without touching weights	No — plug-and-play on the given model	Training-free self-spec: optimises the set of skipped layers at runtime
CLaSp	2025, arXiv:2505.24196	Dynamic in-context layer skip: the pattern of skipped layers adapts to the context	No (dynamic at inference)	The skip is not fixed; it changes with what is being generated
ConfLayers	2026, arXiv:2604.14612	Skips layers according to the confidence of the intermediate state (adaptive per token)	No (confidence criterion)	Variable depth: easy tokens exit earlier, hard ones go deeper
Saguaro	2025–26	Asynchronous formulation: the draft keeps speculating in parallel while verification runs	Depends on the variant	Overlaps draft and verify in time instead of alternating them
SSD for on-device MoE	ACM Web Conf. 2026, doi 10.1145/3774904.3792218	Self-spec exploiting MoE sparsity (few active experts per token)	MoE-specific variant	The shallow draft activates even fewer experts; fits MoE on device

Three operational readings of the table:

The axis that matters most is training yes/no. LayerSkip gives the best α because the model learns to be a good shallow draft (with early-exit loss the intermediate layers are explicitly trained to predict well). But it demands a fine-tune. SWIFT, CLaSp and ConfLayers are training-free: worse α, but they apply to any already-trained model without touching anything. For an SLM you do not control, training-free is the realistic option.
Adaptive skip (CLaSp, ConfLayers) raises α because it fits the draft’s depth to the token: it spends little on the easy ones and more on the hard ones, instead of a fixed k. In exchange, the effective c stops being constant.
Saguaro attacks something else: it does not raise α, it overlaps draft and verify time. It is orthogonal to the rest and can be combined.

The maths: same framework, different `c`

We reuse the apparatus of the speculative post without changing a letter. With α the acceptance rate and γ the number of drafts:

$$E[\text{tokens per step}] = \frac{1 - \alpha^{\gamma+1}}{1 - \alpha}, \qquad \text{Speedup} = \frac{1 - \alpha^{\gamma+1}}{(1 - \alpha)(\gamma c + 1)}$$

And the algorithmic ceiling is the same: lim_{γ→∞} = 1/(1-α). The only thing that changes in self-speculative is the value of c: it is no longer the size ratio of two models, but c = k/L, the fraction of layers the early-exit draft walks.

Numerical example: self-spec with L=32, exit at k=8

Take an SLM with L = 32 layers that exits at k = 8 for the draft: c = k/L = 8/32 = 0.25. Suppose α = 0.7 (reasonable for early-exit on conversational tokens) and γ = 4.

Expected tokens per step: (1 − 0.7⁵) / (1 − 0.7) = (1 − 0.168) / 0.3 = 0.832 / 0.3 = 2.77
Speedup: 2.77 / (4 × 0.25 + 1) = 2.77 / 2.0 = 1.39×

The denominator factor is γc + 1 = 4·0.25 + 1 = 2.0: the early-exit draft, costing a quarter of the model per token, eats part of the benefit. Exiting higher up helps: with k = 4 (c = 0.125), the denominator is 1.5 and the speedup 2.77/1.5 = 1.85×. But exiting higher up normally lowers α, so there is real tension between a small k (cheap) and a high α (accurate).

An honest comparison with an external draft

Put a tiny, well-distilled external draft alongside: c = 0.1 and α = 0.78 (what an EAGLE-style draft can deliver), same γ = 4.

Tokens/step: (1 − 0.78⁵)/(1 − 0.78) = (1 − 0.289)/0.22 = 0.711/0.22 = 3.23
Speedup: 3.23 / (4 × 0.1 + 1) = 3.23 / 1.4 = 2.31×

Configuration	c	α	tokens/step	speedup	extra VRAM	pieces to maintain
Self-spec early-exit (k=8)	0.25	0.70	2.77	1.39×	0	0
Self-spec early-exit (k=4)	0.125	0.65	2.50	1.67×	0	0
Distilled external draft	0.10	0.78	3.23	2.31×	yes (+model +KV)	1 extra model

The reading is exactly what you would expect, and it is worth not dressing it up: if you have a dedicated draft, trained against your target, and it fits in memory, its higher α and lower c give it more speedup. EAGLE-3 with a well-trained draft usually wins on raw speedup. Self-spec does not compete on raw speedup; it competes on total cost. Its winning columns are the two on the right: zero extra VRAM and zero pieces to maintain. Self-spec wins when:

there is no trained draft for your model (your own SLM, an unusual fine-tune),
the draft does not fit (a 4090 already full, a long context that needs the KV),
you are on device (phone, NUC, edge), where a second model and its KV simply do not go in.

It is the same pattern as with MTP in the previous post: sometimes the best draft is the one you do not have to load.

Why it fits small models and device precisely

The regime where self-spec shines is low concurrency, memory-bandwidth-bound, with a tight memory budget, exactly that of an SLM on a single GPU or on device (the reason for the regime is in the inverted roofline). Three reasons:

Zero extra memory is decisive where there is none to spare. On an RTX 4090 (24 GB, Ada Lovelace) serving a quantised 7B–8B with a long context, every GB counts. Self-spec asks for none: it reuses weights and KV. An external draft, however small, forces you to trim context or concurrency. On device the difference is binary: with self-spec you accelerate; with an external draft there is no room, full stop.
There is no second checkpoint to version. Operationally, an SLM at the edge deployed across hundreds of boxes becomes unsustainable if each one needs two synchronised models. A single binary that does draft and verify is far simpler to maintain.
It fits MoE on device. In a fine-grained MoE for device (see native architectures for device), the shallow draft activates even fewer experts, and the memory-bound regime persists even at medium batch, precisely what the work on SSD for on-device MoE (ACM WWW 2026) exploits.

The counterpoint, repeated so it does not get forgotten: on a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8), where memory is not the bottleneck, a dedicated EAGLE-3 draft does fit and its higher α gives it more speedup. There, self-spec is plan B: you use it if the model is exotic and there is no trained draft, not because memory is tight.

Pitfalls

α depends enormously on k. Exiting too high up (small k) makes the draft cheap but sinks α; exiting too low down (k close to L) raises α but the draft costs almost a full forward pass and c → 1, killing the speedup. The optimum is empirical and model-specific. Distrust any speedup number that does not say at which k it was measured.
Training-free is not free in draft quality. SWIFT and CLaSp give lower α than LayerSkip precisely because the model’s intermediate layers were not trained to be good premature exits. The number that matters is α measured on your distribution, not the paper’s.
Sampling temperature and creative outputs lower α just as in classic speculative. At high T, the self-spec speedup erodes faster still because it starts from a lower α.
Large batch neutralises it just as it does classic speculative. As soon as decode turns compute-bound, the drafts stop being “almost free”. Self-spec is for low concurrency.

References

Elhoushi, M., et al. LayerSkip: Enabling Early Exit Inference and Self-Speculative Decoding. 2024. https://arxiv.org/abs/2404.16710
SWIFT: On-the-Fly Self-Speculative Decoding for LLM Inference Acceleration. ICLR 2025. https://openreview.net/forum?id=EKJhH5D5wA
CLaSp: In-Context Layer Skip for Self-Speculative Decoding. 2025. https://arxiv.org/abs/2505.24196
ConfLayers: Confidence-Adaptive Layer Skipping for Self-Speculative Decoding. 2026. https://arxiv.org/abs/2604.14612
Self-Speculative Decoding for MoE on Device. ACM Web Conference 2026. https://doi.org/10.1145/3774904.3792218
Hugging Face blog. Faster Text Generation with Self-Speculative Decoding. https://huggingface.co/blog/layerskip
Leviathan, Y., Kalman, M., Matias, Y. Fast Inference from Transformers via Speculative Decoding. ICML 2023. https://arxiv.org/abs/2211.17192

The roofline flips: why optimising small models is a different performance game

Tue, 09 Jun 2026 01:30:00 +0000

This post is the anchor of a mini-series on inference performance in small models (SLMs). Almost every optimisation post on the blog, KV cache, decode, quantisation, was written with a 70B in mind. Here I argue that when the model shrinks by an order of magnitude, the roofline changes regime and several of those intuitions flip. This is not a nuance: it is a different game.

TL;DR

Autoregressive decode in a large LLM is memory-bandwidth-bound: at every step you have to move all the model weights from HBM to the registers of the SMs, and that dominates over the arithmetic operations. The GPU spends its time waiting for bytes, not computing. That single sentence, that decode “waits on HBM”, is the root of half the optimisations on this blog. In a small model (SLM, say 0.5B–7B) the sentence stops being true in the simple form we used to tell it. At batch 1 you are still memory-bound with respect to the hardware, yes, but the forward pass is so cheap (moving 6 GB at 1 TB/s takes ~6 ms, not 70 ms) that the fixed per-step costs, kernel launches, Python scheduler overhead, the sampler, host↔device copies, the synchronize calls, stop being noise and start eating 20-30 % of the time. The bottleneck moves from HBM to orchestration. The concrete, quantitative consequences: (1) CUDA graphs and cutting scheduler overhead pay off more in SLMs than in large models; (2) weight quantisation gives less latency improvement at batch 1 in SLMs, because proportionally there are fewer weights to move relative to activations, KV cache and fixed overhead; (3) batching has more headroom because you cross the ridge point late; (4) the KV cache can dominate relative memory. All of this comes out of a single model, the roofline, applied with numerical honesty.

The analogy: the pantry and the waiter

A kitchen with two very different services.

Tasting menu service, one enormous slow dish (the 70B LLM). Every dish involves heavy ingredients that the assistant has to fetch from the pantry at the back, several times, carrying boxes. The chef, by contrast, assembles the dish in a moment: the slow part is fetching the ingredients, not cooking them. If you want the service to go faster, you do not hire a more skilful chef: you widen the corridor to the pantry, or make each trip bring more boxes. The pantry is HBM; the trip is memory bandwidth; cooking is compute. The big dish is bound by the pantry.

Tapas service, tiny plates (the SLM). Now each tapa has two ingredients and is assembled in a second. The trip to the pantry per tapa is very short. But a cost appears that was negligible for the big dish: the waiter. For every tapa, the waiter has to walk to the kitchen, pick up the plate, carry it to the bar, walk back, take the order, call it out. That back and forth is fixed: it costs the same for one tapa as for the enormous dish. When the tapa is assembled in a second, the waiter, not the pantry, is the bottleneck. Shortening the corridor to the pantry (widening HBM, quantising the weights) barely improves the service any more; what improves it is the waiter chaining several orders together without going back to the kitchen each time (CUDA graphs) or serving several tables in one pass (batching).

The roofline is the tool that says, with numbers, at what point the waiter dominates over the pantry. That boundary is the ridge point, and the joke in the title is that in SLMs we cross the regime much earlier than large-model intuition led us to believe.

The bare mechanism: what the roofline says

The roofline model (Williams, Waterman and Patterson, 2009) starts from a single quantity: arithmetic intensity, which is how many operations you do per byte moved from memory.

$$\text{AI} = \frac{\text{FLOPs}}{\text{bytes moved from memory}} \quad [\text{FLOP/byte}]$$

The hardware has two ceilings: compute (peak FLOPS) and memory (peak bandwidth × AI). Achievable performance is the minimum of the two:

$$\text{Perf} = \min\big(\text{peak FLOPS},\; \text{BW} \times \text{AI}\big)$$

Where the two lines cross is the ridge point, the AI above which you stop being limited by memory and start being limited by compute:

$$\text{AI}_{\text{ridge}} = \frac{\text{peak FLOPS}}{\text{peak BW}}$$

If your kernel has an AI below the ridge, you are memory-bound (the GPU waits for bytes). Above it, compute-bound (the GPU computes flat out and memory is in surplus). What matters is that the ridge point is a property of the hardware, not of the model. Let us look at the numbers, approximate, and I flag them as such because marketing figures mix dense and sparse, different dtypes and unrealistic thermal conditions.

Generic 4×H100 SXM cluster (320 GB, NVLink, native FP8). Per GPU, ~989 TFLOPS BF16 dense (~1,979 TFLOPS FP8 dense; the figure with sparsity is double that and almost never applies to LLM inference). HBM3 ~3.35 TB/s. The ridge in BF16:

$$\text{AI}_{\text{ridge}}^{\text{H100,BF16}} \approx \frac{989 \times 10^{12}}{3.35 \times 10^{12}} \approx 295 \ \text{FLOP/byte}$$

In FP8 the ridge rises to ~590 FLOP/byte (twice the FLOPS against the same BW). Careful: these are datasheet peak figures; in practice a real kernel rarely gets past 70-80 % of either ceiling.

RTX 4090 (24 GB, Ada Lovelace). ~330 TFLOPS FP16 with FP16 accumulation via tensor cores (the “660 TOPS” figure that circulates is with sparsity), and ~1 TB/s of GDDR6X. The ridge:

$$\text{AI}_{\text{ridge}}^{\text{4090,FP16}} \approx \frac{330 \times 10^{12}}{1.0 \times 10^{12}} \approx 330 \ \text{FLOP/byte}$$

Curiously, the same order as the H100 in BF16: the 4090 has less BW but also fewer FLOPS, and the ratio comes out similar. The ridge sits around 300 FLOP/byte in both cases. Hold on to that number.

And where does decode fall? In decode at batch 1, each weight is loaded once from HBM and used for a single multiply-accumulate (one token, one row of activation). The AI of the decode GEMM at batch 1 is on the order of AI ≈ 1-2 FLOP/byte (each weight byte takes part in ~2 FLOP). With batch B, the same weight loaded once serves B rows of activation, so the AI scales roughly linearly:

$$\text{AI}_{\text{decode}}(B) \approx 2B \ \text{FLOP/byte} \quad (\text{for the weight GEMM part})$$

You cross the ridge when 2B ≈ 300, that is B ≈ 150 as an order of magnitude (in practice earlier, because of attention and overheads, but that is the frame). The clean conclusion: decode at low batch is always deeply memory-bound, nowhere near the ridge. That is why we say “decode waits on HBM”, and why quantising weights (moving fewer bytes) speeds up decode in a large model almost linearly. So far, this is the standard story from the large-model posts.

The point of the title: why it flips in SLMs

The classic roofline has a blind spot that does not matter in large models and is everything in small ones: it only models the work inside the kernel. It assumes the only time is bytes/BW or FLOPs/FLOPS. But a real decode step is not just the GEMM. It is a sequence of dozens of kernels (QKV projections, attention, the two MLP layers, normalisations, residuals, the logits head, sampling) and, around each one, there is a fixed orchestration cost:

Kernel launches (kernel launch): each cudaLaunchKernel costs on the order of 5-10 µs of CPU/driver overhead, regardless of kernel size. A decode forward with ~30-60 kernels launched sequentially drags ~0.3-0.6 ms just in launching.
Python scheduler overhead: the vLLM scheduler loop prepares metadata, decides which requests enter the step, builds the input tensors. In pure Python this is hundreds of µs to a couple of ms per step, especially at low concurrency where it is not amortised.
Sampling and post-processing: applying temperature, top-p, penalties, the argmax/multinomial, copying the token back. Another block of hundreds of µs.
Synchronisations and host↔device copies: every synchronize or small copy adds latency that is neither FLOPs nor HBM bytes.

Let us call the sum of all of that T_fixed, the per-step cost independent of model size, on the order of 1-3 ms in an unoptimised Python stack. The real time of a step is now:

$$T_{\text{step}} \approx \underbrace{\frac{\text{weight bytes}}{\text{BW}}}_{T_{\text{HBM}} \text{ (memory-bound)}} + \; T_{\text{fixed}}$$

In a 70B BF16, moving ~140 GB at 3.35 TB/s takes ~42 ms of T_HBM. Against that, a T_fixed of 1-3 ms is noise (2-7 %). The classic roofline is right: the model is memory-bound, full stop. But in a 3B BF16, T_HBM drops to a few ms, and suddenly T_fixed is the same order as T_HBM. The bottleneck stops being the pantry and becomes the waiter. This is the flip in the title, and four counterintuitive consequences follow from it:

(a) At batch 1 you are still memory-bound with respect to the hardware. The AI has not changed: it is still ~2 FLOP/byte, below the ridge. Anyone reading only the roofline will conclude “memory-bound, quantise the weights”. That is true but incomplete: the roofline does not see T_fixed.

(b) Fixed costs become a huge fraction of the step. This is the central point. In the 70B, T_fixed / T_step ≈ 5 %. In the 3B it can be 20-30 %. The effective bottleneck of the 3B is half HBM, half orchestration.

(c) That is why CUDA graphs and cutting scheduler overhead pay off MORE in SLMs. A CUDA graph captures the whole kernel sequence of the step and relaunches it with a single cudaGraphLaunch, removing almost all of the per-kernel launch overhead and much of the per-iteration Python scheduler work. In the 70B, shaving 0.5 ms off a 42 ms step is a +1 % you barely notice. In the 3B, shaving those same 0.5 ms off a ~7 ms step is +7 %, and if you take away nearly all of T_fixed you can gain 20-30 %. The same optimisation, a different prize, because the denominator changed.

(d) Weight quantisation gives LESS latency improvement at batch 1 in SLMs. This is the most counterintuitive one. In the 70B, T_HBM is almost the whole step; going from BF16 to INT4 quadruples the effective weight bandwidth and almost quadruples decode speed. In the 3B, T_HBM is only part of the step (the rest is T_fixed + attention + KV). By Amdahl’s law, if the weights are 60 % of the step and you speed them up 4×, the total step improves only 1/(0.4 + 0.6/4) = 1.8×, not 4×. And proportionally there are fewer weights to move relative to activations, KV cache and fixed overhead. Aggressive quantisation in SLMs does help, yes, but not through raw latency at batch 1, where it gives diminishing returns, rather through capacity and concurrency (we will see that at the end).

(e) The KV cache can dominate relative memory. With 6 GB of weights (3B BF16), a single long-context session can approach that order of magnitude in KV cache. In a 70B (140 GB of weights) the KV is proportionally small until high concurrency. In SLMs the VRAM balance tips towards KV much earlier (the detail is in KV cache), and that changes which memory optimisation is the lever.

The maths that matter: a 3B on a 4090

Let us do the whole calculation, which is where the flip shows up without rhetoric.

Model: 3B parameters, BF16 → 2 bytes/param → ~6 GB of weights. Hardware: RTX 4090, BW ≈ 1 TB/s.

Memory-bound ceiling for decode (batch 1). Each token requires loading the 6 GB once:

$$T_{\text{HBM}} = \frac{6 \times 10^{9} \ \text{bytes}}{1 \times 10^{12} \ \text{bytes/s}} = 6 \times 10^{-3}\ \text{s} = 6\ \text{ms/token}$$ $$\text{Ceiling} = \frac{1}{6\ \text{ms}} \approx 166\ \text{tok/s}$$

That is the theoretical memory-bound ceiling: 166 tok/s, assuming that moving the weights is the only cost. The classic roofline would stop here and say “166 tok/s, go get more BW or quantise”.

Now the fixed overhead. Let us put T_fixed ≈ 2 ms/step (a reasonable value for a Python scheduler + ~40 kernels launched + sampling, without CUDA graphs). The real step:

$$T_{\text{step}} = T_{\text{HBM}} + T_{\text{fixed}} = 6 + 2 = 8\ \text{ms} \;\Rightarrow\; \frac{1}{8\ \text{ms}} = 125\ \text{tok/s}$$

The overhead has eaten 41 tok/s out of the theoretical 166: T_fixed is 25 % of the step (2 of 8 ms). Compare with the 70B: T_HBM ≈ 42 ms, T_step ≈ 44 ms, T_fixed is 4.5 %. Same absolute overhead, relative impact 5-6× larger in the SLM.

What happens if you apply CUDA graphs and take away, say, 1.5 of the 2 ms of T_fixed:

$$T_{\text{step}}^{\text{graphs}} = 6 + 0.5 = 6.5\ \text{ms} \;\Rightarrow\; 154\ \text{tok/s}$$

From 125 to 154 tok/s: +23 % from orchestration alone, without touching the model or the memory hardware. In the 70B the same intervention would have gone from 44 to 42.5 ms, +3.5 %. There it is, in two numbers: “a different game”.

What happens if you quantise the weights to INT4 (1.5 GB instead of 6 GB), with T_fixed still at 2 ms:

$$T_{\text{HBM}}^{\text{INT4}} = \frac{1.5 \times 10^{9}}{1 \times 10^{12}} = 1.5\ \text{ms};\quad T_{\text{step}} = 1.5 + 2 = 3.5\ \text{ms} \;\Rightarrow\; 285\ \text{tok/s}$$

The 4× weight quantisation did not give 4× in latency: it went from 125 to 285 tok/s, a 2.3×, because the 2 ms of T_fixed now dominates (it is 57 % of the step). In the 70B, quantising to INT4 gives almost the full 4× because T_fixed is still noise. The same quantisation delivers twice the speedup on the large model as on the small one, at batch 1. And if you also apply CUDA graphs on top of INT4 (T_fixed → 0.5 ms): 1.5 + 0.5 = 2 ms → 500 tok/s. The order of the optimisations matters: in SLMs, attacking T_fixed first unlocks the rest.

Configuration (3B, 4090, batch 1)	T_HBM	T_fixed	T_step	tok/s	vs. baseline
BF16, no graphs (baseline)	6.0 ms	2.0 ms	8.0 ms	125	1.00×
BF16 + CUDA graphs	6.0 ms	0.5 ms	6.5 ms	154	1.23×
INT4, no graphs	1.5 ms	2.0 ms	3.5 ms	285	2.28×
INT4 + CUDA graphs	1.5 ms	0.5 ms	2.0 ms	500	4.00×

(Illustrative figures with T_fixed rounded; the point is the pattern, not the decimal. The real T_fixed depends on the stack, the PyTorch/CUDA version and whether there is tensor parallelism. Measure it in your own setup before believing any row.)

Implications per optimisation

With the model in hand, the blog’s levers reorder themselves once the regime changes.

Batching: far more headroom in SLMs. Remember that you cross the ridge at B ≈ ridge/2 ≈ 150 as an order of magnitude. In a large model, VRAM runs out long before you saturate compute (weights + KV do not let you reach batch 150). In an SLM the weights take up little space, so you can fit large batches in VRAM and stay memory-bound over a much wider range: the T_HBM of the weights is amortised across the B requests (you load them once, they serve B), so aggregate throughput per GPU rises almost linearly with B until very high values. This is exactly the opposite of the 70B’s fear of saturating compute. In SLMs, batching is the throughput lever par excellence because you saturate compute late; the batch grid search in vLLM has a much wider plateau of good behaviour. Careful: batching improves throughput, not per-request latency; for single-stream latency the prize is in T_fixed.

Speculative decoding: a different crossover point. Speculative wins when verifying γ tokens is “almost free” because you are memory-bound. In an SLM the target is already cheap, so the draft has to be tiny for c = T_draft/T_target to stay small, and the draft’s own T_fixed (launching its kernels) bites harder. The crossover to compute-bound with batching also arrives earlier in absolute terms of tok/s served. The variant that fits best here avoids a separate draft: self-speculative / early-exit reuses early layers of the model itself and saves the T_fixed of orchestrating two models.

Quantisation: it helps for capacity, not for latency at batch 1. As the table showed, INT4 in an SLM at batch 1 gives diminishing returns in latency. Its real prize in SLMs is capacity: weights 4× smaller free VRAM for more KV cache → more concurrency, and it is at high concurrency (aggregate throughput) that saving bytes pays off again. Aggressive sub-4-bit and ternary quantisation takes this to the extreme: in SLMs it makes sense above all for fitting more sessions per GPU, not for lowering the latency of a single one. And it is worth remembering (see quantisation) that at batch 1 the dequantize adds compute work which, in a regime already grazed by T_fixed, is not always free.

Architecture: fine-grained MoE changes which bytes you move. A fine-grained device-native MoE activates few parameters per token, so T_HBM drops relative to a dense model of the same total size, but the T_fixed fraction rises even further, and the router adds its own fixed overhead. It is the SLM regime taken to its limit: almost the whole game is played in orchestration.

Scheduler and CUDA graphs first. The operational conclusion, inverted with respect to the large-model posts: in SLMs, before touching the model, kill T_fixed. CUDA graphs (see SMs, streams and graphs), a vLLM scheduler with its Python part minimised or compiled, and kernel persistence are the first-order levers. In a 70B they would be marginal polish; in a 3B they are half the available speedup.

Applied to on-premise hardware

On an RTX 4090 (24 GB, Ada Lovelace). This is the scenario where the flip is most visible, because the 4090 has ~1 TB/s (a third of the H100) but T_fixed is the same in absolute terms. A 3B BF16 without CUDA graphs leaves ~125 tok/s on the table when the memory-bound ceiling is 166; enabling graphs and cleaning up the scheduler recovers most of that. The 4090 has plenty of VRAM room for an SLM, so the bottleneck is almost never total memory but orchestration and, at high concurrency, the KV cache. Rule of thumb: on a 4090 with an SLM, profile the per-step overhead first (Nsight Systems on the gap between kernels) before you quantise.

On a generic 4×H100 SXM cluster (320 GB, NVLink, native FP8). The H100 has 3.35 TB/s, so the T_HBM of an SLM is even smaller (a 3B FP8 is ~3 GB → ~0.9 ms) and T_fixed dominates even earlier: a badly orchestrated SLM on an H100 can spend more time in the Python scheduler than moving weights. Serving a single single-stream SLM on an H100 is close to a waste; the right mode is aggressive batching (you saturate compute late, so you push large batches through and per-GPU throughput takes off) or multiplexing many SLMs/sessions per GPU via MPS/MIG. This connects with capacity planning: for SLMs the capacity calculation is governed by concurrency and KV cache, not by the weights. And with the dilemma of one big versus N small: replicating SLMs makes sense precisely because each replica saturates compute late and TP brings nothing (the model already fits; TP would only add communication T_fixed).

What we have not covered

The exact measured T_fixed, kernel by kernel, with Nsight Systems: how much is launch, how much scheduler, how much sampling. That is the content of the next post in the series.
torch.compile / partial captures: alternatives and complements to CUDA graphs when there is dynamic control flow.
The prefill regime in SLMs: prefill is compute-bound even in small models (it processes many tokens at once, high AI), so its roofline is the opposite of decode’s; see prefill.
Attention and KV as the second term of T_HBM: here we have folded them in implicitly; the fine-grained breakdown of attention (which scales with sequence length, not with the weights) deserves its own treatment.

References

Williams, S., Waterman, A., Patterson, D. Roofline: An Insightful Visual Performance Model for Multicore Architectures. Communications of the ACM, 52(4), 2009. https://doi.org/10.1145/1498765.1498785
Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inference. arXiv:2503.08311, 2025. https://arxiv.org/abs/2503.08311
Databricks. LLM Inference Performance Engineering: Best Practices. https://www.databricks.com/blog/llm-inference-performance-engineering-best-practices
NVIDIA. NVIDIA H100 Tensor Core GPU Datasheet. https://resources.nvidia.com/en-us-tensor-core/nvidia-tensor-core-gpu-datasheet
NVIDIA. GeForce RTX 4090 — product specifications (Ada Lovelace tensor core figures; treat as approximate, they mix dense/sparse).
Yuan, Z. et al. LLM Inference Unveiled: Survey and Roofline Model Insights. arXiv:2402.16363, 2024 — application of the roofline specifically to LLM inference.

The corridors and the security guard: PCIe topology, GPUDirect P2P and ACS

Mon, 08 Jun 2026 06:00:00 +0200

This continues the below the engine series. The NVLink and NCCL post explained the shared bench over which the GPUs pass data to each other at 450 GB/s. But that bench only connects GPUs to each other. Everything else, disk, network, the host, travels over another bus, PCIe, and through its corridors. The cold start already brushed against this with GPUDirect Storage; this post opens up the complete floor plan of the corridors and the guard who watches them.

TL;DR

On a 4×H100 SXM node, the GPUs talk over NVLink (450 GB/s per direction, around 7× PCIe), and for the all-reduce of tensor parallel that is the path. But PCIe does not go away: it is where everything else comes in. The weights come up from the NVMe over PCIe (the cold start), the data from another node arrives via the NIC over PCIe (RDMA), and a KV that moves between nodes travels over PCIe. GPUDirect is the family that lets those bytes go straight from the device into the HBM without bouncing through host RAM: P2P (GPU↔GPU), RDMA (GPU↔NIC) and Storage (GPU↔NVMe). The obstacle is a guard called ACS (Access Control Services): a PCIe security feature that by default forces peer-to-peer traffic up to the root complex for inspection, which destroys the direct path and inserts a detour through the CPU. The IOMMU (VT-d) does something similar if it is not in passthrough mode. Disabling them gives performance; keeping them gives isolation and virtualisation, and that is a real decision in an ENS environment. This post explains the topology (nvidia-smi topo -m), GPUDirect, why ACS and IOMMU break P2P with numbers, the 10 knobs and the trap of removing the guard without knowing what it was watching. On the generic 4×H100 SXM cluster.

Where you are: the corridors, not the bench

Picture the kitchen as a building. The elite cooking stations, the GPUs, are in a room with a huge central bench (NVLink/NVSwitch) over which they pass ingredients at full speed without standing up. That bench is for them and them alone.

But the building has more to it: the pantry (the NVMe storage), the loading door (the network, the NIC) and reception (the CPU and its RAM). To reach any of those, the stations do not use the central bench: they use the corridors of the building, the PCIe bus. And here the character of the post appears: at the entrance of each corridor there is a security guard (ACS) who, by default, does not let two stations hand each other something directly along the corridor: it forces them to take the parcel up to reception to be checked, and only then does it come down to its destination. It is safe, but it is an absurd detour when the two stations are right next to each other. GPUDirect is the permission to skip that detour; ACS and IOMMU are the reasons why, often, you cannot.

The topology of a node: two buses, not one

The most common mistake is to think that a node has “a bus”. There are (at least) two, and they do different things:

NVLink / NVSwitch — the high-speed GPU↔GPU mesh. On H100 SXM, 18 links × 50 GB/s = 900 GB/s bidirectional between any two GPUs, with NVSwitch giving an all-to-all with no contention (NVLink, NVIDIA). It is the shared bench.
PCIe Gen5 — the general I/O bus. An x16 link gives 128 GB/s bidirectional (around 64 per direction) (H100 product brief). It connects each GPU with the CPU, the RAM, the NICs and the NVMes. It is the corridor.

The difference is 7×: NVLink moves in one second what PCIe takes seven to move. That is why intra-node tensor parallel goes over NVLink and nobody argues about it. PCIe matters for the other things: pulling weights off disk, receiving from the network, moving KV between nodes.

The tool for seeing this is nvidia-smi topo -m, which prints a matrix of how each pair is connected (NVIDIA forum):

 GPU0 GPU1 GPU2 GPU3 NIC0 CPU Affinity NUMA
GPU0 X NV18 NV18 NV18 PXB 0-47 0
GPU1 NV18 X NV18 NV18 PXB 0-47 0
GPU2 NV18 NV18 X NV18 SYS 48-95 1
GPU3 NV18 NV18 NV18 X SYS 48-95 1

The legend is what matters: NV18 = 18 NVLink links (the bench); PXB = crosses PCIe switches but not the host; PHB = goes through the host bridge; NODE = same NUMA, crossing PCIe; SYS = crosses the inter-socket interconnect (the worst case, it traverses NUMA). The fact that GPU0↔NIC0 is PXB and GPU2↔NIC0 is SYS tells you exactly which GPU should handle that NIC’s traffic, GPU 0, without crossing NUMA. This links directly with the NUMA post and the network NUMA one: PCIe affinity is NUMA affinity.

GPUDirect: skipping reception

Without GPUDirect, moving a piece of data from the NIC (or the NVMe) to the GPU takes a mandatory detour: device → host RAM → GPU. That bounce through RAM consumes CPU bandwidth, spends copies and adds latency. GPUDirect removes the bounce by letting the data go straight from the device into the HBM. Three flavours:

GPUDirect P2P — GPU↔GPU over PCIe (when there is no NVLink between them, or for traffic that does not use the bench).
GPUDirect RDMA — GPU↔NIC: the network card writes directly into the HBM. It is what makes efficient multi-node viable (NCCL over InfiniBand/RoCE).
GPUDirect Storage (GDS) — GPU↔NVMe: the disk writes straight into the HBM, with no host buffer. It is the lever of the cold start.

On an SXM node, the GPU↔GPU traffic of tensor parallel does not use P2P over PCIe: it uses NVLink. That is why GPUDirect matters above all at the edges of the node: the network (RDMA, for multi-node) and the disk (GDS, for startup). That is where ACS does damage.

The guard: why ACS and IOMMU break P2P

ACS (Access Control Services) is a PCIe security feature designed for virtualisation and isolation: it guarantees that one device cannot read from or write to another directly without the root complex mediating. To achieve that, it forces peer-to-peer transactions up to the root complex and back down (GDS best practices, NVIDIA). It is exactly the opposite of what GPUDirect wants: the direct path stops being direct.

The IOMMU (VT-d on Intel, the equivalent on AMD) translates addresses and isolates devices. If it is active and not in passthrough mode, it also redirects P2P traffic through the root complex, with the same effect: performance through the floor or, in extreme cases, hangs (NCCL troubleshooting).

Summarised without detours (Giraud, debugging P2P): ACS forces the trip through the root for security checks; IOMMU forces it for isolation and virtualisation. Both break the goal of P2P (direct communication with no intermediaries) and add overhead. If you do not need that security/virtualisation on that path, disabling them recovers the performance. The operational recipe for maximum GPUDirect performance: ACS off on the switches along the path and IOMMU in passthrough (iommu=pt) or disabled.

The maths that matter: how much the detour costs

Take a KV SWAP of 5 GB (preemption from the scheduler that sends KV to the host, or a transfer between nodes in disaggregated serving):

Path	Effective BW	Time for 5 GB
NVLink (GPU↔GPU intra-node)	~450 GB/s	~11 ms
Direct PCIe Gen5 x16 (P2P, ACS off)	~55 GB/s	~91 ms
PCIe via root complex (ACS on)	~25-30 GB/s*	~170-200 ms

* The detour through the root does not just “add latency”: it saturates the host bridge bandwidth, contends with other traffic and, depending on the topology, can fall well below the direct path. The figure is an order of magnitude, to show the size of the problem, not a benchmark.

The reading: on the path that does use PCIe (network, disk, swap), having ACS on can double or triple the time. And if that time is on the critical path, a cold start, a preemption swap, an inter-node all-reduce, it shows up in the latency the user sees. What disabling ACS does not fix: traffic that already went over NVLink (intra-node TP). There, ACS is irrelevant.

The real tension: performance vs isolation (and ENS)

Here the post gets serious, because the “disable ACS and IOMMU” recipe has a cost that in a regulated environment is not free. ACS and IOMMU exist for a reason: to isolate devices. On a dedicated bare-metal node for inference, with no virtualisation or multi-tenancy, you are not isolating anything that matters and disabling them is reasonable. But:

If you do GPU passthrough to VMs or use containers with strong isolation, the IOMMU is necessary, not optional.
In a multi-tenant scenario where several workloads share a node, ACS provides a guarantee that one device does not snoop on another.
Under ENS (see technical controls), workload isolation and access traceability can be requirements; disabling bus isolation to gain 80 ms is a decision you have to justify and document, not a silent tweak.

The design way out, when you need both: keep the isolation where compliance requires it and design so that the hot path does not depend on P2P over PCIe. Concretely, on an SXM node, the bulk of the critical traffic (TP) already goes over NVLink and ACS does not affect it. For the network, dedicate one NIC per GPU on its own PCIe switch (PXB) and use GPUDirect RDMA only on the path you control. For the disk, cache weights on local NVMe. That way you do not pay the choice between performance and isolation: you avoid it on the path that matters.

The 10 knobs

#	Knob	What it controls	Cost / risk
1	ACS off (switches on the path)	P2P detour through root	you lose bus isolation
2	`iommu=pt` / off	P2P redirection through root	breaks VM passthrough if off
3	`nvidia-smi topo -m`	audit the real topology	— (always worth it)
4	`p2pBandwidthLatencyTest`	measure P2P for real	— (verify before assuming)
5	NIC placement	same PCIe switch as the GPU	SYS if it crosses NUMA
6	NVMe placement	NUMA-local to the GPU	H2D crossing UPI
7	`nvidia-peermem` (GDR)	enables RDMA into HBM	correct driver/kernel
8	PCIe gen/lanes (x16)	width of the corridor	GPU silently at x8
9	relaxed ordering / ASPM	PCIe latency and power	jitter if misconfigured
10	persistence mode	avoids re-init of the path	idle GPU still paid for

How it connects with the rest of the stack

With NVLink and NCCL. The NVLink post covers the GPU↔GPU bench; this one covers the other bus, the one connecting to disk, network and host. They are complementary: ACS affects PCIe, not NVLink.

With the cold start. The GPUDirect Storage of the disk→HBM post is GPUDirect over the storage path; ACS on strangles it just as it strangles P2P.

With NUMA. The PCIe affinity of topo -m is the NUMA affinity of the host post; placing NIC and NVMe on the correct NUMA avoids the SYS path.

With the network. NIC placement and GPUDirect RDMA is the subject of the network NUMA post; the same principle of “get the CPU out of the way”.

With PagedAttention and the scheduler. The preemption SWAP (scheduler) moves KV blocks over PCIe; that is why V1 prefers RECOMPUTE and why this bus matters.

With disaggregated serving. Transferring KV between pools in disaggregated serving travels PCIe→NIC→PCIe; ACS and placement decide whether it is viable.

With ENS. Bus isolation is a technical control; see ENS/42001/AI Act controls.

Traps and things that are not what they look like

“Disable ACS everywhere, it goes faster.” On a dedicated node, fine. On one with virtualisation, multi-tenancy or isolation requirements (ENS), you are removing a security control. The correct decision is per path and documented, not global and silent.

“NVLink and PCIe are the same bus, more or less.” No. They are two buses with a 7× difference and different purposes. TP goes over NVLink; disk, network and host go over PCIe. Confusing them leads to “optimising” ACS for traffic that does not even go through PCIe.

“P2P just works, there is nothing to check.” P2P is silently disabled with ACS/IOMMU active, and many distros enable them by default. Check with p2pBandwidthLatencyTest and nvidia-smi topo -m; do not assume you have the direct path just because the GPUs are on the same node.

“IOMMU off always, for performance.” If you do GPU passthrough to virtual machines, the IOMMU is mandatory; disabling it breaks the passthrough. The correct mode is usually passthrough (iommu=pt): it keeps the necessary mapping without penalising P2P.

“More PCIe lanes = faster GPU.” PCIe is the I/O path, not the compute path. A GPU at x8 instead of x16 takes longer to load and to communicate over PCIe, but generates tokens at the same speed once the weights are inside. The damage from x8 is in the cold start and in multi-node, not in decode throughput.

“GPUDirect fixes any I/O bottleneck.” GPUDirect removes the bounce through the CPU; if your bottleneck is the device itself (saturated NVMe, NIC at full tilt) or the topology (a SYS path crossing NUMA), GPUDirect does not touch it. Measure where the bottleneck is first.

Conclusion

This whole series has been going down floors looking for where the time goes, and this one reaches the wiring of the building. Intuition treats the node as a homogeneous box where “the GPUs talk to everything”; the reality is that there are two buses with opposite purposes, an elite bench for the GPUs (NVLink) and some service corridors for everything else (PCIe), and a security guard in the corridors who, with the best of intentions, makes every parcel go up to reception before it is delivered. GPUDirect is the permission for direct delivery; ACS and IOMMU are the legitimate reasons why sometimes you are not given it. The lesson is not “disable the guard”: it is understanding which path is critical (almost never the one you think) and what the guard was watching before sending it home. On a dedicated node, the direct path is almost free and worth taking. On one that shares workloads or lives under ENS, bus isolation is a control you sacrifice with justification or do not sacrifice at all. Good design does not choose between performance and isolation blindly: it puts the critical traffic on the bench that needs no guard, and leaves the corridors for what can wait.

References

NVIDIA, GPUDirect Storage Best Practices Guide (ACS, IOMMU, paths): https://docs.nvidia.com/gpudirect-storage/best-practices-guide/index.html.
NVIDIA, NCCL Troubleshooting (IOMMU/VT-d and P2P): https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/troubleshooting.html.
M. Giraud, Multi-GPU (NVIDIA) P2P capabilities and debugging tips: https://morgangiraud.medium.com/multi-gpu-nvidia-p2p-capabilities-and-debugging-tips-fb7597b4e2b5.
NVIDIA, H100 Product Brief (PCIe Gen5, NVLink 900 GB/s): https://www.nvidia.com/content/dam/en-zz/Solutions/Data-Center/h100/PB-11773-001_v01.pdf.
NVIDIA, GPUDirect RDMA documentation: https://docs.nvidia.com/cuda/gpudirect-rdma/index.html.

The signature-dish specialist: vLLM's attention backend (FlashAttention, FlashInfer and the prefill/decode asymmetry)

Mon, 08 Jun 2026 05:40:00 +0200

Part of the under the engine series. The PagedAttention post explained where the KV lives (in paged blocks). This one explains who reads it and how: the attention kernel. And it connects with FlashAttention v1-v4, which took that kernel apart from the inside; here we look at the level above, how vLLM chooses between several kernels and why it needs more than one.

TL;DR

An LLM forward is, for the most part, standard matrix multiplications that any library does well. The exception that decides performance is attention, and having one good kernel is not enough: you need two, because the two phases of inference are physically opposite problems. Prefill processes the whole prompt: many queries against many keys, dense and compute-bound, the home ground of FlashAttention’s IO-aware tiling. Decode generates one token: a single query against all the accumulated KV, thin and memory-bound, where the only thing that matters is saturating HBM bandwidth while reading the paged KV. That is why vLLM does not have “the attention kernel” but a switchable backend (FLASH_ATTN, FLASHINFER, TRITON_ATTN and others) plus logic that picks according to the GPU: by default FA4 on Blackwell (SM100), FA3 on Hopper (SM90), FA2 elsewhere, with FlashInfer as the alternative that compiles bespoke kernels (JIT) and knows how to do cascade attention for shared prefixes. This post explains why prefill and decode are opposites (with arithmetic intensity), how the backend reads paged KV, how the engine chooses, what FlashInfer brings, the 10 knobs and the trap of pinning a backend blindly. On the generic 4×H100 SXM cluster.

Where you are: the specialist, not the commis

In the kitchen, almost all the work is chopping and sautéing: standard operations any competent commis chef executes, the matrix multiplications of the feed-forward layers and the projections. There is one dish that is not delegated: the signature dish, the one that defines the restaurant. That dish is attention, and it has a quirk: it is cooked in two radically different ways depending on the moment of service.

During prefill, when a new order arrives with its whole prompt, you cook big: a lot of raw material at once, a lot of fire, an intense operation that fills the stoves. During decode, when a table asks for “one more dish”, you cook à la carte: a single dish, but you have to go to the larder and bring back every ingredient that table has accumulated over the whole meal. One is a problem of firepower; the other, of larder speed. The same specialist does not do both well. That is why vLLM has several, and a chef who decides which one steps up depending on the GPU and the phase. That is the attention backend.

Why prefill and decode are opposite problems

This is the central idea, and one sum proves it: arithmetic intensity (FLOPs per byte read). An operation with high intensity is limited by compute; one with low intensity, by memory.

Prefill. We serve $N$ queries (the whole prompt) against $N$ keys. The $QK^\top$ operation and the $\text{softmax}\cdot V$ do on the order of $N^2 d$ FLOPs and read on the order of $N d$ of data. Intensity grows with $N$:

$$I_\text{prefill} \sim \frac{N^2 d}{N d} = N$$

With large $N$ (a prompt of thousands of tokens), intensity is high: compute-bound. This is where FlashAttention’s tiling squeezes the tensor cores and where you get close to the GPU’s peak TFLOPS.

Decode. We serve one query (the new token) against $L$ keys (all the accumulated KV). FLOPs on the order of $L d$; bytes read on the order of $L d s$ (the entire KV has to be read from HBM). Intensity is:

$$I_\text{decode} \sim \frac{L d}{L d s} = \frac{1}{s} \quad (\approx 0.5 \text{ FLOP/byte in FP16})$$

Constant and tiny: memory-bound. The decode kernel is not limited by how much the GPU can compute but by how fast it reads the KV from HBM. It makes no difference that the H100 has 132 idle SMs (see the SM post): the bottleneck is the 3.35 TB/s of bandwidth, and the decode kernel exists so as not to waste a single one of those bytes/s.

The design consequence: a kernel optimised for prefill (dense tiling, maximum tensor-core occupancy) is not the optimum for decode (coalesced reads of the paged KV, minimum latency). Serious servers have different kernels, or one kernel with two paths. In models with MLA (multi-head latent attention), vLLM goes as far as using separate backends for prefill and decode, selectable independently (attention backends, vLLM).

The scheduler’s trick: prefill and decode in the same forward

Here the circle closes with the scheduler post. Because vLLM V1 mixes requests in prefill and in decode in every step, one single forward has to serve both. The backend receives metadata telling it, for each sequence in the batch, how many queries it brings and how much KV it has to read, and applies the right path to each one. That is why the attention backend and the scheduler are coupled: the first has to digest the heterogeneous batch the second builds.

How the backend reads the paged KV

The kernel does not receive a contiguous KV tensor: it receives the block table from the block manager and does a gather over the physical blocks. This imposes a real constraint on the backend: it has to support the paged layout and vLLM’s block_size. Not every kernel in the world does; the ones vLLM integrates (FlashAttention, FlashInfer, Triton) are adapted to read KV in fixed-size blocks scattered across HBM. That is why you cannot plug in any attention kernel from a paper: it has to speak the language of the pigeonhole larder.

The backends and how the engine chooses

vLLM exposes a backend abstraction with several implementations (vLLM deepwiki):

FLASH_ATTN — the FlashAttention family. By default the version is picked by architecture: FA4 on SM100 (Blackwell), FA3 on SM90 (Hopper), FA2 elsewhere, configurable with flash_attn_version.
FLASHINFER — an attention engine with JIT compilation and specialisable kernels; strong on heterogeneous KV and shared prefixes.
TRITON_ATTN — written in Triton, portable and with no dependency on precompiled CUDA binaries (Triton backend deep dive, vLLM, mar-2026).
Specific backends for MLA and for non-NVIDIA hardware.

Selection is automatic unless you force it with VLLM_ATTENTION_BACKEND. The heuristic tries FlashAttention first; on Blackwell (SM100) the fallback order for MLA is TRT-LLM Ragged → FlashInfer → others; on other GPUs only FlashAttention is considered for the main path (attention backends, vLLM). The decision depends on: architecture (SM), dtype (FP16/BF16/FP8), head dimension, and whether the load needs a feature only one backend has (cascade attention, certain soft caps, FP8 in KV).

What FlashInfer brings: JIT and cascade attention

FlashInfer does not compete with FlashAttention on “being a bit faster”; it attacks a different problem: the heterogeneity of the KV in real serving (FlashInfer, arXiv 2501.01005). Two ideas:

JIT compilation. Instead of a monolithic kernel, FlashInfer generates bespoke kernels for the attention variant, the problem shape and the KV layout you have, injecting functors (query/key/logit transformations, masks). It specialises rather than generalises.

Cascade attention. Here is the jewel for serving with shared prefixes. If $R$ requests share a prefix of $P$ tokens (a common system prompt), naive attention would read that prefix $R$ times. Cascade attention computes it once against the shared prefix and then combines with each request’s own suffix:

$$\text{reads: } \underbrace{R \cdot (P + s_i)}_{\text{naive}} \;\longrightarrow\; \underbrace{P + \textstyle\sum_i s_i}_{\text{cascade}}$$

With $R=50$ requests and a prefix $P=1000$, that is reading 50,000 prefix tokens versus 1,000. It is the natural complement to prefix caching: the block manager shares the prefix’s memory, and cascade attention shares the compute of attending over it.

The arithmetic that matters: when switching backend gives you something

The backend only moves the needle where attention is the bottleneck. In memory-bound decode, a kernel that exploits HBM bandwidth better gives a real improvement; in compute-bound prefill with long sequences, FA3/FA4 getting close to the tensor-core peak gives a real improvement. But if your bottleneck is in another layer, launch overhead, a badly sized scheduler, the cold start, then switching backend does not touch that part. The rule, once again: measure the regime before optimising.

The 10 knobs

#	Knob	What it controls	Cost / risk
1	`VLLM_ATTENTION_BACKEND`	force a backend	mismatch with hardware/feature
2	`flash_attn_version` (2/3/4)	FA version	version unsupported on your SM
3	enable FlashInfer	JIT + cascade	initial JIT compilation time
4	cascade attention	prefix compute reuse	only helps with a heavily shared prefix
5	`kv_cache_dtype` (FP8)	FP8 support in the kernel	not every backend/SM supports it
6	`block_size`	layout the kernel must read	consistency with PagedAttention
7	MLA prefill backend	kernel for the dense phase	MLA models only
8	MLA decode backend	kernel for the thin phase	MLA models only
9	soft cap / sliding window	features that restrict backends	fewer kernel options
10	head_dim / variant	which kernels are eligible	exotic models with no support

How it connects to the rest of the stack

With FlashAttention. The FA post explains the kernel from the inside (tiling, online softmax, FA1-4); this is the level above, how vLLM chooses between kernels and why it needs more than one.

With PagedAttention. The backend reads the KV that the block manager places in blocks; it has to speak block-table.

With the scheduler. The scheduler builds mixed prefill+decode batches; the backend has to serve both regimes in a single forward.

With CUDA graphs. Attention kernels are captured in the CUDA graphs; a backend that launches many small kernels benefits more from capture.

With prefix caching. Cascade attention is the compute side of what prefix caching does in memory.

With FP8. Attending over KV in FP8 requires the backend to have the FP8 path; not all of them do on every architecture.

Traps and things that are not what they look like

“FlashInfer is always faster than FlashAttention.” No. FlashInfer wins when its specialisation (cascade, heterogeneous KV, a particular attention variant) applies to your load; in classic dense prefill, FA3/FA4 usually does as well or better. It depends on the regime, there is no universal winner.

“A good attention kernel works for everything.” The underlying error of this post. Prefill and decode are compute-bound and memory-bound respectively; a kernel tuned for one wastes effort on the other. That is why there are separate paths (and separate backends in MLA).

“Decode is compute-bound because the GPU is at 100%.” nvidia-smi at 100% is misleading (see the SM post): decode is memory-bound, the GPU is moving KV, not computing. Optimising decode’s compute is polishing what is not the bottleneck.

“I pin VLLM_ATTENTION_BACKEND and forget about it.” Pinning a backend by hand can leave you on a suboptimal one when you change GPU or version, or force a slow fallback if your hardware does not support what you asked for. Autoselection usually gets it right; pin it only with a measurement that justifies it.

“Cascade attention always helps.” Only with a heavily shared prefix across many concurrent requests. If each request has its own context, there is nothing to share and the overhead of organising the cascade is not amortised.

“The attention backend is the bottleneck, that is why I am slow.” Almost always the bottleneck is higher up (launch, scheduling, memory) or lower down (bandwidth). The backend matters where attention dominates; measure it with nsys/DCGM before changing it.

Conclusion

Of everything an LLM does while generating text, almost all of it is matrix multiplications that any library resolves. Performance is decided in a single kernel, attention, and the surprise is that it is not even one kernel: it is two opposite problems wearing the same name. Prefill wants fire, dense compute over thousands of tokens, and decode wants a fast larder, reading a token’s entire KV with the minimum waste of bandwidth. That is why vLLM did not pick a winning kernel but an abstraction that switches: FlashAttention tuned to each architecture for the general case, FlashInfer compiling bespoke kernels when there is heterogeneity or prefixes to share, Triton for portability. The head chef does not cook the signature dish just one way: he looks at who is ordering and at what point of service, and sends out the right specialist. The lesson for whoever is tuning is the usual one in this series: before changing specialist, make sure the signature dish really is what is holding you back.

References

vLLM, Attention Backends (selección, FA2/3/4 por arquitectura, MLA): https://docs.vllm.ai/en/latest/design/attention_backends/.
vLLM / DeepWiki, FlashAttention and FlashInfer: https://deepwiki.com/vllm-project/vllm/8.2-flashattention-and-flashinfer.
vLLM, Triton Attention Backend Deep Dive (mar-2026): https://vllm.ai/blog/2026-03-04-vllm-triton-backend-deep-dive.
Z. Ye et al., FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Serving (arXiv 2501.01005): https://arxiv.org/abs/2501.01005.
T. Dao, FlashAttention-2 / FlashAttention-3 (kernel IO-aware, async Hopper): https://github.com/Dao-AILab/flash-attention.

The pantry of pigeonholes: PagedAttention and the vLLM block manager

Mon, 08 Jun 2026 05:20:00 +0200

Part of the under the engine series. The scheduler post ended with a loose end: the scheduler has a second budget, the KV blocks, and when they run out it preempts. This post opens up that budget. It is the piece the KV cache post took for granted, what gets stored, in order to explain how it is managed in memory. And it is the one the FlashAttention post had been promising for months.

TL;DR

The KV cache grows a little with every token generated, and the problem was never its total size but the way it is reserved. The first servers asked, per request, for a contiguous chunk of HBM the size of the maximum possible context. Since almost no request reaches that maximum, the result was catastrophic: 60-80% of the HBM wasted on fragmentation. PagedAttention applies to the KV the oldest and most battle-tested idea in operating systems, paging: split the KV into fixed-size blocks (16 tokens by default), store them in non-contiguous HBM wherever there is room, and keep a block table that translates each sequence’s logical block into its physical one. Waste falls to ~4% (only the last block, half-filled). And since each block can be identified by the hash of its content, two requests that share a prefix point at the same physical block and share memory, with copy-on-write when one of them diverges: that is the engine of prefix caching. This post explains fragmentation with numbers, the block manager, the block table, COW, the block-size trade-off, the 10 knobs and the trap of confusing “fragmentation solved” with “zero waste”. On the generic 4×H100 SXM cluster.

Where you are: the pantry, underneath the scheduler

Go back to the kitchen of the previous post. The front-of-house manager assembles trays, but behind him there is a pantry where the ingredients each table accumulates over its meal are stored, its KV cache. The question in this post is how that pantry is organised.

The naive way: each table gets a whole contiguous shelf, sized for the greediest imaginable customer. The problem is obvious: a table that orders little leaves almost all of its shelf empty, but that shelf is already reserved and nobody else can use it. With many tables, the pantry fills up with half-empty shelves and no new table fits, even though the gaps added together leave plenty of room.

The PagedAttention way: the pantry is divided into small, identical pigeonholes. Each table is given the pigeonholes it needs, one at a time, wherever there is room, and they do not have to be next to each other. A map book records which physical pigeonholes each table has and in what order. When a table leaves, its pigeonholes go back to the pile. There are no half-empty shelves: the only waste is the last pigeonhole of each table, the half-filled one. That is, almost literally, the virtual memory of an operating system applied to the KV cache.

Why contiguous memory fragmented

Reserving contiguously and up front produces three distinct kinds of waste:

Reservation fragmentation. You set aside space for max_model_len (8192 tokens, say) even though the request will use 800. Reserved and never used.
Internal fragmentation. Within what you reserved, everything above what you actually use at any given moment.
External fragmentation. Gaps between contiguous reservations that are too small for a new request, even though added together they would be plenty.

The original vLLM paper measured that earlier systems wasted 60% to 80% of the KV memory through these three routes (Kwon et al., SOSP 2023). That is: on a GPU with room for 100 real requests, only 20-40 fitted. Paging attacks all three at once. It eliminates reservation waste (on-demand allocation) and external waste (blocks need not be contiguous), and leaves only a remainder of the internal kind: the last partial block.

The mechanism: blocks, block table and the kernel gather

The KV of a sequence is chopped into logical blocks of $b$ tokens (by default $b = 16$). Each logical block is mapped, via the block table, to a physical block somewhere in the HBM. The block table is the “map book”: a per-request list of which physical block corresponds to each logical one (vLLM implementation).

The key point is that the attention kernel knows how to read like that. Instead of assuming a contiguous KV tensor, the PagedAttention kernel receives the block table and does a gather: for each sequence it walks its physical blocks in logical order and reads K and V as if they were together. That is why PagedAttention is not just a data structure: it is a kernel that knows how to attend over paged memory. And that is why the attention backend and the block manager are tied together, the second deciding where the KV lives and the first knowing how to read it from there.

The block manager: the pantry librarian

The block manager (in V1, the KVCacheManager) is the one that keeps the map book. Its responsibilities:

Maintain a pool of free physical blocks (a queue of available blocks).
Allocate blocks to a sequence as it grows (a new block every $b$ tokens).
Free the blocks when the sequence finishes or is preempted.
Maintain the block tables (logical→physical) of each request.
Manage prefix caching: detect blocks with identical content and share them.
When free blocks run out, tell the scheduler to preempt (see the scheduler post).

When the block manager says “no blocks left”, the scheduler has to take somebody off the train. That is why the two budgets, tokens and blocks, are the two hands of the same engine.

Here is the elegant part. If two requests start with the same prefix, the same system prompt, the same context document, the first KV blocks of both are identical byte for byte. Why compute and store them twice?

vLLM gives each block a hash summarising its content (the tokens that formed it, plus the hash of the previous block, so that the hash captures position). It keeps a global table of blocks by hash. When a new request produces a block whose hash already exists, it allocates no new memory: it points its block table at the physical block that was already there (automatic prefix caching, vLLM).

Copy-on-write is the safeguard: while A and B share a block, neither can modify it. The moment one of the two needs to write something different into that block (because their sequences diverge, or in parallel sampling / beam search where several branches share a prefix), the block manager copies the block for that branch and only then writes (details, vLLM). It is the same COW that fork() uses in an OS: share until somebody writes.

The saving is direct: if 50 requests share a system prompt of 1,000 tokens, instead of 50 copies of that prefix’s KV there is one. How to maximise that saving in practice is the subject of the prefix cache hit rate post.

The maths that matter: how much KV, how many blocks

KV bytes per token. For a transformer block with $L$ layers, $h_{kv}$ KV heads (GQA), per-head dimension $d$ and $s$ bytes per element (2 in FP16):

$$\text{KV/token} = 2 \cdot L \cdot h_{kv} \cdot d \cdot s$$

For a Llama-70B ($L=80$, $h_{kv}=8$, $d=128$, FP16):

$$\text{KV/token} = 2 \cdot 80 \cdot 8 \cdot 128 \cdot 2 = 327680 \text{ bytes} \approx 320 \text{ KB}$$

A block of 16 tokens takes up $16 \times 320\,\text{KB} = 5.12$ MB.

How many requests fit. If after loading the weights ~120 GB of the node’s 320 are left for KV:

$$\text{KV tokens} = \frac{120 \cdot 10^9}{327680} \approx 366000 \text{ tokens} \approx 22900 \text{ blocks}$$

With average contexts of 4,000 tokens (250 blocks each), that is ~90 concurrent requests. That number, not max_num_seqs, is the real concurrency ceiling, and it is exactly the “block budget” of the scheduler.

The waste that remains. PagedAttention does not get to zero: each sequence wastes, on average, half a block (the last one, half-filled). With 16-token blocks and sequences of 4,000, that is $8 / 4000 = 0.2\%$ per sequence, and the famous aggregate “~4%” from the paper includes other overheads. The lesson: the waste does not disappear, it is bounded to the size of one block.

The block-size trade-off

The block_size (16 by default) is a trade-off, not a magic constant:

Block	Advantage	Drawback
Small (8)	less internal waste; finer-grained prefix sharing	more block table entries; more management and gather overhead
Large (32)	fewer metadata; more efficient gather	more waste in the last block; prefix caching shares at a coarser grain (fewer hits)

A large block shares worse: prefix caching can only reuse complete and identical blocks, so with 32-token blocks two prompts that agree on 20 tokens share nothing (they do not fill a common block), whereas with 8-token blocks they share two blocks. The default of 16 is the point vLLM found reasonable for most workloads; it is worth testing if your workload has short, heavily repeated prefixes.

The 10 knobs

#	Knob	What it controls	Cost if you overdo it
1	`block_size`	tokens per block	waste / overhead (see table)
2	`enable_prefix_caching`	sharing blocks by hash	almost none; usually on
3	`gpu_memory_utilization`	how many physical blocks there are	OOM if too high
4	`kv_cache_dtype` (FP8)	bytes per KV element	quality (measure, do not assume)
5	`swap_space`	blocks that fit on the host (SWAP)	PCIe traffic on preemption
6	`max_model_len`	maximum length per request	fewer requests if too high
7	eviction policy	who gets blocks taken away	prefix cache hit rate
8	sliding window	discarding old KV	quality on long contexts
9	TP / KV sharding	splitting the KV across GPUs	NVLink traffic
10	num_gpu_blocks (override)	forcing the block count	OOM or underuse

How it connects to the rest of the stack

With the scheduler. The “block budget” of the scheduler is administered by this block manager. When it says there are no blocks, the scheduler preempts (RECOMPUTE by default).

With the KV cache. The KV cache post explains what each token stores; this one, how it is placed in memory without fragmenting.

With prefix caching. COW and block hashes are the mechanism; hit rate engineering is how to squeeze it (prompt structure, prefix-aware routing).

With KV quantisation. Moving the KV to FP8 halves the bytes/token: the same node holds twice the tokens. It is the most direct lever on concurrency.

With the attention backend. The FlashAttention/FlashInfer kernel has to know how to attend over paged blocks; the block manager decides where they live, the kernel knows how to read them.

With disaggregated serving. Moving a request from a prefill pool to a decode pool in disaggregated serving is, at bottom, transferring its KV blocks between engines, over NVLink or over the network.

With multi-LoRA. In multi-LoRA serving, the base shares prefix KV between requests from different adapters as long as the prefix is identical.

Traps and things that are not what they seem

“PagedAttention eliminates waste.” It bounds it, it does not eliminate it. What remains is the last partial block per sequence (~half a block) plus the block table metadata. It is ~4% instead of 60-80%, but it is not zero. Sizing as if it were zero leaves you without a cushion.

“Bigger blocks always perform better.” The gather is somewhat more efficient, yes, but you lose sharing granularity: prefix caching shares worse and the waste in the last block grows. On workloads with many short repeated prefixes, small blocks can win.

“Prefix caching shares KV between users, that is a privacy problem.” It shares only blocks that are identical token by token (same system prompt, same document). It does not expose one user’s content to another: if the tokens do not match, there is no common block. What is worth watching is information leaking through timing side channels (a hit is faster than a miss), relevant only in very adversarial multi-tenant scenarios.

“FP8 in the KV is free: twice the concurrency.” It does double the tokens that fit, yes, but KV in FP8 degrades quality measurably on long contexts. It is a real lever, not a free lunch: quality has to be measured (FP8 end-to-end), not assumed.

“Going back to contiguous memory would be simpler and almost as good.” That is nostalgia for the contiguous tensor. The “simple” option reintroduces 60-80% fragmentation: on a GPU, that is the difference between 30 and 90 concurrent requests. The complexity of the block table pays for itself many times over.

SWAP versus RECOMPUTE when preempting. Configuring a lot of swap_space “so as not to lose KV” puts gigabyte transfers over PCIe on the critical path. In V1, RECOMPUTE is usually better; swap is for specific cases.

Conclusion

The bottleneck in serving an LLM was never only how much memory you have, but how you hand it out. The first servers treated the KV cache as a contiguous shelf per client and threw two thirds of the HBM in the bin without it showing up on any dashboard. PagedAttention stole from the operating system its best idea of fifty years ago, paging, and applied it at the exact place where it hurt: small pigeonholes, a map book, on-demand allocation and, as a bonus, the possibility that two requests starting the same way share the same pigeonholes until they stop resembling each other. The result is not magic: the waste is still there, but bounded to the size of one block instead of to the size of the worst imaginable case. And that difference, from 70% to 4%, is what turned a GPU serving thirty clients into one serving ninety, without touching the hardware. The pantry did not get bigger; it got better organised.

References

W. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023): https://arxiv.org/pdf/2309.06180.
vLLM, Automatic Prefix Caching (design, block hashing): https://docs.vllm.ai/en/v0.8.1/design/automatic_prefix_caching.html.
vLLM, Automatic Prefix Caching — Implementation (block table, COW): https://docs.vllm.ai/en/v0.6.1/automatic_prefix_caching/details.html.
H. Elshafie, Paged Attention from First Principles: A View Inside vLLM: https://hamzaelshafie.bearblog.dev/paged-attention-from-first-principles-a-view-inside-vllm/.
vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention (alternative, critical context): https://arxiv.org/pdf/2405.04437.

The pass: the maître d' who builds every round — vLLM's scheduler step

Mon, 08 Jun 2026 05:00:00 +0200

Part of the under the engine series. Earlier posts looked at the silicon that runs the kernels (SMs and CUDA graphs) and at loading the weights (from disk to HBM). This one goes up a floor: who decides what runs in each forward. Before the GPU launches a single kernel, someone has had to build this round’s order. That someone is the scheduler, and it is the heart of continuous batching.

TL;DR

An LLM server does not serve one whole request and then the next: it advances every live request at once, a little on each engine iteration. The piece that decides how far each one advances in each step is the scheduler, and its output is deceptively simple: a dictionary {req_id: number of tokens} that the model runner turns into a single forward on the GPU. vLLM V1’s most important decision was to erase the distinction between prefill and decode: for the scheduler, a prompt token and a freshly generated token are the same thing, tokens that need processing, and that is why it can put a 4000-token prompt in the same batch as 200 one-token decodes. There are four pieces: the token budget (max_num_batched_tokens), a fixed-size tray filled every round; chunked prefill, which splits a huge prompt into chunks so it does not hog the tray and blow up everyone else’s latency; the two queues (waiting, in arrival order, and running); and preemption, because when the KV cache runs out somebody has to get off the train. This post explains the loop, the arithmetic of the budget and of concurrency, the 10 knobs and the headline trap: raising the budget improves throughput but worsens ITL (inter-token latency), and almost nobody measures both at once. On the generic 4×H100 SXM cluster.

Where you are: the pass, before the kitchen starts

Picture the pass of a restaurant with a busy dining room. There is no one cook per customer; there is a shared kitchen and a maître d’ who, every few seconds, looks at all the open orders and builds a tray to send to the stoves. That tray holds, say, 8000 “units of work”. The maître d’ decides what goes in: tables already eating (requests in decode) get one more dish each; new tables that have just ordered (requests in prefill, with their whole prompt still to process) get as much of their order as fits in whatever is left of the tray. The tray goes out, the kitchen executes it in one go, and it starts again. Hundreds of times per second.

That maître d’ is the scheduler. The kitchen is the GPU running a forward pass. And the house rule is that the kitchen never stops to wait for a single table: if a new order is enormous (a 30,000-token prompt), it is not sent whole in one go blocking everyone else, it is sent in chunks. That is, in one sentence, the whole mechanics of vLLM’s scheduler.

The engine loop: one dictionary per iteration

The inference engine is a very short loop. On each pass:

The scheduler looks at the queues and produces a decision.
The model runner runs a forward with that batch on the GPU.
The sampler draws one new token for each active sequence.
State is updated (KV cache, positions, finished requests) and it goes back to 1.

The surprising part is the shape of the decision in step 1. In vLLM V1 it is not a complex structure with phases: it is literally a dictionary

$$\text{schedule} = \{\, \text{req id} \rightarrow n_\text{tokens} \,\}$$

that says, for each request entering this round, how many tokens of it get processed. For a request that is generating text, n_tokens = 1 (one autoregressive step). For a new request, n_tokens can be the entire length of its prompt. And it can be anything in between, a chunk of prompt, under chunked prefill, prefix caching or speculative decoding (vLLM V1 docs).

That the decision fits in a {id: number} dictionary is not a neat implementation detail: it is why continuous batching works. Because the scheduler does not think in “phases” but in “how many tokens for each”, it can mix requests at any point of their life in the same forward. The GPU receives a single heterogeneous token tensor and processes it in one go.

The death of the prefill/decode distinction

This is the idea that costs the most effort and matters the most. In the first LLM server architectures, a request lived in two separate phases: first prefill (process the whole prompt and fill the KV cache), then decode (generate token by token). The scheduler had to choreograph the move from one phase to the other, and mixing them was hard.

vLLM V1 removed the distinction (V1 design). The scheduler treats prompt tokens and generated tokens uniformly: they are all tokens the model has to process in a forward. The practical consequence is enormous. A 4000-token prompt and a sequence that has generated 800 tokens and needs one more are, to the scheduler, “4000 tokens of request A” and “1 token of request B”. They fit together on the same tray. There is no phase choreography, just a budget to hand out.

This unlocks the pattern that really pays: mixing prefill and decode in every step. Prefill is compute-bound work (a lot of matrix to multiply); decode is memory-bound (little compute, a lot of KV to move). Mixing them in the same batch fills the gaps: while the GPU is busy with the heavy prefill, it advances the light decodes “for free”. It is the same efficiency principle as continuous batching, taken to its cleanest form.

The token budget: the fixed-size tray

The tray has a size: max_num_batched_tokens. It is the maximum number of tokens the scheduler can put in a single step (vLLM optimisation). The policy, with chunked prefill on (it always is in V1), is clear:

Decodes first. Room is reserved for one token per request in the running queue. They are cheap, and they are diners already eating: they are not left waiting.
What is left goes to prefills. With the remaining budget, prompt tokens from waiting requests are added, in arrival order (FCFS), split into chunks if needed.

An example with numbers. Budget max_num_batched_tokens = 8192, and at this instant there are 200 requests generating:

$$\text{decode} = 200 \times 1 = 200 \text{ tokens}$$ $$\text{free budget} = 8192 - 200 = 7992 \text{ tokens for prefill}$$

If a new request arrives with a 4000-token prompt, it fits whole in this step (4000 < 7992) and 3992 are left for another. If one arrives with 30,000 tokens, it does not fit: the scheduler sends it a 7992-token chunk this step, and the remaining 22,008 in later steps. That is chunked prefill.

The budget matters because it fixes how many forwards are needed to swallow a prompt. A 30,000-token prompt with an 8192 tray takes 4 steps in prefill alone before it emits its first token. With a 2048 tray it takes 15 steps, but each of those steps leaves more room for other people’s decodes, so the other diners feel the jam less.

The two queues and preemption: when someone gets off the train

The scheduler handles two queues. The waiting queue holds requests that have not started yet (their prompt has not been processed), served in arrival order, FCFS by default, though there is a priority policy. The running queue holds those already alive and generating (vLLM scheduling).

There is a second budget, harder than the token one: the KV cache. Every live token occupies KV blocks in HBM, and they are finite (gpu_memory_utilization fixes them). When the scheduler wants to advance the running requests but there are no free blocks for the next token’s KV, someone has to get off: that is preemption.

vLLM V1 preempts by RECOMPUTE by default, not by SWAP (V1 guide). The difference:

SWAP: copy the victim’s KV to host RAM and bring it back later. It moves gigabytes across PCIe (see the PCIe and P2P post).
RECOMPUTE: throw the victim’s KV away and, when there is room again, redo its prefill from scratch. It sounds expensive, but in the V1 architecture it works out cheaper than swapping, because prefill is work the GPU does very fast and you save the round trip across the bus.

The victim is usually the newest request in the running queue (so as not to penalise whoever has been waiting longest for their answer). The danger is thrashing: admit too many requests at once and the system enters a preempt-recompute-preempt cycle that drives throughput into the floor. That is why the second ceiling exists.

The arithmetic that matters: concurrency and the budget trade-off

How many requests fit at once. The real concurrency limit is not max_num_seqs (the nominal ceiling on simultaneous sequences); it is usually the KV cache. If a node has $B$ free KV blocks, each block holds $b$ tokens (16 by default), and each request occupies $L$ context tokens on average, the maximum sustainable concurrency is:

$$N_\text{max} \approx \frac{B \cdot b}{L}$$

Take a 70B model in FP16 on 4×H100 SXM (320 GB), with the bulk of HBM in weights and, say, ~120 GB free for KV. With a KV of ~0.3 MB/token (an order-of-magnitude figure, it depends on layers and heads), that is ~400,000 KV tokens. With average contexts of 4000 tokens:

$$N_\text{max} \approx \frac{400000}{4000} = 100 \text{ concurrent requests}$$

Raising max_num_seqs to 400 does not give you 400 concurrent: it gives you preemption and thrashing as soon as contexts grow. The KV rules.

The budget trade-off. Raising max_num_batched_tokens puts more work in each forward, so fewer forwards for the same total work, and more throughput. But a large budget lets a huge prefill occupy almost the whole tray in one step, and that step takes longer, so everyone else’s decodes wait for that entire step, and ITL (inter-token latency) rises for everybody. The rule of thumb (vLLM optimisation):

Budget	Effect	At the cost of
Low (e.g. 2048)	more interleaving, stable ITL	less peak throughput
High (e.g. 16384)	maximum throughput	ITL spikes when a large prefill arrives

There is no “correct” value: there is a point for your load. And you only find it by measuring throughput and ITL at the same time, which is exactly what almost nobody does.

The scheduler’s 10 knobs

#	Knob	What it controls	Cost if you overdo it
1	`max_num_batched_tokens`	tray size per step	high ITL if too large
2	`max_num_seqs`	nominal concurrency ceiling	preemption if the KV falls short
3	`gpu_memory_utilization`	KV blocks available	OOM if too high
4	chunked prefill (threshold)	prompt chunk size	chunking overhead if too fine
5	policy (`fcfs`/`priority`)	who gets served first	starvation of low priority
6	preemption mode	RECOMPUTE vs SWAP	PCIe traffic / recompute
7	`enable_prefix_caching`	reuse prefix KV	little; almost always on
8	`max_model_len`	maximum context per request	pessimistic KV reservation
9	CUDA graph sizes	align batch with buckets	padding / capture (see below)
10	speculative tokens	extra tokens per step	wasted work if acceptance drops

How it connects to the rest of the stack

With continuous batching. The scheduler is continuous batching turned into code. The batching post explains the what (advance every request at once); this one explains the how (the per-step token dictionary).

With the KV cache and the block manager. The second budget, the blocks, is managed by the PagedAttention block manager. The scheduler asks for blocks; if there are none, it preempts. The two pieces are coupled through memory.

With CUDA graphs. CUDA graphs are captured for specific batch sizes (buckets). The scheduler should produce batches whose size lands in those buckets to avoid padding; otherwise part of the graph’s benefit is lost.

With chunked prefill and the prefix cache. Chunking a prompt interacts with prefix caching: chunks that match an already cached prefix skip the compute, and the scheduler reflects that by lowering that request’s n_tokens.

With speculative decoding. Speculative decoding makes a step verify several tokens at once; the scheduler models it as n_tokens > 1 for a request in decode.

With disaggregated serving. In disaggregated prefill/decode there are two schedulers, one per pool, each with its own budget; the phase distinction that V1 erased inside an engine reappears between engines.

With autoscaling. The metrics that drive autoscaling with KEDA, waiting-queue length and preempted requests, come straight out of the scheduler’s state.

Traps and things that are not what they look like

“Raising max_num_batched_tokens always helps.” It improves throughput and worsens ITL. If you only look at tokens/s in a large-batch benchmark, you “confirm” that more is better; in interactive production, your users feel the stutter. Measure both metrics or you are not measuring.

“The engine does all the prefills first and then the decodes.” That is the old architecture’s intuition. In V1 there are no phases: each step mixes prefill and decode according to the budget. Reasoning with the phase model leads to wrong conclusions about why latency rises.

“More max_num_seqs = more throughput.” Only until the KV cache runs out. Past that, more nominal concurrency produces preemption, and cascading preemption (thrashing) lowers throughput. The real ceiling is the KV, not the parameter.

“RECOMPUTE is a waste, SWAP is better.” In V1, RECOMPUTE usually wins: prefill is extremely fast on GPU and SWAP puts a gigabyte-scale round trip across PCIe on the critical path. Switching to SWAP “to avoid recomputing” can make latency worse.

“The scheduler is the bottleneck.” Almost never. The decision is a dictionary built in microseconds; the cost of the round is the forward on the GPU, three orders of magnitude above. If your scheduling CPU shows up in the profiler, the problem is usually host-thread jitter (see NUMA and CPU isolation), not the scheduler logic.

Chunked prefill that is too fine. Tiny chunks make a large prompt take many steps and add fixed per-step overhead. Chunking is there to bound the ITL impact, not to pulverise the prompt.

Conclusion

All the power of a modern LLM server, swallowing hundreds of requests at once without any of them blocking the rest, rests on a decision that fits in a {request: how many tokens} dictionary, taken hundreds of times per second. The idea that made it possible was not a faster kernel or a bigger GPU: it was to stop thinking in phases. When a prompt token and a generated token are the same thing, the scheduler can fill every tray by mixing the heavy and the light, and the kitchen never stops. The rest is two budgets, tokens and KV blocks, and a rule for when the second runs out. The maître d’ does not cook; he only decides what goes to the stoves in each round. But that decision, repeated without pause, is what separates an idle GPU waiting for orders from a kitchen running flat out. And the uncomfortable lesson for whoever is tuning: throughput and latency meet in the budget, and optimising one blindly means worsening the other without noticing.

References

vLLM, vLLM V1: A Major Upgrade to vLLM’s Core Architecture: https://openlm.ai/vllm-v1/.
vLLM, vLLM V1 User Guide (chunked prefill por defecto, preemption RECOMPUTE): https://docs.vllm.ai/en/v0.9.2/usage/v1_guide.html.
vLLM, Optimization and Tuning (max_num_batched_tokens, presupuesto y trade-off): https://docs.vllm.ai/en/stable/configuration/optimization/.
A. Wong, Understanding vLLM Scheduling: Token Budgets, Chunked Prefill, and Policies: https://audreywongkg.medium.com/understanding-vllm-scheduling-token-budgets-chunked-prefill-and-policies-2c879e3980e3.
W. Kwon et al., Efficient Memory Management for Large Language Model Serving with PagedAttention (SOSP 2023): https://arxiv.org/pdf/2309.06180.

The chef who calls out every order: SMs, CUDA streams and CUDA graphs, or why the GPU gets bored generating tokens

Sun, 07 Jun 2026 09:00:00 +0200

This closes the “outside the API” pair. The previous post moved the weights from disk into HBM; here we look at what happens once they are inside, in the silicon that executes them. It is the floor below the kernel launch that the NUMA post mentioned without opening: who launches those kernels, how, and why in decode the GPU spends more time waiting for orders than computing.

TL;DR

An H100 has ~132 streaming multiprocessors (SMs), the “hobs” that execute the compute, and occupancy measures how many warps (groups of 32 threads) it has active to hide latency. But the decode bottleneck is rarely the power of those SMs. Each decode step launches hundreds of tiny kernels (several projections per layer × ~80 layers), and each kernel launch costs 5-10 µs of serial CPU. Since decode kernels are small (small batch, a single token), the GPU finishes them before the CPU calls out the next one: bubbles appear and the GPU sits bored waiting for orders. That regime is called launch-bound, and it is the deep reason, not power, not memory, why --enforce-eager delivers 54 tok/s where with optimisations you reach 89-140. The solution is CUDA graphs: record the entire kernel sequence once and replay it as a single submission, removing the per-launch overhead (~28% of the latency per iteration). vLLM captures ~102 graphs at startup and pads the real batch to the nearest bucket so it can replay a graph with fixed shapes. This post explains SM, occupancy, streams, launch overhead with the maths, CUDA graphs, the 10 knobs, and the trap that this capture is the second half of the cold start from the previous post. With scepticism about what actually moves the needle. On the generic 4×H100 SXM cluster.

Where you are: the silicon, below the kernel launch

The analogy: the chef who calls out every order

Last scene in the restaurant of the series. The kitchen is set up, the pantry is stocked (the previous post). Now the food has to be plated. The hobs are the SMs: 132 stations cooking in parallel. The head chef is the CPU: he calls out the orders, each kernel launch is a shout of “one matrix multiplication, coming up!”. The cooks (the SMs) execute what the chef calls out.

In prefill, processing the whole prompt, each order is a huge dish: a giant matmul over hundreds of tokens at once. The chef calls out an order and the hobs take a good while to produce it. The chef has plenty of time to call out the next one. The hobs are flat out: compute-bound.

In decode, generating one token at a time, each order is minuscule: a matmul over a single token. The hob finishes it in an instant… and sits staring at the chef waiting for the next one. But the chef can only call out one order every 5-10 µs, and there are hundreds of orders per token. The hobs, lightning fast, get bored between shouts. The restaurant is not slow because the cooks are bad: it is slow because the chef does not call out fast enough. That is the launch-bound regime.

The solution is not more hobs or faster cooks. It is to stop calling out order by order. If the chef prints the whole night’s sequence on a single sheet and hands it to the line (“do this, in this order, without waiting for me”), the hobs run without pauses. That is a CUDA graph: record the kernel sequence once and replay it in one go, without the CPU calling out each one. And --enforce-eager is exactly the opposite: forcing the chef to call out order by order, all night long.

The mechanism: SM, warps and occupancy

An H100 SXM has ~132 SMs. Each SM executes threads in groups of 32 called warps, and can have several warps “in flight” at once. Occupancy is the fraction of active warps against the maximum the SM supports. What is the point of having many active warps? To hide latency: while one warp waits for data from HBM (hundreds of cycles), the SM executes another ready warp. With few warps, the SM runs out of anyone to give a turn to and stalls.

But, and this is key, occupancy is a necessary, not sufficient condition, and it only matters if the SM has work to do. In decode, the typical problem is not low occupancy inside a kernel: it is that between kernels the SM has nothing, because the CPU has not launched the next one yet. Raising the occupancy of a kernel that lasts 8 µs does not help if the GPU spends 6 µs waiting for it to be launched.

The mechanism: streams, the order queue

A CUDA stream is a queue of operations that the GPU executes in order. Operations in the same stream are sequential; operations in different streams can overlap. That is what allows, for instance, copying data H2D in one stream while another stream computes, the compute/copy overlap. vLLM uses streams to overlap work, but a stream on its own does not remove the cost of launching each kernel: it only decides ordering and parallelism. The launch cost is still there, order by order, until the graphs come in.

The maths that matters: when the GPU is left waiting

The number that governs everything: a kernel launch costs 5-10 µs of CPU, serially. Take a Llama-70B with ~80 layers. Each layer, without fusion, launches on the order of ~10 kernels (Q/K/V projections, attention, output projection, the two or three MLP matmuls, the normalisations, RoPE and so on). That gives:

$$ N_{\text{kernels}} \approx 80 \text{ layers} \times 10 \approx 800 \text{ launches per token} $$

At 5 µs per launch, serially:

$$ T_{\text{launch}} \approx 800 \times 5\,\mu s = 4{.}0 \text{ ms per token} $$

Those 4 ms are CPU calling out orders only, not counting how long the SMs take to cook. If the GPU could compute instantaneously, the launch ceiling would be ~250 tok/s, and with synchronisation points between kernels, worse. Now compare with the memory ceiling of decode: each token reads the 140 GB of weights once from HBM:

$$ T_{\text{mem}} = \frac{140 \text{ GB}}{3{.}35 \text{ TB/s}} \approx 42 \text{ ms} \;\Rightarrow\; \approx 24 \text{ tok/s (one sequence, no batch)} $$

Here is the subtlety almost nobody has in mind. For a single sequence, decode is memory-bound at ~24 tok/s, and the 4 ms of launch fit inside the 42 ms of reading: the launch hides. But batching changes everything. When serving a batch of B sequences, the weights are read once and serve all B, so the memory cost per token amortises and falls. The GPU stops being memory-bound… and what was underneath emerges: the launch cost, which does not amortise with batch because you have to launch the same kernel sequence regardless. The result: the better you batch, the more launch-bound you become, and the more CUDA graphs pay off. That is why the raw measurement confirms it, --enforce-eager gives 54 tok/s where graphs give 89, and up to 8× in configurations where decode is very small and launch dominates completely.

vLLM’s CUDA graphs, specifically

vLLM does not capture a single graph: it captures ~102 at startup, on the order of 51 piecewise (for the mixed prefill+decode steps) and 51 full (for pure decode). Each one is recorded for a fixed batch size (a bucket: 1, 2, 4, 8… up to a maximum). In service the real batch almost never falls exactly on a bucket, so vLLM pads with zeros up to the next bucket up, replays that graph, and trims the output to the real size. That is the price of graphs: they need static shapes, and padding is what makes them static.

This has two consequences that show up in the knobs and the traps:

Capture costs time and memory. Recording 102 graphs at startup adds seconds to the cold start, the second half of the startup that the previous post left pending, and consumes HBM (each graph retains its buffers). The FULL_AND_PIECEWISE mode (the default) is the fastest in service but the one that asks for the most memory and the longest capture; FULL_DECODE_ONLY saves both in exchange for not accelerating the mixed steps.

Padding wastes some compute. Padding a batch of 33 up to the bucket of 64 computes 31 phantom sequences. It is a small cost against what removing the launch overhead saves, but it exists, and it grows if the buckets are badly chosen.

The 10 knobs worth touching

Knob 1 — Measure whether decode is launch-bound

Before touching anything: is the GPU computing or waiting? With nsys (Nsight Systems) you can see the gaps between kernels on the timeline; if there are gaps in decode, it is launch-bound and graphs will help. If the GPU is at 100% with no gaps, the bottleneck is something else (memory or compute) and graphs will not work miracles. nvidia-smi dmon showing low utilisation in decode but poor TPS is the cheap signal.

Knob 2 — Do not use `--enforce-eager` in production

--enforce-eager disables CUDA graphs. It is a debugging tool (to isolate which kernel fails), not a production one. Leaving it on “because it used to start up” throws away 26-50% of decode throughput. If it is in your production command, take it out and measure.

Knob 3 — Capture buckets (`cudagraph_capture_sizes`)

Which batch sizes to capture. Buckets too far apart make padding expensive; too many of them make capture slow and eat HBM. Tuning them to the real distribution of batch sizes you see in production is the fine adjustment, but only after measuring that distribution.

Knob 4 — CUDA graph mode

FULL_AND_PIECEWISE (default, fastest, more memory/capture), FULL_DECODE_ONLY (saves memory and capture, ideal for pure decode pods in disaggregated serving), PIECEWISE, or NONE (= eager). The right mode depends on whether the pod does pure or mixed decode.

Knob 5 — `torch.compile`

vLLM leans on torch.compile to fuse and optimise kernels before capturing them in graphs. Fewer kernels (fusion) = fewer launches = less dependence on the graph and better decode even in eager. The compilation level is a knob, with its startup time cost.

Knob 6 — Batch size: fill the hobs

Memory-bound decode amortises with batching (as we saw in continuous batching): read the weights once for B sequences. More batch = more SM occupancy and more memory amortisation. The limit is set by the HBM available for the KV cache. It is the knob that moves aggregate throughput most.

Knob 7 — Do not break stream overlap

vLLM overlaps compute and copy with streams. Patching the code to “simplify” can serialise what was overlapped. If you do not know why there are several streams, do not collapse them.

Knob 8 — Persistence mode + locked clocks

nvidia-smi -pm 1 keeps the driver resident (avoiding reinitialisations that add launch latency). Locking clocks to the boost frequency stops the GPU dropping P-state between tiny decode kernels and paying the ramp-up latency. It is the same anti-jitter spirit as the NUMA post, applied to the GPU.

Knob 9 — Fused kernels (FlashAttention, FP8 kernels)

Fewer kernels = fewer orders to call out. FlashAttention fuses attention into one kernel instead of several; fused FP8 kernels reduce the count. Fusion attacks the problem at the root: it does not speed up the launch, it removes launches.

Knob 10 — Accept the capture cost in the cold start

Graph capture adds seconds to startup. In a pod that lives for hours, it amortises easily. In one that scales up and down every minute, that cost is paid over and over; there, FULL_DECODE_ONLY (shorter capture) or accepting slightly less throughput can be worth it. It is the same warm-vs-elastic tension as in the cold start.

Summary table

#	Knob	What it attacks	Risk / cost
1	`nsys` / `dmon`	knowing whether it is launch-bound	none; do it first
2	remove `--enforce-eager`	graphs disabled	it was for debugging; reactivate if a bug returns
3	capture buckets	expensive padding / slow capture	requires measuring the real distribution
4	graph mode	memory and capture	less coverage on mixed steps
5	`torch.compile`	unfused kernels	startup time
6	batch size	occupancy + memory	HBM for KV cache
7	streams	broken overlap	do not touch if not understood
8	persistence + clocks	jitter / P-states	electrical consumption
9	fused kernels	number of launches	kernel compatibility
10	capture vs cold start	slower startup	less throughput if trimmed

How this connects with the rest of the stack

With the cold start. CUDA graph capture is the second half of the startup that the previous post opened: loading weights + capturing graphs = the complete cold start.

With continuous batching. Continuous batching is what makes decode launch-bound (it amortises memory and leaves the launch exposed), and that is why graphs and batching reinforce each other.

With the KV cache. The KV cache decides how much batch fits in HBM, and the batch decides SM occupancy and how much launch overhead matters. Everything is coupled through memory.

With the interconnect. In TP, between the compute kernels there are all-reduces (NVLink/NCCL) that also get launched and synchronised. vLLM’s custom all-reduce is integrated into the same graph so the sequence is not broken by a CPU synchronisation.

With NUMA. Who launches the kernels is the CPU from the host post; if that thread suffers jitter or lands on the wrong socket, launch overhead gets worse. Graphs reduce the dependence on that thread, which is another reason they help.

With disaggregated serving. The pure decode pods of disaggregated serving are the ideal case for FULL_DECODE_ONLY: they maximise the graph benefit exactly in the most launch-bound phase.

Traps and things that are not what they look like

“Raising occupancy will fix slow decode.” Not if the problem is launch-bound. Occupancy matters inside a kernel that has work; if the GPU is idle between kernels waiting for the CPU, more occupancy does not touch that bubble. Measure before optimising what is not the bottleneck.

“CUDA graphs always speed things up.” They speed things up when decode is launch-bound. If the GPU is already at 100% (compute-bound in prefill, or memory saturated with a huge batch), graphs add little. Their territory is decode with small kernels.

"--enforce-eager gives more stable results." It gives slower results. The stability it appears to give is that it avoids graph capture bugs on new hardware (for example a newly supported architecture). It is a temporary patch, not a production configuration.

Capturing too many buckets “just in case”. Each bucket adds capture time and HBM. Capturing 30 sizes when in production you only see 4 means paying cold start and memory for graphs that are never replayed. Tune to the real distribution.

Confusing utilisation with efficiency. nvidia-smi at 100% “utilisation” only says there is a kernel running, not that the SM is full of useful work. A low-occupancy kernel keeps “utilisation” high while wasting the SM. nvidia-smi utilisation is a coarse thermometer; to know whether the silicon is performing you need nsys/DCGM and to look at real occupancy and gaps.

Optimising the silicon before memory. If decode is limited by HBM bandwidth (large batch, large model), fighting with graphs and occupancy is polishing what is not the bottleneck. The right order: measure the regime (memory / compute / launch) and attack the one in charge.

Conclusion

Intuition says that a GPU generating slow tokens is “working hard”. Almost never: in decode it is waiting for orders. The 132 SMs cook a tiny token in an instant and then sit staring at the CPU, which can only call out one order every 5-10 µs and has hundreds to call out per token. That bottleneck, neither power nor memory but launch, is invisible on any dashboard that looks at “GPU utilisation”, and it is the real reason why --enforce-eager performs at half speed. CUDA graphs solve it with a simple idea: stop calling out order by order and hand over the whole sheet for the night, so the silicon runs without pauses. And there is an uncomfortable truth that reorders the optimisation priority: the better you batch, the more launch-bound you become, because batching kills the memory bottleneck and exposes the launch one. That is why graphs and batching are not separate optimisations: they are the same lever seen from two sides. The chef who learns not to call out every dish is what finally makes the kitchen run as fast as the hobs always could.

References

vLLM, CUDA Graphs (diseño, modos FULL/PIECEWISE, captura): https://docs.vllm.ai/en/stable/design/cuda_graphs/.
vLLM, Inside vLLM: Anatomy of a High-Throughput LLM Inference System: https://blog.vllm.ai/2025/09/05/anatomy-of-vllm.html.
NVIDIA, Getting Started with CUDA Graphs: https://developer.nvidia.com/blog/cuda-graphs/.
NVIDIA, Achieved Occupancy (ocupación de SM): https://archive.docs.nvidia.com/gameworks/content/developertools/desktop/analysis/report/cudaexperiments/kernellevel/achievedoccupancy.htm.
PyTorch, torch.compile y CUDA Graphs para inferencia LLM: https://docs.vllm.ai/en/stable/design/cuda_graphs/.
Understanding the Overheads of Launching CUDA Kernels (ICPP 2019): https://www.hpcs.cs.tsukuba.ac.jp/icpp2019/data/posters/Poster17-abst.pdf.

The pantry lift: from disk to HBM, or why the kitchen opens late

Sun, 07 Jun 2026 08:30:00 +0200

This is a trip down to the basement. The under the engine series optimised the hot path, what happens with every token once you are already serving. This post looks at the journey that comes before serving: how the weights climb from disk to HBM. It is the first of a pair on the two things that happen outside the API and that almost nobody times: model loading (this one) and execution on the silicon (the next one).

TL;DR

Before an inference pod generates its first token, it has to push the whole model into HBM. A Llama-70B in FP16 is 140 GB travelling along a path nobody draws: disk → page cache → host buffer → PCIe → HBM. Intuition fails here: HBM is not the bottleneck, it moves 3.35 TB/s and swallows 140 GB in 42 ms; the bottleneck is the supply chain. The Gen5 NVMe disk reads at ~14 GB/s (10 s for 140 GB); PCIe Gen5 copies host→GPU at ~50 GB/s (2.8 s); and the default safetensors loader, which deserialises tensor by tensor and bounces every byte through a CPU buffer, inflates all of that up to 30-60 s. That time is the cold start, and it is the hidden tax paid by autoscaling (scale-from-zero), by canary/blue-green and by disaggregated serving every time a pod is born. There are three families of solution, GPUDirect Storage (direct DMA disk→HBM, no bounce through the CPU), fastsafetensors (4.8-7.5× over the default loader) and the Run:ai Model Streamer (concurrent reads that saturate the disk), plus the simplest lever of all: move fewer bytes (FP8 is half of FP16). This post explains the path, the maths, the 10 knobs, and the cruellest trap: “the second time it loaded fast” is not your loader being good, it is the page cache lying to you. On the generic 4×H100 SXM cluster.

Where you are: the basement, before opening

The analogy: the pantry lift

Let us stay in the restaurant of the series. The shared table was the NVSwitch, the floor next door was NUMA, the maître was the kubelet. All of that describes the restaurant running, with diners seated. But there is a moment no post looked at: before opening, somebody has to bring the entire pantry up from the basement store to the kitchen.

The model weights are the ingredients. They live in the basement store (the disk). The kitchen, the hot line where tokens are plated, is the GPU’s HBM. And between one and the other there is a service lift: the path disk → host → PCIe → HBM. The kitchen cannot serve the first dish until the pantry is upstairs and in place. That restocking time is the cold start.

The intuition trap: the kitchen (HBM) is enormous and blazingly fast, it places ingredients at 3.35 TB/s. So we blame the kitchen when the restaurant opens late. But the kitchen is standing idle waiting for the lift. The bottleneck is never the hot line: it is the lift and the store. And worse still, there is a porter (the default loader) who, instead of carrying whole crates, takes the ingredients out one by one, writes each one down in a notebook and packs them again before sending them up. That porter, not the lift, is half the problem.

The mechanism: what really happens when a model loads

When vLLM starts up with a model in safetensors, the 140 GB of the Llama-70B FP16 make this journey:

Disk → page cache. The kernel reads the .safetensors files from NVMe into the page cache (host RAM). If this is the first time after a reboot, it is a physical read from disk (~14 GB/s on Gen5). If the files are already in page cache from an earlier start-up, this is almost free, and here is where the trap we will see is born.
Deserialise. The default Hugging Face loader mmaps the file and builds the tensors one by one, copying them into a CPU tensor before moving them. This is single-threaded CPU work that saturates neither the disk nor PCIe: most of the “slow” loading time goes here, not on moving bytes.
Host → HBM (H2D). Each tensor is copied from the host buffer into HBM over PCIe Gen5 x16 (~50 GB/s in practice). For the DMA to be efficient, the host buffer should be pinned, which connects directly with the hugepages and pinned memory of the NUMA post.
Place in HBM. HBM receives the 140 GB. At 3.35 TB/s, this takes 42 ms. It is never the bottleneck.

The path has a shortcut: GPUDirect Storage (GDS). Instead of bouncing through the CPU buffer (steps 2-3), a DMA engine close to the NVMe controller writes straight from disk to HBM, without involving the CPU. It is the same principle as the GPUDirect RDMA of the network: get the CPU out of the way. fastsafetensors uses GDS and reaches 26.4 GB/s reading a Llama-70B from NVMe across 4 GPUs.

Why the problem exists: the economics of bytes

The size of the model in bytes is decided by quantisation, and that sets the floor of the cold start, because all those bytes have to move before the first token. For a 70B parameter model:

Format	Bytes/param	Size 70B	Read 1 NVMe @14 GB/s	H2D PCIe @50 GB/s	HBM @3.35 TB/s
FP16 / BF16	2	140 GB	10.0 s	2.8 s	42 ms
FP8	1	70 GB	5.0 s	1.4 s	21 ms
INT4 (GPTQ/AWQ)	0.5	~35 GB	2.5 s	0.7 s	10 ms

Three readings of this table:

HBM never shows up as a problem. The last column is always milliseconds. Anyone saying “the GPU is slow to load” is blaming the wrong place.

Quantising is the most underrated cold start lever. Going from FP16 to FP8 does not only double inference throughput (less HBM bandwidth per token, as we saw in quantization): it also halves the cold start, because there are half as many bytes to push up. It is a two-for-one that sizing usually ignores.

The disk is the bottleneck on bytes; the loader is the bottleneck on time. The figures in the table are the theoretical floor, just moving bytes. The default loader adds single-threaded deserialisation on top, which is the difference between the theoretical 10 s and the real 30-60 s. That is why the solutions attack on two fronts: fewer bytes (quantisation) and a better porter (concurrent loaders / GDS).

The maths that matter: cold start as a tax on autoscaling

Cold start is not paid once. It is paid every time a pod is born. And on an elastic platform, pods are born continuously.

Take an autoscaling setup with KEDA that scales from 2 to 6 replicas when the queue grows. The 4 new replicas take this long to be ready:

$$ T_{\text{ready}} = T_{\text{schedule}} + T_{\text{pull-image}} + T_{\text{load-weights}} + T_{\text{cuda-graphs}} $$

With a Llama-70B FP16 and the default loader, $T_{\text{load-weights}}$ dominates: it can be 40 s of the ~60 s total. During those 40 s, the queue that triggered the autoscaling keeps growing, since the new replicas absorb no traffic until they have loaded. The real number in the formula is not “how many replicas”, it is how long each one takes to start serving, and that number is written by the loading path.

This has a harsh operational consequence: scale-to-zero is unworkable for workloads with a latency SLO if the cold start is 40 s. Nobody waits 40 s for the first token. The real elasticity of an inference platform is not limited by GPU availability, it is limited by how long that GPU takes to have the model inside it. Bringing the cold start down from 40 s to 8 s (with streamer + FP8) is what turns “theoretical scale-to-zero” into “usable scale-to-zero”.

The published numbers give the order of magnitude of the improvement: fastsafetensors cuts start-up from 12.39 s to 4.74 s on a Llama-2-13B over 4×L40S, and from 16.04 s to 6.88 s on 1×A100, 4.8-7.5× over the default deserialiser. The Run:ai Model Streamer loads in 4.88 s from S3 at concurrency 32 and 7.53 s from SSD IO2 at concurrency 8; integrated into vLLM, total time to ready drops to ~23 s from S3. There is no magic here: it is getting the CPU out of the loop (GDS) and reading in parallel (concurrency) to saturate the disk instead of leaving it half idle while one thread deserialises.

The 10 knobs worth touching

Knob 1 — Measure where the time goes (read vs deserialise vs H2D)

Before touching anything: time it. Is the time going on reading from disk, on deserialising, or on the H2D? iostat -x 1 during the load tells you whether the NVMe is saturated (disk bottleneck) or nearly idle (loader/CPU bottleneck). If the disk is at 20%, the problem is not the disk: it is the porter. Changing disk would fix nothing; changing loader would.

Knob 2 — `--load-format`: choosing the porter

vLLM exposes several loaders via --load-format: safetensors (default), runai_streamer, fastsafetensors, tensorizer. The default is the slowest. Changing one flag can be the cheapest 4-7× there is.

# Run:ai Model Streamer (concurrent reads, saturates the disk)
vllm serve meta-llama/Llama-3.1-70B --load-format runai_streamer
# fastsafetensors (GPUDirect Storage, direct DMA disk→HBM)
vllm serve meta-llama/Llama-3.1-70B --load-format fastsafetensors

Knob 3 — Streamer concurrency

The Run:ai Model Streamer splits the read across N threads according to the size of each tensor in order to saturate the storage bandwidth. Concurrency is the key parameter: 16 is usually enough for local NVMe; 32 (sometimes 64) for high-throughput network storage. One thread does not saturate a Gen5 NVMe; 32 do.

vllm serve <model> --load-format runai_streamer \
 --model-loader-extra-config '{"concurrency": 32}'

Knob 4 — GPUDirect Storage (GDS) + fastsafetensors

If there is an nvidia-fs driver, a supported filesystem and local NVMe, GDS writes straight disk→HBM with no bounce through the CPU. It is the difference between the two paths in the diagram. But: it requires the stack to be in place, and it only wins if the bottleneck is the CPU bounce, not the disk itself. Check with gdscheck.

Knob 5 — Local NVMe for the weights, not network

Serving the weights from Ceph RGW / NFS is convenient (one shared place) but it puts the network into the loading path. For cold start, weights on the node’s local NVMe (or a local cache). Network storage is for the model repository; the inference node should have a hot local copy.

Knob 6 — Pre-pull / local model cache

An initContainer that downloads the model onto a local or hostPath NVMe volume before starting vLLM turns a cold start “from the network” into one “from local NVMe”. Combined with a per-node cache DaemonSet, new pods on an already warm node read from local disk, not from the network.

Knob 7 — Quantisation: move fewer bytes

Weights already in FP8 or INT4 on disk = half or a quarter of the cold start. It is the knob from the table above. And it composes with all the others: FP8 + streamer + GDS is multiplicative.

Knob 8 — Parallel loading across GPUs (vLLM V1)

vLLM’s V1 engine (the default since 0.19) loads the weight shards in parallel across the GPUs of a TP group, instead of sequentially. At TP=4, each GPU loads its quarter at the same time. Check it is active; in old versions loading was serial and the cold start of TP=4 was almost 4× that of TP=1.

Knob 9 — NUMA locality of the NVMe

The NVMe hangs off a PCIe root under a specific socket, exactly the same NUMA map as the host post. If the load’s host buffer lands on the wrong socket, the H2D crosses the UPI. The fifth list to align, alongside isolcpus, reserved-cpus and the NIC IRQs: which socket is local to the NVMe and to the destination GPU. nvidia-smi topo -m shows it.

Knob 10 — Do not pay the cold start: keep pods warm

Sometimes the answer is not to load faster, but not to unload. A floor of always-live replicas (no scale-to-zero), or a pool of preloaded warm standbys, swaps “wait 40 s” for “0 s”. It is the cost of idle GPU in exchange for start-up latency: a capacity planning decision, not a technical one.

Summary table

#	Knob	What it attacks	Risk / cost
1	`iostat` while loading	knowing whether the bottleneck is disk or loader	none; always do it first
2	`--load-format`	single-threaded deserialisation	format compatibility
3	streamer concurrency	underused disk	host RAM for buffers
4	GPUDirect Storage	bounce through the CPU	requires nvidia-fs + supported FS
5	local NVMe vs network	the network in the path	duplicating weights per node
6	pre-pull / node cache	network on every start-up	local disk space
7	FP8/INT4 quantisation	bytes to move	quality (measure, do not assume)
8	V1 parallel loading	serial loading across GPUs	none if V1 is active
9	NUMA of the NVMe	H2D crossing the UPI	align with the rest of the NUMA lists
10	warm pods / no zero	the whole cold start	idle GPU paid for

How it connects with the rest of the stack

With autoscaling. The whole of scale-from-zero with KEDA rests on this: real elasticity is limited by the cold start, not by GPU availability. An autoscaler with 40 s of loading reacts late to every peak.

With disaggregated serving. In disaggregated prefill/decode, bringing up a decode pool on demand pays the cold start of loading the model into each new pod. The elasticity of the pattern depends on how fast those pods start.

With canary/blue-green. Every canary deployment loads a new version of the model alongside the old one. The validation time of a canary includes its cold start; large models make deployments slower and more expensive.

With NUMA and hugepages. The load’s host buffer wants to be pinned and NUMA-local, the same thing the host post asked for on the hot path. The loading path is another client of the same NUMA map.

With quantisation. FP8/INT4 is not only inference throughput: it is the direct lever on the bytes of the cold start.

With capacity planning. Sizing that ignores cold start underestimates how many replicas are needed to absorb a peak: if they take 40 s to start, you need a bigger permanent cushion.

Traps and things that are not what they seem

“The second time it loaded really fast.” This is the star trap. The first load filled the page cache (host RAM); the second reads from RAM, not from disk, and flies. But in production pods are ephemeral and are born on different nodes: the start-up that counts is the cold one, on a node where those files are not in page cache. Benchmarking the second load is measuring a situation that almost never happens at the moment that matters (the peak that triggers the autoscaler).

“GDS always speeds things up.” No. GDS removes the bounce through the CPU; if your bottleneck is the disk itself (saturated NVMe) or the deserialisation, GDS does not touch that part. Measure first (knob 1). It also demands nvidia-fs, a supported filesystem, and sometimes it does not work over whatever network storage you have.

“mmap makes loading instant.” mmap maps the file but reads nothing yet: the cost is deferred to the first access to each page. The time does not disappear, it moves, and the first token pays the page faults that start-up did not. You have moved the cold start into the latency of the first request, which is probably a worse place to have it.

Weights on network storage “because it is cleaner”. Sharing a model repository on Ceph RGW is fine for storing them; serving the cold start from there puts the network (and its latency and its contention) into the critical path. Local NVMe cache on the inference node.

Loading FP16 and quantising at start-up. Quantising on the fly during loading (for example FP16→FP8 on the GPU) can be slower than having the weights already quantised on disk: you move twice the bytes and on top of that you do conversion work. If you are going to serve in FP8, store the weights in FP8.

Optimising the load and ignoring T_cuda-graphs. Bringing weight loading down to 8 s and forgetting that CUDA graph capture adds several more seconds leaves the cold start half done. That second half of start-up is the subject of the next post.

Conclusion

The whole series optimised what happens with every token: the cable, the host, the network, the silicon. But before the first token there is a journey almost nobody times and that decides whether an inference pod opens in 10 s or in 60 s: pushing the model from disk to HBM. Intuition blames the GPU, and the GPU is innocent: HBM swallows 140 GB in 42 ms. The bottleneck is the supply chain: a disk reading at 14 GB/s, a PCIe copying at 50, and above all a default loader that deserialises tensor by tensor with a single thread and turns 10 s of bytes into 60 s of waiting. The solutions attack the two right fronts, fewer bytes (quantisation) and better transport (GDS, concurrent streamers, local NVMe), and give 4-7× almost for free. And above the technique, an idea that reorders the priorities: on an elastic platform, cold start is not a start-up detail, it is the ceiling on elasticity. The fastest GPU in the world does not scale if it takes 40 s to have the model inside. The pantry lift, the one nobody timed, is what decides what time the kitchen really opens.

References

NVIDIA, Reducing Cold Start Latency for LLM Inference with NVIDIA Run:ai Model Streamer: https://developer.nvidia.com/blog/reducing-cold-start-latency-for-llm-inference-with-nvidia-runai-model-streamer/.
vLLM, Loading models with Run:ai Model Streamer: https://docs.vllm.ai/en/stable/models/extensions/runai_model_streamer/.
vLLM, Loading model weights with fastsafetensors: https://docs.vllm.ai/en/stable/models/extensions/fastsafetensor/.
foundation-model-stack, fastsafetensors (high-performance loader, GDS): https://github.com/foundation-model-stack/fastsafetensors.
Speeding up Model Loading with fastsafetensors (arXiv 2505.23072): https://arxiv.org/html/2505.23072v1.
NVIDIA, GPUDirect Storage: A Direct Path Between Storage and GPU Memory: https://developer.nvidia.com/blog/gpudirect-storage/.
NVIDIA, Magnum IO GPUDirect Storage (benchmarking and configuration): https://developer.nvidia.com/gpudirect-storage.

The kitchen door the maître d' never looked at: network NUMA, Cilium eBPF and DRANET, the fourth leg of pinning

Sat, 06 Jun 2026 12:00:00 +0200

Fourth instalment, a coda, of “under the engine”. The series closed with three legs of locality: the cable between GPUs, the host by hand and the declarative orchestration of the kubelet. But the maître d’ of the last post seated the group looking at CPU, memory and GPU, and never asked which door the dishes come through. That door is the NIC. Here is the fourth leg.

TL;DR

The Topology Manager admits a pod under single-numa-node if its CPUs, its memory and its GPU fit on the same NUMA node. The NIC is not part of that count: the kubelet has no Hint Provider for the network card. On an inference node with the network at 200/400 Gb/s, the case of disaggregated serving, where the KV-cache travels over RDMA between the prefill pool and the decode pool, a NIC on the wrong socket makes every packet cross the UPI/QPI, exactly the “remote NUMA” the series fights on the compute side, but coming in through the network door. And there is a second front: the softirq (NET_RX) that processes the datapath runs on the CPU that services the NIC’s IRQ; if that CPU is one of the cores isolcpus/reserved-cpus gave exclusively to vLLM, the softirq steals cycles from it and injects jitter into the p99 tail. Cilium eBPF replaces two RKE2 pieces, kube-proxy (with eBPF/XDP load balancing) and the default CNI Canal (with a native datapath), and its own tuning guide tells you to kill irqbalance and pin the NIC IRQs: a fourth list to align alongside isolcpus and reserved-cpus. The 2026 state of the art closes the gap from above: netkit (kernel ≥6.8, zero namespace overhead), BIG TCP (192k super-packets for 100Gb/s+), host-routing (iptables bypass), and above all DRA/DRANET, the network driver that finally co-schedules NUMA-local GPU and NIC on the same PCIe root, enabling GPUDirect RDMA with +59.6% bus bandwidth in all_gather and +58.1% in all_reduce. On a generic RKE2 cluster with 4×H100 SXM nodes.

Where you are: the network plane the trilogy did not open

The analogy: the door the dishes come through

Go back to the restaurant of the previous post. The maître d’, the Topology Manager, seated the party of eight at a single table (one NUMA node) because the diners (CPUs), the cutlery (memory) and the reserved bottle (the GPU) all fitted. A perfect table. But the maître d’ never looked at where the kitchen pass is: the door every dish comes in and goes out through.

That door is the NIC. The prompt comes in there, the tokens go out there, and, in disaggregated serving, the KV-cache that the prefill pool sends to the decode pool circulates there. If the table is in the left-hand room (socket 0) but the kitchen pass is in the right-hand one (socket 1), every dish crosses the entire restaurant (the UPI/QPI), over and over, however impeccably the table is laid. The diner does not notice the perfect table: they notice the dish arriving late and cold.

And there is a finer detail: the waiter who crosses the room with the dishes (the softirq that processes the packets) is one of the seated diners. If the maître d’ gave them a chair of their own so they could eat in peace (a core isolated with isolcpus for vLLM) but the restaurant also has them waiting on the far door, that diner does not eat: they spend the whole dinner crossing the room. The jitter shows up exactly where you thought you had bought calm.

The trilogy levelled three legs of the table: the cable, the host and the orchestration. The fourth, which door the dishes come through and who carries them, is levelled by no kubelet manager. Until 2026.

The gap: why the Topology Manager does not look at the NIC

The mechanism of post 3 is a coordinator (Topology Manager) that consults three Hint Providers: CPU Manager, Memory Manager and Device Manager (the GPU plugin). Each says on which NUMA node it can satisfy its part; the coordinator computes the intersection and admits or rejects.

The problem is one of census: the classic NIC is not a Device Manager “device”. A standard Ethernet/InfiniBand card is managed by the CNI and the kernel, it is not requested in the pod’s resources: like nvidia.com/gpu, and therefore it emits no NUMA hint. The Topology Manager aligns CPU+memory+GPU and leaves the NIC wherever the hardware put it, which may be the other socket. The maître d’ has three assistants and is missing the fourth: the one who knows which door the dishes come through.

This did not matter when a node’s network was 10/25 Gb/s and the bottleneck was elsewhere. It matters now, with two workloads that saturate the node’s network:

Disaggregated serving. The KV-cache that travels between the prefill pool and the decode pool moves over RDMA. These are large transfers, sensitive to latency and bandwidth, which in multi-node setups go out through the NIC.
Multi-node NCCL collectives. When tensor/pipeline parallelism crosses the node boundary, the all-reduce/all-gather operations no longer go over NVLink but over GPUDirect RDMA on the NIC.

In both, where the NIC sits relative to the GPU and to the pod’s cores decides performance. And the kubelet, on its own, does not coordinate that.

The network datapath under NUMA: IRQ, softirq and DMA

To see why NIC locality carries weight, you have to look at the path of an incoming packet:

Three kernel facts the analogy compresses:

The IRQ has affinity. Each NIC queue raises an interrupt that the kernel services on a specific CPU (/proc/irq/<n>/smp_affinity). The heavy processing is deferred to a softirq (NET_RX/NET_TX), which runs on that same CPU. If irqbalance is left loose, it migrates them non-deterministically, poison for the p99.
The softirq competes with the pod. If the IRQ lands on a core that isolcpus reserved for vLLM, that queue’s NET_RX steals cycles from the model. The signal in /proc/softirqs: a NET_RX column that spikes on a single CPU. It is the same jitter from post 2, coming in through the network.
DMA has a NUMA origin. The NIC writes the packet by DMA into the RAM of the socket of its PCIe root. If the consumer (the pod’s thread) is on the other socket, it reads across the UPI/QPI. RFS (Receive Flow Steering) tries to bring the processing to the consumer’s CPU, but it cannot teleport the NIC to the other socket.

One number, with its caveat

Take a 2-socket node, a 400 Gb/s = 50 GB/s NIC on the PCIe root of socket 0, and a decode pod pinned to socket 1. If the NIC saturates, those ~50 GB/s of receive traffic cross the UPI towards socket 1. A UPI 2.0 link runs at around 20–40 GB/s per direction and link depending on generation; even with several links, 50 GB/s of network traffic going against the grain eats a far from negligible fraction of the inter-socket budget, the same budget already contended by the pod’s remote memory accesses and, if there is multi-node traffic, the KV-cache of the disaggregation. I am not giving a firm “X% degradation” because it depends on CPU generation, number of UPI links, MTU and traffic pattern; without that methodology, any exact figure is marketing.

What is measured with public methodology is the aggregate effect of aligning GPU and NIC: the DRANET project reports +59.6% bus bandwidth in all_gather and +58.1% in all_reduce (NCCL collectives) when the assigned NIC is NUMA-local to the GPU rather than not. That is the size of the gap the Topology Manager was leaving open.

What RKE2’s Cilium eBPF replaces (and why it touches this story)

RKE2 ships Canal (Flannel + Calico) by default as the CNI and kube-proxy (iptables/IPVS rules) for Service balancing. Switching to Cilium (cni: cilium in /etc/rancher/rke2/config.yaml) replaces both pieces with an eBPF datapath:

RKE2 piece	What it does	What Cilium eBPF puts in
`kube-proxy` (iptables/IPVS)	Service balancing	LB in eBPF; with `kubeProxyReplacement=true`, and acceleration in XDP (driver layer)
Canal (Flannel+Calico)	VXLAN overlay + NetworkPolicy	native datapath (`routingMode=native`), L3/L4 and L7 NetworkPolicy in eBPF
veth per pod	namespace interface pair	netkit (kernel ≥6.8): namespace overhead ~0
host iptables traversal	netfilter hooks	eBPF host-routing: bypasses iptables and the upper part of the stack

Up to here this is pure networking and it does not touch the kubelet’s resource managers: Cilium does not assign exclusive CPUs nor emit compute NUMA hints. The ten knobs of post 3 stay identical whether you run Canal or Cilium.

But Cilium does enter the fourth leg through two doors. The first: its own tuning guide recommends, literally, “kill irqbalance and pin the NIC IRQs to specific CPUs for maximum workload isolation”, along with the tuned network-latency profile, the performance governor and CONFIG_PREEMPT_NONE. In other words: the eBPF datapath only really performs if you coordinate IRQ affinity, and that affinity has to point at the housekeeping cores (reserved-cpus), never at the isolated ones. So a fourth list appears that must be kept consistent with isolcpus and reserved-cpus:

isolcpus = 2-31,34-63 # cores exclusive to vLLM (host, post 2)
reserved-cpus = 0-1,32-33 # kubelet housekeeping (post 3)
IRQ affinity = 0-1,32-33 # NIC IRQs → housekeeping ONLY (this post)
 # never 2-31: there the softirq would steal from the model

The second door: netkit + host-routing + BIG TCP reduce how many times the packet crosses the stack and the namespace, which cushions (does not eliminate) the cost of the NUMA crossing. BIG TCP assembles super-packets of up to 192k (against 64k) for 100Gb/s+; fewer stack traversals means less softirq work on the core, and therefore less pressure on the inter-socket budget. It is the continuous batching analogy applied to the network stack: amortising a fixed cost over larger batches.

Cilium performance profile (state 1.19, kernel ≥6.8)

# Helm, recommended performance profile (summary of the tuning guide)
helm install cilium cilium/cilium --version 1.19.4 \
 --namespace kube-system \
 --set routingMode=native \
 --set bpf.datapathMode=netkit \  # namespace overhead ~0 (kernel >=6.8)
 --set bpf.masquerade=true \
 --set kubeProxyReplacement=true \  # replaces RKE2's kube-proxy
 --set enableIPv4BIGTCP=true \  # 192k super-packets (mlx5/ice NIC)
 --set enableIPv6BIGTCP=true \
 --set bpf.distributedLRU.enabled=true \# per-CPU BPF maps: less spinlock contention
 --set bpf.mapDynamicSizeRatio=0.08 \
 --set bpfClockProbe=true

# Verification inside a Cilium pod:
cilium status --verbose | grep -E "Device Mode|Host Routing|BIG TCP|XDP"
# Device Mode: netkit · Host Routing: BPF · IPv4 BIG TCP: enabled · XDP Acceleration: Native

Sceptical caveat: netkit and BIG TCP are beta and require kernel ≥6.8 and specific NICs (mlx4/mlx5/ice). They are not in-place: they change datapath fundamentals and force you to restart pods or, better, to apply them by per-node config only on new nodes. For an ENS cluster in production, that is a maintenance window, not a blind helm upgrade.

The 2026 state of the art: DRA and DRANET, the maître d’ who finally looks at the door

What closes the gap at the root is not Cilium, it is the admission mechanism the kubelet did not have for the NIC: Dynamic Resource Allocation (DRA), beta since Kubernetes 1.32 and with advances in every release up to 1.36 (May 2026). DRA generalises the “devices” model beyond the GPU: a driver discovers the hardware, publishes ResourceSlices with its attributes, including the NUMA topology and the PCIe root, and the scheduler resolves ResourceClaims that can demand affinity between devices.

DRANET (a kubernetes-sigs project) is the network DRA driver. It discovers the NICs (including the RDMA-capable ones), advertises them as ResourceSlices, and via NRI injects them into the pod’s namespace, compatible with whatever CNI you already have, Cilium included. The key piece for this story: combined with the NVIDIA GPU DRA driver, it allows co-scheduling GPU and NIC that share a PCIe root (the relationship NVIDIA calls NODE), which is exactly the condition for GPUDirect RDMA. The maître d’ finally has a fourth assistant: “is there a NIC NUMA-local to this GPU?”.

The ResourceClaimTemplate uses CEL selectors to request exactly that alignment:

# Request an RDMA NIC NUMA-local to the assigned GPU (illustrative DRANET/DRA schema)
apiVersion: resource.k8s.io/v1beta1
kind: ResourceClaimTemplate
metadata:
 name: gpu-nic-numa-aligned
spec:
 spec:
 devices:
 requests:
 - name: rdma-nic
 deviceClassName: dra.net  # NICs published by DRANET
 constraints:
 - requests: ["rdma-nic"]
 matchAttribute: "dra.net/pcieRoot" # same PCIe root as the GPU
 # → enables GPUDirect RDMA over a NUMA-local path

Why it matters for inference, not for abstract “AI training”: in disaggregated serving, RDMA is what moves the KV-cache between the prefill pool and the decode pool with the latency that TTFT demands; and in multi-node, GPUDirect RDMA replaces NVLink as the medium of the collective. Aligning GPU+NIC on the same PCIe root is what turns an “RDMA that works” into an “RDMA that performs”, the +60% bus bandwidth from DRANET.

Status and caveats: DRA is beta (gates to enable by hand), DRANET is young (a SIG project, still evolving) and the managed offering exists mostly in cloud (GKE managed DRANET in preview, AKS for RDMA). For on-premise ENS it is a path, not a finished product: the value today is understanding that the fourth leg already has a standard OSS mechanism, and starting to pilot it on a lab node, not putting it into critical production this quarter.

How it connects with the rest of the stack

With the host (post 2). The NIC’s IRQ affinity is a third list to marry with isolcpus and reserved-cpus. The IRQs go to housekeeping; the isolated cores, untouched. Uncoordinating them lets in through the network door the jitter that isolcpus threw out through the compute one.

With the orchestration (post 3). DRA is the natural extension of the Topology Manager: the same “admit only if it fits on the NUMA node” principle carried over to the NIC. Where the Device Manager left the network out of the census, DRANET puts it in.

With the interconnect (post 1). Inside the node NVLink rules; on crossing the node boundary, GPUDirect RDMA over the NIC is the medium of the collective. The kubelet’s NUMA policy guarantees that GPU and CPUs share a socket; DRANET adds that the NIC does too, and only then does the RDMA take the short path.

With disaggregated serving. The prefill→decode KV-cache is the traffic that punishes a badly placed NIC hardest. The fourth leg is what stops separating prefill and decode from being paid for in transfer latency.

With capacity planning. Sizing gains a dimension: “GPUs per node and cores per NUMA node” is not enough; you have to count how many NICs NUMA-local to a GPU the chassis has. A node with 4 GPUs and a single NIC on socket 0 has two GPUs “far from the door”.

With observability. What confirms that the fourth leg is properly set is not an application dashboard: it is /proc/softirqs (is NET_RX concentrated on housekeeping?), nvidia-smi topo -m (is the GPU↔NIC relationship NODE/PHB?) and the NIC counters. It fits with GPU observability with DCGM: the GPU “at 60% for no reason” may be the host waiting for packets that cross the socket.

Traps and things that are not what they look like

Believing that switching to Cilium “already optimises the network”. Cilium eBPF replaces kube-proxy and Canal and performs better out of the box, but the default deployment prioritises compatibility, not performance. Without irqbalance disabled, without IRQs pinned to housekeeping and without netkit/host-routing, you leave most of the improvement on the table. Cilium’s docs say so; plenty of people do not read the tuning guide.

Pinning the NIC IRQs to isolated cores. The symmetric mistake to knob 6 of post 3: if you set IRQ affinity over isolcpus, the NET_RX softirq steals cycles from vLLM on exactly the cores you isolated so nobody would disturb it. The IRQs go to reserved-cpus, always.

Assuming the Topology Manager already aligns the NIC. It does not: the classic NIC is not a Hint Provider. If you need NIC↔GPU locality, today the mechanism is DRA/DRANET, not a kubelet policy. Waiting for single-numa-node to solve it is waiting for something that is not in its design.

Putting DRA/DRANET into ENS production this quarter. It is beta and young. The sensible move is to pilot it on a lab node, measure all_reduce/all_gather with and without alignment, and decide with data. The +60% figure is from a specific environment; reproduce it in yours before promising it.

BIG TCP / netkit without reading the requirements. Kernel ≥6.8, mlx4/mlx5/ice NICs, no tunnel or encryption for BIG TCP, and nothing in-place: it forces pod restarts or per-node config. On a cluster with IPsec or with unsupported NICs, part of this does not apply. Check cilium status --verbose before taking for granted that it is active.

Confusing the eBPF datapath (kernel) with the Cilium agent (pod). cilium-agent is a Burstable DaemonSet that should live on housekeeping (covered by system-reserved). But datapath processing runs in softirq, governed by the host’s IRQ affinity, not by reserved-cpus. They are two different things; pinning the pod well does not pin the softirq.

Conclusion

The “under the engine” series chased one idea: performance that looks like an engine problem (vLLM is slow) or a model problem (quantisation) is, far too often, a locality problem one layer down. The trilogy covered three: the cable (NVLink not used), the host (remote NUMA, jitter) and the orchestration (pinning that never happened). The fourth was missing: the network. The Topology Manager seats the pod at a perfect NUMA table and never asks which door the dishes come through nor who carries them. On a node at 25 Gb/s it made no difference; on one at 400 Gb/s with the KV-cache crossing over RDMA, that door decides TTFT and the collective’s bandwidth. Cilium eBPF replaces kube-proxy and Canal with a datapath that performs, if you coordinate IRQ affinity with isolcpus/reserved-cpus, a fourth list to align, and DRA/DRANET finally provides the census that was missing: co-scheduling NUMA-local GPU and NIC on the same PCIe root, with an improvement of a size (+60% NCCL bus bandwidth) that measures how big the gap was. Going down a level is not snobbery: it is that the root cause lived, once again, one layer below where the dashboard looks.

References

Cilium, Tuning Guide (netkit, host-routing, BIG TCP, XDP, pin the IRQs and kill irqbalance): https://docs.cilium.io/en/stable/operations/performance/tuning/.
Cilium 1.19 (February 2026), Cilium at Ten Years — hardening of encryption, policies and observability: https://www.infoq.com/news/2026/02/cilium-119/.
Isovalent, Cilium 1.18 (IPv6, encrypted overlay, ingress bandwidth, policy perf): https://isovalent.com/blog/post/cilium-1-18/.
RKE2, Network Options (Canal by default; Cilium with kube-proxy replacement): https://docs.rke2.io/networking/basic_network_options.
Kubernetes, Dynamic Resource Allocation: https://kubernetes.io/docs/concepts/scheduling-eviction/dynamic-resource-allocation/.
Kubernetes blog, v1.36: More Drivers, New Features, and the Next Era of DRA (May 2026): https://kubernetes.io/blog/2026/05/07/kubernetes-v1-36-dra-136-updates/.
DRANET (kubernetes-sigs), the network DRA driver and the paper The Kubernetes Network Driver Model (+59.6% all_gather / +58.1% all_reduce): https://github.com/kubernetes-sigs/dranet.
AKS Engineering, Optimizing RDMA performance for AI workloads on AKS with DRANET (April 2026): https://blog.aks.azure.com/2026/04/01/dranet-rdma-optimization-for-ai-on-aks.
Linux network tuning — IRQ affinity, RSS/RPS/RFS and NUMA softirq: https://andreaskaris.github.io/blog/networking/rss-irq-affinity-and-rps/.

Choosing the switchboard: which OSS gateway to put in front, and why the licence decides before the features

Sat, 06 Jun 2026 08:30:00 +0200

This post is the decision companion to The LLM inference router. That one explained what an inference router is and why it exists (catalogue, traffic splitting, cross-cutting policy, prefix-aware routing). This one answers the question that actually blocks a deployment: which one to choose, with verified licences and an order of criteria that is not the one most people use.

TL;DR

The gateway is the piece 100 % of the traffic passes through and the most expensive to replace later: the client SDK, the auth and rate-limit policies, the tracing and the model catalogue all couple to it. That is why the order of the criteria matters, and it is not the usual one. First the licence: not “is it open source?” but “are the features I need under a permissive licence or behind an Enterprise wall?”. Second the architectural fit: is it a citizen of your Kubernetes (Gateway API) or a separate process you have to operate on the side? Third, maturity and documentation. Fourth, the features. With data verified as of June 2026: LiteLLM is MIT in the core, but SSO, audit logs, fine-grained RBAC and several guardrails are Enterprise; Kong has an Apache 2.0 core but the AI plugins that matter (semantic cache, prompt guard, AI proxy advanced) are gated; Envoy AI Gateway, Gateway API Inference Extension (GIE), Higress, APISIX and Bifrost are Apache 2.0 end to end. For an RKE2 + vLLM stack with a K8s-native priority (the case we assume here), the recommendation is to adopt the Gateway API Inference Extension model, the Endpoint Picker that routes with awareness of prefix cache, KV and LoRA, which is exactly what multiplies the hit rate. Implement it with Envoy AI Gateway if the AI-native trajectory weighs more, or with Higress if today’s maturity weighs more; and put LiteLLM (MIT) behind it as an optional multi-provider control plane. With explicit scepticism about v0.x releases, about “OpenAI-compatible ≠ inference-aware” and about phone-home telemetry.

The principle that reorders the criteria: the gateway is a marriage, not a date

Some pieces of the stack can be swapped on a Sunday afternoon: the embedding model, the reranker, even the inference engine behind an OpenAI-compatible interface. The gateway cannot. It is the piece everything else couples to: every client SDK points at its URL, the security policies live in it, the tracing is born in it, it defines the model catalogue. Ripping it out two years later means touching every consumer at once. It is the infrastructure decision with the highest cost of reversal in the whole serving layer.

When a decision is expensive to reverse, the dominant criterion is not “what does it do today?” but “can I own it and operate it for years without surprises?”. And that puts the licence ahead of the features. A brilliant tool whose SSO, audit log or RBAC live behind an Enterprise contract is a tool that, the day your ENS deployment needs those controls, forces you to pay or to migrate, which is exactly the scenario the choice was supposed to avoid. That is why this post’s filter order is licence → fit → maturity/docs → features, and not the other way round.

The field of candidates (June 2026)

The real OSS pieces anyone would put in front of an on-premise vLLM fleet:

Candidate	What it is	Language
LiteLLM Proxy	OpenAI-compatible gateway, 100+ providers, virtual keys, spend	Python (FastAPI)
Envoy AI Gateway	AI layer on top of Envoy Gateway; integrates GIE/EPP, InferencePool	Go (Envoy)
Gateway API Inference Extension (GIE)	K8s-SIG extension: inference-aware Endpoint Picker	Go
Higress	API gateway + AI plugins, Envoy/Istio, CNCF	Go/C++
Apache APISIX	Mature API gateway with AI plugins	Lua/Nginx
Kong (+ AI plugins)	API gateway; Apache 2.0 core, Enterprise AI plugins	Lua/Nginx
Bifrost	High-performance AI-first gateway	Go
vLLM Production Stack / Semantic Router	vLLM-specific router, KV/prefix/intent-aware	Go/Python

Filter 1 — Licence: where the small print is

This is where half the options die, and where “open source” misleads if you do not look at the detail. What has been verified:

Candidate	Core licence	What is gated (paid)
LiteLLM	MIT	SSO (Okta/Azure AD), audit logs, JWT auth, fine-grained RBAC, several guardrails (llmguard, llamaguard, prompt-injection) → Enterprise (~250 USD/month and rising)
Envoy AI Gateway	Apache 2.0	— (full AI layer is OSS)
GIE (Inference Extension)	Apache 2.0	— (K8s-SIG project)
Higress	Apache 2.0	— (AI plugins included)
APISIX	Apache 2.0	— (more built-in AI than Kong OSS)
Kong	Apache 2.0 (core)	AI Semantic Cache, AI Prompt Guard, AI Proxy Advanced, AI RAG Injector → Enterprise
Bifrost	Apache 2.0	—

Two conclusions that settle the field:

Kong falls out for an OSS AI deployment. Its core is Apache 2.0, but precisely the AI plugins that would justify choosing it (semantic cache, prompt guard, proxy advanced) are Enterprise, with contracts that the sector’s own comparisons put above five figures a year. For AI on an OSS budget, APISIX offers more built-in at no cost. Kong remains excellent as a classic API gateway; as an OSS AI gateway, no.
LiteLLM really is MIT in the core, and that is genuine: you can fork it, modify it and use it commercially. But SSO, audit logs, fine-grained RBAC and several guardrails are Enterprise. Under the criterion we set (“a permissive core is enough, governance is solved with OIDC and external auditing in the stack”) LiteLLM stays in play; under an “all OSS or nothing” criterion it would be wounded. It pays to know exactly what falls on which side of the line before committing.

LiteLLM: what is gated and its OSS equivalent

The nuance that decides whether LiteLLM’s gating is a real problem or not: “paid built-in feature” is not the same as “capability impossible in OSS”. In almost every case the capability is achieved by changing the how, wiring OSS through LiteLLM’s open hook, or solving it in the layer next door (the gateway, the observability). The breakdown, verified against the docs:

Capability	In LiteLLM Enterprise	OSS equivalent
Guardrails	prebuilt integrations (`llmguard`, `llamaguard`, `lakera`, `aporia`, `hide_secrets`)	the framework + custom guardrail hook + Presidio are OSS (MIT core); you invoke LLM Guard, Llama Guard or Presidio from the hook yourself
Audit log	turnkey UI with retention policies	request/response logging + custom callbacks + OTel/Langfuse exporters (official OSS integration) → you build the trail and you own it
RBAC	fine-grained (`enforce_rbac`, org/team/user roles)	coarse (virtual keys per team/budget/model) is OSS; fine-grained is done at the edge in the gateway (`ext_authz` + OPA)
SSO	Admin UI SSO (Okta/Azure/Google/OIDC)	user/API SSO is solved by fronting with OSS OIDC (Keycloak + oauth2-proxy) or in the gateway itself; JWT auth + JWT→virtual-key mapping are already in the core

The conclusion that closes the decision: in a K8s-native design with a gateway in front, OIDC and authorisation live in the gateway (Envoy/Higress + OPA) and the audit trail in Langfuse/OTel, both Apache 2.0 and already in the stack, so LiteLLM placed behind barely needs its Enterprise tier: the governance pieces sit where they architecturally belong and happen to be OSS. The only real cost is the DIY integration and maintenance, and, for ENS, designing the guarantees of the audit trail (retention, immutability/WORM) is on you, it does not come out of the box.

The rest, Envoy AI Gateway, GIE, Higress, APISIX and Bifrost, pass the licence filter clean: Apache 2.0 end to end, with no critical feature behind a wall.

Filter 2 — Fit: Kubernetes citizen or tenant?

With a K8s-native priority (RKE2), the question is whether the gateway is modelled as cluster resources, versioned with GitOps, observed with the same tools and integrated with the scheduler, or whether it is a separate process you have to operate on the side. This is where the structural novelty of 2025-2026 appears: the Gateway API Inference Extension (GIE).

GIE is an extension of the standard Kubernetes Gateway API, from SIG-Network, which adds two pieces: the InferencePool (a pool of replicas of a model as a native resource) and the Endpoint Picker (EPP), a scheduler that decides which replica each request goes to according to the inference state: queue length, available LoRA adapters, and, the key piece, the prefix cache state of each replica. It is exactly the prefix-aware routing that the router post explained as the reason hit rate goes from 5-15 % to 60-85 %, now as a community standard instead of a proprietary feature of each product.

Why this settles the decision when the priority is K8s-native: GIE turns inference-aware routing into a standard interface that several implementations satisfy (Envoy AI Gateway, Higress, kgateway, Istio, GKE Inference Gateway). You pick an implementation today and, if you change it tomorrow, the resource model (InferencePool, HTTPRoute) is preserved. That is precisely the property you wanted from a decision that is expensive to reverse: the coupling is to a standard, not to a product. A standalone proxy like LiteLLM, however good it is, lives outside this model. It is one more Deployment, with its own config, its own catalogue format and its own management plane.

Filter 3 — Maturity and documentation: the uncomfortable honesty

Here comes the tension no marketing comparison admits: what is K8s-native-correct and what is battle-tested-today do not quite coincide in June 2026.

Envoy AI Gateway: the AI layer is at v0.5, pre-1.0 and young. But it runs on Envoy and Envoy Gateway, which are among the most mature things that exist in proxies. It has the most complete GIE/EPP integration, model virtualisation and OpenInference tracing. Risk: the AI layer still moves fast (breaking changes are possible). Docs are growing, and good.
GIE: the project is reaching GA (one of the first standardised inference-aware extensions). Standard maturity is high and rising; the maturity of each implementation varies.
Higress: the most mature today among the K8s-native Apache-2.0 options with AI. Production at Alibaba scale, CNCF, Envoy/Istio base, AI plugins included, Gateway API support. If “mature” weighs more than “cutting-edge AI-native”, it is the safe bet.
APISIX: a very mature classic gateway, with growing AI plugins; less specialised in inference-aware routing (prefix/KV) than the GIE model. Battle-tested at massive scale.
LiteLLM: the one with the best documentation and the widest adoption in the field, by a distance. Mature as a proxy. Its ceiling is the language (Python/FastAPI: ~250-300 RPS per instance, scaling out by replicas) and the governance gating already covered.
Bifrost: Apache 2.0, Go, the highest performer (~11 µs overhead at 5,000 RPS), with semantic caching and governance built in. Younger and less proven over years, but technically strong.

Decision matrix

Weighting the four filters for the case of RKE2 + vLLM, K8s-native priority, permissive core acceptable:

Candidate	Licence	K8s-native	Maturity	Docs	Inference-aware	Verdict
Envoy AI GW + GIE/EPP	✅ Apache 2.0	✅✅ standard	⚠️ v0.5 (mature Envoy)	✅ good	✅✅ prefix/KV/LoRA	Primary (trajectory)
Higress (+ GIE)	✅ Apache 2.0	✅✅	✅✅ production	✅ good	✅ via GIE/plugins	Primary (maturity today)
APISIX	✅ Apache 2.0	✅	✅✅	✅✅	⚠️ less specialised	Solid alternative
LiteLLM	⚠️ MIT core, gov. gated	❌ separate proxy	✅✅	✅✅✅	⚠️ basic	Control plane behind
Bifrost	✅ Apache 2.0	⚠️	⚠️ young	✅	✅	One to watch (perf)
Kong	⚠️ AI plugins Enterprise	✅	✅✅	✅✅	❌ gated	Ruled out (OSS AI)

The recommendation

For the case we fixed, RKE2 + vLLM, K8s-native priority, permissive licence with a sufficient core, the recommendation has two layers, not one:

Data layer (what sits in front of the replicas): adopt the Gateway API Inference Extension model. It is the decision that ages well because it couples you to an Apache 2.0 standard, not to a product, and because it brings the prefix/KV/LoRA-aware routing that really moves the hit rate. For the implementation:

If the AI-native trajectory weighs more and you accept operating a v0.x layer on top of a mature Envoy: Envoy AI Gateway + GIE/EPP. That is where the cutting edge and the most complete integration are.
If today’s maturity outweighs everything else (ENS, critical production, zero appetite for v0.x in the path of 100 % of the traffic): Higress with GIE. CNCF, proven at scale, Apache 2.0, and migratable to the standard EPP.

Control layer (optional, behind): LiteLLM (MIT) if you need a single multi-provider point with virtual keys, spend tracking and fallbacks to external models. It goes behind the data gateway, not in its place, and you accept that SSO, audit and RBAC are solved with the cluster’s own OIDC and auditing (as we set out in the criterion). If your front end is only on-premise vLLM with no external providers, this layer is probably unnecessary.

What I would not recommend here: Kong (its AI features are gated, which contradicts the licence criterion); and starting with LiteLLM as the main data gateway in a stack that will be multi-replica and K8s-native, since its Python ceiling and its position outside the Gateway API model turn it into debt the day you scale.

Applied to our infrastructure: RKE2 + vLLM + 4×H100

The concrete deployment on the generic reference cluster:

# The InferencePool groups the replicas of a model (GIE resource)
apiVersion: inference.networking.x-k8s.io/v1alpha2
kind: InferencePool
metadata: { name: llama-70b-pool }
spec:
 selector: { app: vllm-llama70b }
 extensionRef: { name: llama-70b-epp }  # the Endpoint Picker
---
# The standard Gateway API route points at the pool
apiVersion: gateway.networking.k8s.io/v1
kind: HTTPRoute
metadata: { name: llama-70b }
spec:
 parentRefs: [ { name: ai-gateway } ]
 rules:
 - backendRefs:
 - group: inference.networking.x-k8s.io
 kind: InferencePool
 name: llama-70b-pool

Three notes on how this fits the blog series:

It lives in layer 1 of the seven-layer stack, in front of the replicas pinned by the RKE2 resource managers. The gateway routes; the kubelet pins; both are resources of the same GitOps cluster.
The EPP reads the prefix cache state of each replica, which connects directly with the engineering of prefix cache hit rate: the gateway is what turns that affinity into actual routing.
The gateway’s gen_ai.* tracing feeds the OTel observability pipeline and lands in Langfuse, closing the loop of the Observe layer.

Traps and things that are not what they seem

“It is open source” without reading what falls under Enterprise. The mistake this post tries to avoid: Kong and LiteLLM are OSS, but the features that justify choosing them for AI (AI plugins in Kong; SSO/audit/RBAC/guardrails in LiteLLM) are partly or wholly gated. “Open source” is the wrong question; “is what I need permissive?” is the right one.

“OpenAI-compatible” ≠ “inference-aware”. Almost all of them expose the OpenAI API. That does not mean they understand the KV cache state, the queue of each replica or the LoRA adapters. API compatibility is table stakes; intelligent routing (EPP) is a different league. Do not confuse “speaks OpenAI” with “routes well”.

Choosing by the RPS benchmark. Bifrost wins on raw overhead, but the bottleneck of an on-premise vLLM fleet is rarely the proxy: it is the GPUs. Optimising the gateway for 5,000 RPS when your replicas serve 300 req/s is solving the problem you do not have. The latency the gateway adds matters; its peak throughput almost never does, in this context.

Phone-home telemetry in a sovereign deployment. Several gateways send telemetry home by default. In an ENS or sovereignty context, that is an audit finding, not a detail. Verify and disable the phone-home before putting the gateway in the path of 100 % of the traffic; document it for the ENS/42001/EU AI Act controls file.

Betting on a v0.x in the critical path with no rollback plan. Envoy AI Gateway v0.5 is promising, but it sits in the path of 100 % of the traffic. If you choose it, have the rollback rehearsed and follow the changelog: v0.x releases break things. The GIE standard mitigates this (you can change implementation), but the specific implementation has to be operated with a safety net.

Confusing the data gateway with the control plane. LiteLLM in front of everything seems to simplify, but it mixes two roles: routing data traffic (where you want K8s-native + inference-aware) and managing multi-provider keys and spend (where LiteLLM shines). Separate them: a GIE-native data gateway, with LiteLLM as a control plane behind it if needed.

Conclusion

The gateway is the piece with the highest cost of reversal in the serving layer, and that forces you to invert the intuitive order of criteria. First the licence, not “is it OSS?” but “is what I need permissive or is it gated?”, which already rules Kong out for OSS AI and puts an asterisk on LiteLLM for its Enterprise governance. Then the fit, where the Gateway API Inference Extension turns inference-aware routing into an Apache 2.0 standard you can couple to without marrying a product. And only after that maturity, docs and features, where honesty forces you to admit that the cutting-edge AI-native option (Envoy AI Gateway v0.5) and the battle-tested-today one (Higress) are still not the same thing. For an RKE2 + vLLM stack with a K8s-native priority, the answer is not a product but a model, GIE/EPP, implemented with Envoy AI Gateway if you look at the trajectory or with Higress if you look at maturity, and LiteLLM as an optional control plane behind. The decision that ages well is the one that couples you to a standard, not to a vendor; in gateways, in June 2026, that standard finally has a name.

References

Kubernetes, Introducing Gateway API Inference Extension (blog oficial, jun 2025): https://kubernetes.io/blog/2025/06/05/introducing-gateway-api-inference-extension/.
Gateway API Inference Extension, doc del proyecto (Endpoint Picker, InferencePool): https://gateway-api-inference-extension.sigs.k8s.io/.
Envoy AI Gateway, sitio y release notes (v0.x, GIE integration): https://aigateway.envoyproxy.io/.
LiteLLM, Enterprise Features (qué está gated): https://docs.litellm.ai/docs/proxy/enterprise · licencia MIT: https://github.com/BerriAI/litellm/blob/main/LICENSE.
Kong, Announcing Kong’s Open Source AI Gateway y matriz de plugins Enterprise: https://konghq.com/blog/product-releases/announcing-kong-ai-gateway.
Higress (Apache 2.0, CNCF, AI gateway): https://higress.cn/en/.
Apache APISIX, APISIX vs Kong (cobertura de IA OSS): https://apisix.apache.org/learning-center/apisix-vs-kong/.
vLLM, Semantic Router v0.1 Iris y Production Stack (routing específico de vLLM): https://blog.vllm.ai/2026/01/05/vllm-sr-iris.html.

The maître d' who only seats you if you all fit at one table: CPU, Memory and Topology Manager on RKE2

Sat, 06 Jun 2026 08:00:00 +0200

The close of the “under the engine” series. We saw the cable between GPUs and the host: NUMA, hugepages and CPU isolation done by hand. Here is the piece that makes it declarative and at scale: how the RKE2 kubelet pins each vLLM pod to the right NUMA node without a single script.

TL;DR

Pinning NUMA with numactl/isolcpus/taskset, the subject of the previous post, does not scale to a cluster where pods come and go and there are dozens of nodes. The kubelet automates it with three components that work as Hint Providers for a central coordinator, the Topology Manager: the CPU Manager (assigns exclusive CPUs to containers of Guaranteed pods with whole CPUs), the Memory Manager (NUMA-local memory and hugepages) and the Device Manager/GPU plugin (which knows which GPU is on which NUMA node). With the single-numa-node policy, the Topology Manager only admits the pod if its CPUs, its memory and its GPU fit in the same NUMA domain; if they do not fit, it rejects the pod, strict admission, like the maître d’ who will not seat a party of eight unless there is a table for eight. On RKE2 all of this is configured with kubelet-arg in /etc/rancher/rke2/config.yaml. This post explains the mechanism, gives the 10 knobs and takes apart the gotchas that break pinning silently: the cpu_manager_state file you have to delete when changing policy, the QoS that has to be exactly Guaranteed, and the reserved-cpus that must match the host’s isolcpus. On a generic RKE2 cluster with 4×H100 SXM nodes.

Where you are: the orchestration that materialises the host

The analogy: the maître d’ of a restaurant whose tables cannot be pushed together

A restaurant has tables of different sizes and, house rule, tables are never pushed together. A party of eight arrives. The maître d’ looks for a single table where all eight fit. If there is one, he seats them; if only tables for four are left, he does not accept them. He would rather turn the booking away than seat the group split across two separate tables, because he knows a split dinner goes badly.

That maître d’ is the Topology Manager under the single-numa-node policy. The “party” is an inference pod asking for CPUs, memory and a GPU. The “table” is a NUMA node. The maître d’ asks three assistants (are there free CPUs on any node? the CPU Manager; is there free memory? the Memory Manager; is there a free GPU? the Device Manager) and only admits the pod if all three resources fit on the same node. If not, he rejects it (the pod ends up Failed with TopologyAffinityError), and the scheduler will try another node.

The difference from the previous post: there you were the maître d’, seating each process by hand with numactl. Here the maître d’ is the kubelet, and it does it for every pod, on every node, automatically, rejecting whatever does not fit. That is what turns artisanal pinning into a declarative property of the cluster.

The mechanism: Hint Providers and the coordinator

The Topology Manager does not assign resources; it coordinates the ones that do. The flow, when a Guaranteed pod arrives at a node:

The three managers are Hint Providers: each one tells the Topology Manager on which NUMA node(s) it could satisfy its part. The Topology Manager computes the intersection and, depending on the policy, decides:

none (default): no coordination; each manager does its own thing without aligning. No NUMA guarantee.
best-effort: it tries to align on one node; if it cannot, it admits anyway (on whichever node). Better than nothing, no guarantee.
restricted: if it fails to align, it rejects the pod. Strict, but it allows multi-node affinity if the intersection provides it.
single-numa-node: it requires everything to fit on a single NUMA node, or it rejects. The strictest one, and the one that genuinely guarantees the locality of the previous post.

And two preconditions without which none of this kicks in:

The pod has to be QoS Guaranteed: requests == limits on CPU and memory, and whole CPUs (not 500m). Only then does the CPU Manager assign exclusive CPUs.
The CPU Manager has to be on the static policy (not none).

Without those two, the Topology Manager has nothing to align and pinning does not happen, even with the policy in place. That is gotcha no. 1.

How it is configured on RKE2

RKE2 passes arguments to the kubelet with the kubelet-arg key in /etc/rancher/rke2/config.yaml. The reference configuration for GPU inference nodes:

# /etc/rancher/rke2/config.yaml (on every agent node with GPUs)
kubelet-arg:
 - "cpu-manager-policy=static"
 - "topology-manager-policy=single-numa-node"
 - "topology-manager-scope=pod"
 - "memory-manager-policy=Static"
 - "reserved-cpus=0-1,64-65" # housekeeping; must match the host
 - "system-reserved=memory=8Gi"
 - "kube-reserved=memory=4Gi"
 - "reserved-memory=0:memory=4Gi;1:memory=4Gi" # required by Memory Manager Static
node-label:
 - "fibercli.local/numa-pinned=true"

After deploying it: systemctl restart rke2-agent. Critical gotcha: if the node has already run with cpu-manager-policy=none, there is a state file /var/lib/kubelet/cpu_manager_state that pins the old policy; changing the arg without deleting that file makes the kubelet fail to start or ignore the new policy. You have to stop the agent, rm /var/lib/kubelet/cpu_manager_state, and start again. (The same applies to memory_manager_state.)

And the vLLM pod, to be eligible, Guaranteed with whole CPUs:

resources:
 requests:
 cpu: "16" # whole, not a weird fractional "16000m"
 memory: "200Gi"
 nvidia.com/gpu: "2" # TP=2 → 2 GPUs on the same NUMA node
 hugepages-1Gi: "16Gi"
 limits:
 cpu: "16" # == requests → Guaranteed QoS
 memory: "200Gi"
 nvidia.com/gpu: "2"
 hugepages-1Gi: "16Gi"

With this, on a node with the config above, the kubelet assigns 16 exclusive CPUs from the NUMA node where the 2 requested GPUs live, its local memory and the hugepages, or it rejects the pod if they do not fit together. The artisanal pinning of the previous post, now declarative.

The 10 knobs to reach for

Ordered by dependency (the first ones are a precondition for the later ones). The canonical reference is the Kubernetes Topology Manager documentation and the RKE2 configuration.

Knob 1 — `cpu-manager-policy=static`: the foundation

Without this there are no exclusive CPUs and nothing else kicks in.

kubelet-arg: [ "cpu-manager-policy=static" ]

Gotcha: changing it requires deleting /var/lib/kubelet/cpu_manager_state and restarting the kubelet, or startup fails. It is cause no. 1 of “I set the policy and it does not pin”.

Knob 2 — `Guaranteed` QoS + whole CPUs: the pod’s precondition

This is not node config, it is pod config, but without it knob 1 does nothing for that pod. requests == limits on CPU and memory, and whole CPUs. A cpu: 500m or a requests != limits downgrades the pod to Burstable and it loses the pinning. Plenty of people set the node policy and forget the pod’s QoS.

Knob 3 — `topology-manager-policy=single-numa-node`: strict admission

kubelet-arg: [ "topology-manager-policy=single-numa-node" ]

The strict maître d’. For GPU inference it is the right policy: it guarantees that CPU+memory+GPU share a node. best-effort guarantees nothing (it admits misaligned); restricted allows multi-node affinity. Start with single-numa-node and drop to restricted only if you have admission problems.

Knob 4 — `topology-manager-scope=pod`: group the whole pod

kubelet-arg: [ "topology-manager-scope=pod" ]

With container scope (the default), each container is aligned separately; with pod scope, the whole pod goes to the same node. For a vLLM pod with sidecars (metrics, proxy), pod scope stops the sidecar from dragging the main container onto another node. Recommended for inference.

Knob 5 — `memory-manager-policy=Static` + `reserved-memory`: NUMA-local memory

kubelet-arg:
 - "memory-manager-policy=Static"
 - "reserved-memory=0:memory=4Gi;1:memory=4Gi"

Memory Manager Static forces cpuset.mems so that the pod’s memory comes out of the right node (and so do the hugepages). It requires declaring reserved-memory per node, or the kubelet does not start. It is the declarative equivalent of the previous post’s --membind.

Knob 6 — `reserved-cpus`: the housekeeping cores (must match `isolcpus`)

kubelet-arg: [ "reserved-cpus=0-1,64-65" ]

It reserves cores for the system and the daemons; the rest are left for exclusive pods. Key point of the series: these reserved-cpus must be the same cores you left out of isolcpus on the host (previous post). If the host isolates 2-31 but RKE2 reserves 0-3, there is a mismatch: isolated cores that the kubelet assigns to pods without them actually being quiet. Coordinate the two layers.

Knob 7 — GPU plugin with NUMA topology (NVIDIA GPU Operator)

The Device Manager can only give a correct NUMA hint if the GPU plugin exposes which node each GPU is on. The NVIDIA device plugin / GPU Operator does this, but you have to verify that the topology information arrives (in some versions it requires flags). Without a GPU hint, the Topology Manager aligns CPU and memory but not the GPU, and GPU locality is precisely the one that matters most.

Knob 8 — hugepages as a pod resource

resources:
 limits:
 hugepages-1Gi: "16Gi" # the node must have them pre-reserved (post 2, knob 4)

The hugepages you reserved at host boot (previous post) are requested as a resource. The Memory Manager assigns them NUMA-local. If you request them without having reserved them on the node, the pod does not get scheduled.

Knob 9 — `system-reserved` / `kube-reserved`: do not oversubscribe

kubelet-arg:
 - "system-reserved=cpu=500m,memory=8Gi"
 - "kube-reserved=cpu=500m,memory=4Gi"

It reserves resources for the system and the K8s components so the node does not run out of air under load. Badly calibrated, either the node chokes (too little reserved) or you waste capacity (too much). It must be consistent with reserved-cpus.

Knob 10 — Labels + taints: so vLLM lands here and nothing else does

# GPU node: taint to repel anything that does not need a GPU
node-taint: [ "nvidia.com/gpu=present:NoSchedule" ]
node-label: [ "fibercli.local/numa-pinned=true" ]

Keep the NUMA-pinned nodes for inference and push out whatever does not need it (databases, the Langfuse backend, runners). A ClickHouse stealing memory bandwidth from a carefully pinned vLLM pod throws away all the work of the previous nine knobs. Workload isolation is the closing move.

Summary table

#	Knob	Where	Function
1	`cpu-manager-policy=static`	kubelet-arg	exclusive CPUs (foundation)
2	`Guaranteed` QoS + whole CPUs	pod spec	pinning precondition
3	`topology-manager-policy=single-numa-node`	kubelet-arg	strict NUMA admission
4	`topology-manager-scope=pod`	kubelet-arg	group the whole pod
5	`memory-manager-policy=Static`	kubelet-arg	NUMA-local memory/hugepages
6	`reserved-cpus`	kubelet-arg	housekeeping (match isolcpus)
7	GPU plugin with topology	GPU Operator	NUMA hint for the GPU
8	`hugepages-1Gi`	pod spec	hugepages as a resource
9	`system/kube-reserved`	kubelet-arg	do not oversubscribe
10	taints + labels	node config	isolate GPU workloads

Verifying that the pinning really happened

Do not trust that the config “is in place”. Check:

# Is the active policy the one you set?
cat /var/lib/kubelet/cpu_manager_state | jq .policyName # "static"

# Which exclusive CPUs does the container have?
kubectl exec <pod> -- cat /sys/fs/cgroup/cpuset.cpus.effective

# Inside the pod: is the assigned GPU local to those cores?
kubectl exec <pod> -- nvidia-smi topo -m

# Were there topology rejections?
kubectl describe pod <pod> | grep -i TopologyAffinityError

A pod in Failed with TopologyAffinityError is not a bug: it is the maître d’ doing his job. That node did not have a table where CPU+memory+GPU fitted together. The answer is to review the sizing of the pod or the node, not to relax the policy casually.

How it connects with the rest of the stack

With the host (previous post). This post is the declarative automation of that one. cpu-manager-policy=static materialises the taskset; memory-manager-policy=Static materialises the --membind; reserved-cpus must match isolcpus. The two layers are one single decision seen from two places: the host executes it, the kubelet declares it. Uncoordinating them (isolcpus 2-31 vs reserved-cpus 0-3) breaks both.

With the interconnect (post 1). The Topology Manager pins the right GPU to the pod, but if you ask for 2 GPUs for TP=2, you will want those two to share NVLink. The NUMA policy guarantees they are on the same socket; that they are NVLink-connected is guaranteed by the baseboard hardware (post 1, knob 1). The two things together are what makes TP=2 perform.

With autoscaling. When KEDA scales vLLM pods, each new replica goes through Topology Manager admission. If the node has no free “table”, the pod stays pending. Pod autoscaling and node autoscaling (cluster-autoscaler) have to account for NUMA granularity, not just aggregate CPU/memory.

With capacity planning. Sizing changes: it is not “128 vCPU per machine”, it is “128 minus the reserved-cpus, in blocks that fit per NUMA node”. A machine with 2 sockets × 64 cores does not serve a pod asking for 80 cores under single-numa-node: they do not fit at one table. Planning has to reason per NUMA node, not per machine.

With service coexistence. The knob 10 taint is what keeps Langfuse, databases and runners off the inference nodes. Without that boundary, all the fine-grained pinning gets eaten by a noisy neighbour. Observability goes on its own nodes; inference, pinned, on its own.

Traps and things that are not what they seem

Policy in place, QoS forgotten. The most common mistake: cpu-manager-policy=static on the node but the pod is Burstable (requests != limits or fractional CPU). Pinning does not happen and nobody warns you. Guaranteed QoS with whole CPUs is a necessary condition.

Fossilised cpu_manager_state. Changing policy without deleting /var/lib/kubelet/cpu_manager_state (and memory_manager_state) makes the kubelet fail or ignore the change. Stop agent → delete file → start.

reserved-cpus ≠ isolcpus. If the host isolates some cores and RKE2 reserves others, the managers assign cores to pods that are not really quiet, or they leave isolated cores idle. The two lists have to be consistent. It is the coordination failure between the previous post and this one.

GPU plugin without NUMA topology. If the device plugin does not expose each GPU’s NUMA node, the Topology Manager aligns CPU and memory but leaves the GPU to chance, and GPU locality is the one that weighs most. Verify that the GPU Operator publishes the topology.

single-numa-node that rejects too much. If the pods ask for more resources than fit on one node (for example, more cores than a socket has), rejection is constant. The answer is not to drop to best-effort (which silences the problem by serving misaligned), but to size the pod so it fits at one table, or to accept restricted with full knowledge of what you are doing.

Believing best-effort “is nearly the same”. best-effort admits the pod even if it fails to align: it gives you a false sense of NUMA-awareness while you serve from the wrong socket. For inference with a tail SLO, single-numa-node or restricted; best-effort only if the alternative is scheduling nothing at all.

Conclusion

The previous post pinned by hand; this one does it at scale and with a guarantee that artisanal numactl did not give: strict admission. The kubelet, via CPU Manager, Memory Manager and Topology Manager, acts as a maître d’ who only seats the pod if its CPUs, its memory and its GPU fit at the same NUMA table, and who rejects what does not fit instead of serving a split dinner. Of the ten knobs, the first two, cpu-manager-policy=static and Guaranteed QoS with whole CPUs, are the precondition without which the other eight do nothing, and they are exactly the ones most often forgotten; the rest tune the policy, the memory, the hugepages and the coexistence. The thread that closes the series: the inference performance that looked like an engine problem (vLLM is slow) or a model problem (quantisation) is, far too often, a cable problem (NVLink not used), a host problem (remote NUMA, jitter) or an orchestration problem (pinning that never happened because the QoS was wrong). Going down a level is not infrastructure snobbery: it is where the root causes live that no application-layer dashboard is ever going to point out for you.

References

Kubernetes, Control Topology Management Policies on a node: https://kubernetes.io/docs/tasks/administer-cluster/topology-manager/.
Kubernetes, Control CPU Management Policies on the Node: https://kubernetes.io/docs/tasks/administer-cluster/cpu-management-policies/.
Kubernetes, Control Memory Management Policies on a Node: https://kubernetes.io/docs/tasks/administer-cluster/memory-manager/.
RKE2, Configuration Options (kubelet-arg in config.yaml): https://docs.rke2.io/install/configuration.
RKE2, Advanced Options and Configuration: https://docs.rke2.io/advanced.
rancher/rke2, discussion #3034 CPU Management Policies for RKE2 (the cpu_manager_state gotcha): https://github.com/rancher/rke2/discussions/3034.

The floor next door: NUMA, hugepages and CPU isolation, or why your GPU waits on the kernel

Sat, 06 Jun 2026 07:30:00 +0200

Second post in the “below the engine” series. The first opened up the cable between GPUs (NVLink/NCCL). This one goes down to the host: the cores, the memory and the kernel that surround those GPUs and that, badly configured, leave them waiting. The third will explain how Kubernetes automates all of this; here is the raw layer, the one you have to understand before delegating it.

TL;DR

A node with 4×H100 SXM is, physically, a two-socket server = two NUMA domains. Each socket has its own cores, its own memory channels and PCIe lanes towards half of the GPUs and NICs. Inference is not only GPU: the host does work on the hot path of every token, launching CUDA kernels, sampling the next token, tokenising, moving pinned buffers between host and GPU, running the NCCL threads. If those threads and their memory land on the socket that is not local to the GPU, every access crosses the inter-socket link (UPI/Infinity Fabric): 2-3× slower and with p99 spikes. There are three kernel levers that decide the latency tail and that almost nobody touches: locality (NUMA affinity: CPU, memory, GPU and NIC on the same “floor”), page tables (hugepages: a few large pages instead of millions of small ones, plus pinned memory for DMA), and jitter (CPU isolation with isolcpus/nohz_full/rcu_nocbs plus IRQ affinity, so that the kernel does not interrupt the thread launching the next decode kernel). This post explains the mechanism, gives the 10 real knobs, and connects with the interconnect and latency-bound decode. With scepticism about which knobs move the needle in inference and which are cargo cult inherited from low-latency trading.

Where you are: the host below the cable

The analogy: the two-floor office

Picture a consultancy in a building with two floors. On each floor there are desks (CPU cores), an archive with the files (the memory of that socket) and a loading bay connecting to the outside (the PCIe lanes towards the GPUs and NICs of that socket). An analyst works fast while everything they need is on their floor: they reach out and take the file from the archive next to them.

The problem starts when the analyst is on floor 1 but their file is in the archive on floor 2. Every time they need it, they take the lift. The work “functions”, but every lookup costs a trip. And if on top of that the loading bay through which their materials arrive (their GPU) is on the other floor, every delivery crosses the building. This is NUMA: local access (same floor) is fast; remote access (another floor, via the inter-socket link) is 2-3× slower.

And there are two more ways to ruin that analyst even when they are on the right floor:

Interrupting them constantly. Every few minutes, a tannoy announcement, a colleague with a question, a fire alarm test. Every interruption breaks their concentration right when they were about to deliver. This is kernel jitter: the scheduler tick, device IRQs, RCU callbacks, interrupting the host thread right when it was about to launch the next GPU kernel. CPU isolation is putting them in an office with a “do not disturb” sign.
Giving them an index of a thousand tiny tabs. If to find each file they have to search an index with a million minuscule entries, they lose time in the search. If the index has a few large entries, they find it instantly. These are hugepages: pages of 2 MB or 1 GB instead of 4 KB reduce the pressure on the TLB (the cache of the page index).

The thesis: the GPU is expensive and fast, but it spends a surprising fraction of decode waiting on the host. If the host is on the wrong floor, interrupted, and searching a giant index, the GPU, the 30,000 € resource, waits. The three levers in this post exist so that it does not wait.

The mechanism: what the host does on the token path

It is tempting to think that in inference “the GPU does everything”. It is not true. For every token, the host (CPU) does, at a minimum:

Launch the CUDA kernels for each operation. The GPU does not decide what to execute; the host keeps putting kernels in its queue. In decode, where each kernel is short, the host has to run ahead feeding the queue; if the host thread stalls, the GPU runs out of work: a bubble.
Sample the next token (argmax/top-p/top-k over the logits), which comes back from device to host.
Tokenise the input and detokenise the output.
Move pinned buffers (page-locked) between host and GPU by DMA: prompts, logits, and in offload configurations, part of the KV cache.
Run the NCCL threads that coordinate the collectives of the previous post.

All of that is CPU and host memory work. And all of it suffers if:

The process runs on the socket that is not local to its GPU → every DMA and every memory access crosses the inter-socket link.
The kernel interrupts the threads → bubbles in the GPU queue.
The memory is not pinned or uses small pages → page faults, TLB misses, and worse: if the memory of the DMA transfer is not pinned, the driver makes an intermediate copy.

The map: `nvidia-smi topo -m`

Everything starts by looking at the map. nvidia-smi topo -m shows, for each GPU, which NUMA node and which cores it is local to, and through what kind of path it talks to each NIC and to each other GPU:

 GPU0 GPU1 GPU2 GPU3 NIC0 CPU Affinity NUMA Affinity
GPU0 X NV18 NV18 NV18 PIX 0-31,64-95 0
GPU1 NV18 X NV18 NV18 SYS 0-31,64-95 0
GPU2 NV18 NV18 X NV18 SYS 32-63,96-127 1
GPU3 NV18 NV18 NV18 X SYS 32-63,96-127 1

Read it like this: GPU0 and GPU1 are local to NUMA node 0 (cores 0-31, 64-95); GPU2 and GPU3 to NUMA node 1. NV18 between GPUs = 18 NVLink links (the good stuff, from the previous post). In the NIC column: PIX = a single PCIe switch in between (optimal for GPUDirect RDMA); SYS = the path crosses the inter-socket link (the worst). The rule: the process serving on GPU0/1 must be pinned to cores 0-31/64-95 and to the memory of node 0; and if it also uses RDMA, you want the NIC that is on PIX with its GPU.

The maths that matter: a kernel tick is a GPU bubble

The figure worth internalising is not the bandwidth one, it is the jitter one. In decode, the host launches many short kernels per token. If the host thread that launches them is preempted by the scheduler (a timer tick, an IRQ, an RCU callback) for $t_{\text{stall}}$, and the GPU drains its queue in that time, a bubble appears: the GPU stops.

Put numbers on it. A typical timer tick or the handling of an IRQ costs of the order of tens of microseconds of diversion. If a decode kernel lasts around 50-100 µs and the queue carries 2-3 kernels in flight, a host stall of 50-100 µs drains the queue and the GPU sits idle until the host resumes. Multiply by the interrupt frequency of a non-isolated kernel (the default tick is 250-1000 Hz, plus network and disk IRQs): the p99/p999 tail of TTFT and of inter-token time fills up with these episodes.

$$ \text{jitter}_{p99} \approx f_{\text{interrupts}} \times t_{\text{stall}} \times \mathbb{1}[\text{GPU queue drained}] $$

The intuition: in average throughput it is barely noticeable (the bubbles average out), but in the tail, which is what an SLO measures, kernel jitter is a first-order contributor. That is why CPU isolation, which was born in low-latency trading, makes sense in LLM decode: it is the same physics, a critical thread that cannot afford the kernel stopping it.

And the NUMA cost, in parallel: a remote memory access (another floor) has latency around 1.5-2× the local one and half the bandwidth. For the pinned buffers moved by DMA on every step, and for the vLLM scheduler structures that live on the host, that penalty is paid token by token.

The three levers, one by one

Locality (NUMA): getting everything on the same floor

The goal is that the inference process using GPU0/1 has its cores, its memory and (if applicable) its NIC on NUMA node 0. Raw, without Kubernetes:

# Pin process to node 0 (cores and memory) to serve on GPU0/1
numactl --cpunodebind=0 --membind=0 \
 vllm serve meta-llama/Llama-3-70B --tensor-parallel-size 2

--membind=0 is the key: it forces all the process memory to be allocated on node 0. Without --membind, the kernel can place pages on node 1 under pressure, and you start paying for the lift without knowing it.

Page tables (hugepages): a few large pages and pinned memory

Two different things under the same umbrella. First, hugepages reduce TLB pressure for the large host buffers (pinned, KV offload). Second, pinned memory (page-locked) is what allows direct DMA with no intermediate copy. The silent trap is transparent hugepages (THP): their background compaction causes latency spikes, exactly what you do not want.

Jitter (CPU isolation): the office with “do not disturb”

Three kernel boot parameters, coordinated:

isolcpus=2-31,66-95 # take these cores out of the scheduler's balancing
nohz_full=2-31,66-95 # tickless: no timer tick if there is only 1 runnable thread
rcu_nocbs=2-31,66-95 # offload RCU callbacks to housekeeping cores

isolcpus sets the cores aside; you have to pin the inference threads there (the non-isolated cores, 0-1, are left for the system). nohz_full removes the periodic tick (it only works if there is a single runnable thread on the core). rcu_nocbs takes RCU work off those cores. And separately, IRQ affinity: moving device interrupts away from the inference cores.

The 10 knobs worth touching

Ordered by impact/frequency. Almost all of them are sysctls, kernel boot parameters or numactl. The low-level reference is Rigtorp’s low-latency guide and Red Hat’s real-time docs.

Knob 1 — `nvidia-smi topo -m`: look at the map before touching anything

Just as in the interconnect post: the map first. Which GPU is local to which NUMA node and to which cores, and what path (PIX/PHB/SYS) there is to each NIC. Without this, any pinning is blind. Half of the “inference has latency spikes” problems are processes running on the wrong socket without anyone having looked.

Knob 2 — `numactl --cpunodebind --membind`: pin to the local node

numactl --cpunodebind=0 --membind=0 <process> # cores AND memory on node 0
numactl --hardware # see nodes, distances, free memory

The --membind is what really matters: without it, the memory scatters. It is the highest-impact knob on sustained throughput.

Knob 3 — `kernel.numa_balancing=0`: turn off automatic migration

sysctl -w kernel.numa_balancing=0

The kernel’s automatic NUMA balancing migrates pages between nodes trying to “bring them closer”, but that background work causes jitter and, with the memory already pinned by knob 2, it adds nothing. On dedicated inference nodes, turn it off.

Knob 4 — Explicit hugepages (1 GB) for host buffers

# Kernel boot, for KV offload / large pinned buffers
default_hugepagesz=1G hugepagesz=1G hugepages=32

grep Huge /proc/meminfo # verify the reservation

Useful when there is host memory on the hot path (vLLM with --cpu-offload-gb, or large pinned buffers). If your deployment does not touch host memory in the hot path, explicit hugepages add little, so do not set them out of cargo cult.

Knob 5 — THP on `madvise` or `never`: avoid the compaction spikes

echo madvise > /sys/kernel/mm/transparent_hugepage/enabled
echo never > /sys/kernel/mm/transparent_hugepage/defrag

Transparent hugepages on always save TLB but their compaction triggers unpredictable latency. For tail-sensitive workloads, madvise (only where the app asks for it) or never is what is recommended. It is one of the few knobs with clear consensus: THP always is bad for latency.

Knob 6 — `isolcpus`: set the inference cores aside from the scheduler

isolcpus=2-31,66-95

Takes those cores out of the scheduler’s load balancing; the system (kernel threads, daemons) stays on the non-isolated ones. You have to pin the inference threads explicitly to the isolated cores (via numactl/taskset or, on K8s, the CPU Manager of the next post). Isolating without pinning is useless.

Knob 7 — `nohz_full` + `rcu_nocbs`: tickless and no RCU on the critical cores

nohz_full=2-31,66-95 rcu_nocbs=2-31,66-95

Removes the periodic timer tick and the RCU callbacks from the inference cores. Two warnings from practice: nohz_full only eliminates the tick if there is a single runnable thread on the core (if you pin two threads there, the tick comes back); and nohz_full is not compatible with the intel_pstate driver in some configurations, which has to be validated, not assumed.

Knob 8 — IRQ affinity: interrupts, away from the inference cores

systemctl stop irqbalance # or configure it to respect isolcpus
# move a device's IRQs to the housekeeping cores (0-1)
echo 3 > /proc/irq/<N>/smp_affinity # mask for cores 0-1

A network or disk IRQ landing on an inference core is a direct interruption of the thread feeding the GPU. Move them to the housekeeping cores. (irqbalance can respect isolcpus automatically if it is configured to.)

Knob 9 — CPU governor `performance` + C-states

cpupower frequency-set -g performance
# stop idle cores entering deep C-states (wakeup latency)
cpupower idle-set -D 0 # or limit the C-state depth

With the powersave/ondemand governor, a core that was idle takes time to ramp up its frequency: wakeup latency exactly when work arrives. performance keeps it flat out. On servers dedicated to inference, the energy saving does not compensate for the latency tail.

Knob 10 — Memory locking + `swappiness=0`

sysctl -w vm.swappiness=0 # do not evict inference pages to swap
# and in the app / container: ulimit -l unlimited (memlock) for pinned memory

An inference page that the kernel decides to swap to disk is a page fault of milliseconds on the hot path. swappiness=0 and adequate memlock limits (so the driver can pin memory) close that door. On K8s, this translates into Guaranteed QoS and memory limits, the bridge to the next post.

Summary table

#	Knob	Mechanism	What it attacks
1	`nvidia-smi topo -m`	diagnosis	see GPU–NUMA–NIC affinity
2	`numactl --cpunodebind --membind`	pinning	locality (the biggest lever)
3	`kernel.numa_balancing=0`	sysctl	jitter from page migration
4	explicit 1G hugepages	boot param	TLB on host buffers
5	THP `madvise`/`never`	sysfs	compaction spikes
6	`isolcpus`	boot param	scheduler off the critical cores
7	`nohz_full`+`rcu_nocbs`	boot param	tick + RCU jitter
8	IRQ affinity	`/proc/irq`	device interrupts
9	`performance` governor	cpupower	frequency wakeup latency
10	`swappiness=0` + memlock	sysctl/ulimit	page faults on the hot path

How it connects with the rest of the stack

With the interconnect (previous post). The NCCL host threads want cores local to their GPU; and for multi-node, the RDMA NIC must be on the PIX path with its GPU (knob 1). A GPUDirect RDMA with the NIC under the other socket loses half its advantage. NUMA and NVLink are the same story seen from the host and from the cable.

With vLLM and decode. Decode is latency-bound: the host thread feeding the kernel queue is exactly the one CPU isolation protects. And vLLM’s --cpu-offload-gb puts host memory on the hot path, where NUMA locality plus hugepages (knobs 2, 4) go from “nice” to “critical”. Continuous batching helps here too: more tokens per iteration amortise both the collective latency and the fixed cost of the host launches.

With Kubernetes (next post). Everything in this post is done by hand (numactl, taskset, boot parameters). In production it is not done by hand: the kubelet automates it with CPU Manager, Memory Manager and Topology Manager. The next post is exactly how this is declared so that every vLLM pod is born pinned to the right NUMA node, with no scripts.

With observability. The p99 spikes from jitter or from remote access are visible: in DCGM, low GPU utilisation with a full queue (bubbles); in system metrics, inter-socket traffic and CPU migrations. GPU observability with DCGM is where you diagnose a “the GPU is at 60 % and I do not know why” that many times is the host waiting.

With capacity planning. Reserving cores for the system (housekeeping) and dedicating the rest to inference changes the calculation of how many pods/replicas fit per node. Capacity planning has to count those reserved cores, not assume that all 128 vCPUs are available for serving.

Traps and things that are not what they look like

Cargo cult from low-latency trading. Many isolcpus/nohz_full guides come from HFT, where the last microsecond is squeezed out. In LLM inference, CPU isolation does help on the decode tail, but do not expect a miracle: if your bottleneck is HBM bandwidth or the interconnect, isolating cores does not move the needle. Measure first; apply it where the host is the limit.

Isolating without pinning. isolcpus takes the cores out of the scheduler, but if you do not pin the inference threads there, those cores stay empty and inference runs on the housekeeping ones, worse than before. Isolating and pinning always go together.

Forgotten --membind. Pinning cores but not memory (--cpunodebind without --membind) lets the pages scatter to the other node under pressure. Memory pinning is the half that gets forgotten most and the one that pays off most.

THP always “because it saves TLB”. It saves TLB and throws in latency spikes from compaction. For workloads with a tail SLO, it is a bad trade. madvise/never.

nohz_full with two threads on the core. Tickless only works with a single runnable thread per core. If you pin two inference threads to the same isolated core, the tick comes back and you have complicated the kernel boot for nothing.

Assuming the topology instead of reading it. Different servers wire GPUs and NICs to different sockets. nvidia-smi topo -m and numactl --hardware are the truth; the manufacturer’s diagram is indicative. Read it on every node model.

Conclusion

The GPU is the expensive resource, but in decode it spends a surprising part of its time waiting on the host: waiting for it to launch the next kernel, to sample, to move a buffer. If that host is on the wrong floor (remote NUMA), interrupted (kernel jitter) or searching a giant index (4 KB pages), the GPU is left with an empty queue and p99 blows up, without any API dashboard saying why. Of the ten knobs, the first (reading the map with nvidia-smi topo -m) and the second (pinning cores and memory to the local node with --membind) solve most of it; CPU isolation (isolcpus/nohz_full/IRQ affinity) is the second layer, the one that trims the decode tail, and it makes sense where the host is the limit, not as a ritual. The idea that reorders the intuition: inference is not “all GPU”; it is a dance between GPU and host, and the host dances better close by, uninterrupted, and with few large pages. The next post shows how Kubernetes choreographs that dance for every pod without a single hand-written script.

References

Erik Rigtorp, Low Latency Tuning Guide (isolcpus, nohz_full, IRQ affinity, THP): https://rigtorp.se/low-latency-guide/.
Red Hat, Usage, constraints and implications of isolcpus=, nohz_full= and rcu_nocbs=: https://access.redhat.com/articles/3720611.
NVIDIA, nvidia-smi topo -m and the GPU–NUMA–NIC affinity matrix (CUDA docs / Developer Forums): https://forums.developer.nvidia.com/t/nvidia-smi-topo-m-revisited/216584.
Chaim Rand, The Crucial Role of NUMA Awareness in High-Performance Deep Learning: https://chaimrand.medium.com/the-crucial-role-of-numa-awareness-in-high-performance-deep-learning-99ae3e8eb49a.
SUSE Labs, CPU Isolation – nohz_full (part 3): https://www.suse.com/c/cpu-isolation-nohz_full-part-3/.
Linux kernel, Automatic NUMA Balancing and Transparent Hugepage Support (Documentation/admin-guide): https://docs.kernel.org/admin-guide/mm/transhuge.html.

The shared bench: NVLink, NVSwitch and NCCL, the cable every token crosses in tensor parallel

Sat, 06 Jun 2026 07:00:00 +0200

This post goes one floor below the engine. In the seven-layer inference stack and in one big replica or many small ones the decision was how many GPUs and how to split the model; here we explain the cable that makes that split work, or that strangles it. It is the first of a mini-series “below the engine”: interconnect (this one) → kernel and NUMA → Kubernetes resource managers.

TL;DR

Tensor parallelism (TP) does not split a model into four chunks that run on their own. It spreads each layer across the GPUs, but after attention and after the MLP the GPUs have to add up their partial results with an all-reduce. On a Llama-70B with 80 layers, that is around 160 all-reduces per generated token. That all-reduce travels over the interconnect, so the interconnect sits on the critical path of every token, not in the background plumbing. On an HGX H100 baseboard, the 8 GPUs all talk to each other at 900 GB/s bidirectional via four NVSwitch chips; without NVSwitch/NVLink, that same traffic falls back to the CPU over PCIe and loses an order of magnitude. NCCL is the library that decides how each collective is done (ring, tree, or NVLS = NVLink-SHARP, which offloads the summation to the switch itself). And there is an asymmetry almost nobody keeps in mind: decode is latency-bound (tiny messages, 16 KB) and prefill is bandwidth-bound (huge batched activations), which is why “more NVLink bandwidth” speeds up prefill but barely touches token-by-token decode. This post explains the mechanism, gives the 10 real NCCL/driver knobs worth touching, and connects with vLLM’s custom all-reduce, disaggregated serving and GPU observability. With scepticism about which levers move the needle.

Where you are: the floor below the engine

The analogy: four mechanics and a single bench

Four mechanics assemble one and the same car engine. It is not that each builds their own engine in parallel, that would be four cars (four replicas of the model, a different strategy). Here they build a single one, at the same time, splitting the parts: one does the pistons, another the cylinder head, another the crankshaft. The problem is that the parts fit into each other: before going on, all four have to put together what they have and check that it matches. That “put together and check” happens dozens of times during the build.

There are two ways to organise the workshop:

A single large bench, everyone around it (NVSwitch). Each mechanic reaches out and hands their part directly to any other, all at once, without standing up. It is instant and simultaneous. This is NVLink + NVSwitch: the GPUs form an all-to-all where anyone talks to anyone at 900 GB/s at the same time.
Four separate workshops with a courier (PCIe via CPU). Every part a mechanic wants to hand to another goes into a box, goes down to reception (host memory, via CPU), and from there up to the destination workshop. Slower, and serialised by reception. This is what happens when there is no NVLink: inter-GPU traffic falls back to PCIe and bounces through the CPU, around 14× slower than NVLink.

The thesis of the post follows on its own: tensor parallelism only makes sense if the mechanics share the bench. As soon as the “put together and check” (the all-reduce) has to go through reception, splitting the work costs more than it saves. That is why, on a serious platform, TP does not cross the NVLink boundary: TP=4 or 8 inside the baseboard where there is NVSwitch, and above that you replicate or use pipeline, never stretch TP over PCIe or over the network. When each option makes sense is covered in one big replica or many small ones; here we explain why the cable dictates that decision.

The mechanism: what an all-reduce really is and why there are 160 per token

Tensor parallelism splits the weight matrices by columns/rows across the $N$ GPUs. Each GPU computes a portion of the layer output. But the next operation needs the complete output, so it has to be recombined. That recombination is a collective operation: an all-reduce, which adds the partial tensors of all the GPUs element by element and leaves the result identical on all of them.

In a standard transformer block there are two synchronisation points per layer:

After the output projection of attention (the o_proj that recombines the split heads).
After the second matrix of the MLP (the down_proj that recombines the split feed-forward).

$$ \text{all-reduces per token} = 2 \times L_{\text{layers}} $$

For a Llama-70B ($L = 80$): $2 \times 80 = 160$ all-reduces per generated token. Not per request, not per sequence: per token. Multiply by decode throughput and you understand why the interconnect is not background infrastructure but a hot path.

How the all-reduce is done: ring, tree, NVLS

NCCL does not have a single way of doing an all-reduce; it picks an algorithm according to topology and message size:

Ring. The GPUs form a ring; each one passes a chunk to its neighbour, adds, and rotates. It takes $2(N-1)$ steps. It is bandwidth-optimal for large messages: the cost of moving the data is $\frac{2(N-1)}{N} \times M$ bytes over the link, almost independent of $N$. The downside: $2(N-1)$ latency hops, bad for small messages.
Tree. Tree reduction: $\log N$ levels. Better latency for small messages and many nodes, worse bandwidth utilisation.
NVLS (NVLink SHARP). The Hopper trick: the summation is not done by the GPUs, it is done by the NVSwitch. The switch has reduction units; the GPUs send their tensors, the switch adds them in flight and returns the result. It takes work away from the GPUs (frees up SMs) and reduces hops. Available only with a 3rd-generation NVSwitch (NVLink4) plus Hopper or newer.

The mental rule: decode (tiny messages) wants latency → tree/LL or vLLM’s custom kernel; prefill (huge messages) wants bandwidth → ring/NVLS. That is why there is no global “optimal NCCL_ALGO”; it depends on which phase you are looking at.

The maths that matter: why decode and prefill stress the cable the other way round

Here is the asymmetry almost everyone skips. The size of the tensor that gets all-reduced in each layer is, approximately:

$$ M \approx B \times S \times h \times 2\ \text{bytes (BF16)} $$

where $B$ = batch, $S$ = tokens processed in this forward pass, $h$ = hidden size.

In decode, you generate 1 token per sequence per iteration. For a single sequence ($B \times S = 1$) and $h = 8192$ (Llama-70B):

$$ M_{\text{decode}} \approx 1 \times 8192 \times 2 = 16\ \text{KB per all-reduce} $$

16 KB is tiny. At 900 GB/s, moving 16 KB takes around 18 nanoseconds of pure transfer, but the real cost is dominated by the launch latency of the collective (synchronisation, kernel launch), of the order of single-digit microseconds. With 160 all-reduces per token:

$$ t_{\text{comms/token}} \approx 160 \times (5\text{–}10\,\mu s) \approx 0.8\text{–}1.6\ \text{ms} $$

That is the communication floor per token, independent of bandwidth. An uncomfortable and counter-intuitive implication: buying more NVLink bandwidth does not speed up token-by-token decode of a single sequence. What helps in decode is lowering the latency per collective (LL protocol, vLLM’s custom all-reduce, NVLS to remove hops) and batching (raising $B$ amortises the fixed latency over more tokens, the deep reason continuous batching exists, covered in continuous batching).

In prefill, you process the whole prompt at once: $S$ can be thousands of tokens, and with batching $B \times S$ reaches tens of thousands. There:

$$ M_{\text{prefill}} \approx 8000 \times 8192 \times 2 \approx 131\ \text{MB per all-reduce} $$

131 MB does stress the bandwidth. At 900 GB/s (NVSwitch) the ring all-reduce moves $\frac{2 \cdot 3}{4} \times 131 \approx 196$ MB effective in around 0.22 ms; over PCIe (around 64 GB/s aggregate, bouncing through the CPU) it would be around 3 ms and serialised. Here the cable is the bottleneck and NVLS/bandwidth rule.

Summary in one line: prefill is bandwidth-bound, decode is latency-bound. Any interconnect tuning that does not say which phase it helps is noise.

The hardware: NVLink 4 and NVSwitch on the HGX baseboard

On the generic reference cluster, 4×H100 SXM inside an HGX baseboard, the concrete figures:

Each H100 SXM5 has 18 NVLink 4 links, each one 50 GB/s bidirectional ⇒ 900 GB/s bidirectional aggregate per GPU. That is >14× the bandwidth of a PCIe Gen4 x16 (around 64 GB/s bidir).
On an 8-GPU HGX H100 baseboard, the 18 links of each GPU are spread against four 3rd-generation NVSwitch chips (grouped 5+4+4+5). The result is all-to-all: any GPU talks to any other at 900 GB/s simultaneously, without going through CPU or PCIe.
A 4-GPU baseboard is half a board: same principle, via NVSwitch. Design key: if your 4 H100s are connected by NVSwitch, you have real all-to-all; if they are on different boards connected by PCIe (some “4×PCIe” configurations), you do not have NVLink between all of them and TP=4 suffers. Verify it, do not assume it.

The 10 knobs worth touching

Almost all of them are NCCL environment variables (injected into the inference engine process) or driver settings. Ordered by impact/frequency in an on-premise deployment. The canonical detail is in the NCCL env vars docs.

Knob 1 — `NCCL_DEBUG` + topology dump: see what is happening before touching anything

Do not optimise blind: first confirm which topology and algorithms NCCL picked. This tells you whether it really is using NVLink or whether, silently, it fell back to PCIe/SHM, the number one failure and the most expensive one.

NCCL_DEBUG=INFO # prints topology, rings/trees built, transport chosen
NCCL_DEBUG_SUBSYS=GRAPH,TUNING,NET # narrows it down to what matters
# look in the log for: "via NVLink" / "via P2P" (good) vs "via SHM" / "via PCI" (bad)

If you see via SHM or via PCI between GPUs that should have NVLink, you have a topology problem (PCIe ACS enabled, IOMMU, GPUs on different boards) and no other knob fixes it. This is knob 1 for a reason: half of the “NVLink is slow” cases are “NVLink is not being used”.

Knob 2 — `NCCL_ALGO`: ring vs tree vs NVLS

Forces or excludes algorithms. By default NCCL picks according to size, and it usually gets it right; touch it only with a measurement in front of you.

NCCL_ALGO=NVLS,Tree,Ring # order of preference
NCCL_ALGO=^Ring # exclude Ring (^ prefix)

Rule: prefill/training (bandwidth) ⇒ Ring/NVLS; decode (latency) ⇒ Tree or, better, vLLM’s custom kernel (knob 10/stack). In most inference workloads, leaving it on auto and validating with knob 1 is the right thing; forcing it “just in case” usually makes things worse.

Knob 3 — `NCCL_PROTO`: LL / LL128 / Simple

The protocol controls the latency/bandwidth trade-off at low level:

NCCL_PROTO=Simple # maximum bandwidth, more latency (large messages)
NCCL_PROTO=LL # low-latency, half-bandwidth (tiny messages: decode)
NCCL_PROTO=LL128 # compromise, default on platforms that support it

LL (low-latency) uses flags instead of barriers and wins on the 16 KB messages of decode; Simple wins on the 131 MB of prefill. The default LL,LL128,Simple lets NCCL choose by size, which again is normally the best option.

Knob 4 — `NCCL_NVLS_ENABLE`: offload the summation to the NVSwitch

NVLink SHARP (NVLS) makes the switch do the reduction, freeing up GPU SMs:

NCCL_NVLS_ENABLE=1 # default: ON where there is an NVLink4+ NVSwitch (Hopper)

An important sceptical caveat: NVLS requires NVSwitch (3rd gen, NVLink4). On a node with NVLink through direct GPU-to-GPU bridges (no switch) or on 4×PCIe, NVLS is not available and this knob does nothing. Before “enabling it”, confirm with knob 1 that your topology has a switch. Where it does apply, its biggest advantage is freeing SMs for compute, which is relevant when comms and kernels compete (knob 5).

Knob 5 — `NCCL_MIN_NCHANNELS` / `NCCL_MAX_NCHANNELS`: how many SMs communication steals

Each NCCL “channel” consumes GPU SMs to move data. More channels = more collective bandwidth, but fewer SMs for the inference kernel. It is a split of a fixed resource.

NCCL_MIN_NCHANNELS=4
NCCL_MAX_NCHANNELS=16 # raising it helps prefill (bandwidth); steals SMs from decode

In decode, where the GPU is under-used on compute but tied to latency, trimming channels rarely hurts and sometimes helps; in prefill, more channels squeeze out the bandwidth. A measurement knob, not a faith knob.

Knob 6 — `NCCL_BUFFSIZE`: the buffer size per channel

NCCL_BUFFSIZE=8388608 # 8 MB (default 4 MB); larger buffers → better BW on large messages

Raising it helps bandwidth-bound prefill at the cost of memory per channel. For workloads dominated by small messages (pure decode), the default is plenty.

Knob 7 — `NCCL_P2P_LEVEL` / `NCCL_P2P_DISABLE`: guarantee P2P over NVLink

P2P is what lets one GPU read another’s memory directly over NVLink without going through the host. If it is disabled or degraded, traffic falls back to SHM/PCIe.

NCCL_P2P_LEVEL=NVL # use P2P up to the NVLink level
# NCCL_P2P_DISABLE=1 ← only as a workaround if P2P HANGS (multi-NUMA PCIe, certain Blackwell)

Watch out for the trap: NCCL_P2P_DISABLE=1 and --disable-custom-all-reduce are recommended as a patch when vLLM hangs on PCIe-only multi-NUMA topologies. It is a robustness patch that sacrifices performance: use it if it hangs, never “by default”.

Knob 8 — GPUDirect RDMA for multi-node: `NCCL_NET_GDR_LEVEL`

When TP fits in one node, this does not apply. When you have to cross nodes (huge model, pipeline parallel between baseboards), GPUDirect RDMA lets the GPU talk to the NIC without bouncing through host memory:

NCCL_NET_GDR_LEVEL=PHB # enables GDR according to GPU–NIC proximity on the PCIe bus

Without GDR, every inter-node hop adds a host copy. With InfiniBand/RoCE plus GDR, the KV or the activations travel GPU→NIC→network→NIC→GPU. It is the basis of serious multi-node work and of mixed environments.

Knob 9 — `NCCL_IB_HCA` / `NCCL_SOCKET_IFNAME`: pin the right NIC

The most common and silent multi-node mistake: NCCL picks the management NIC (1 GbE) instead of the fabric one (InfiniBand/100 GbE). Result: collectives at a snail’s pace with no visible error.

NCCL_SOCKET_IFNAME=eth0 # control interface (bootstrap)
NCCL_IB_HCA=mlx5_0,mlx5_1 # the real InfiniBand HCAs of the fabric
NCCL_IB_GID_INDEX=3 # correct GID for RoCE v2

Pin them explicitly. “Auto” gets it right on clean clusters and fails as soon as there is more than one NIC.

Knob 10 — Driver: persistence mode, clocks and NVLink error counters

Below NCCL, the driver has levers and, above all, telemetry you have to look at:

nvidia-smi -pm 1 # persistence mode: avoids driver re-init (latency/jitter)
nvidia-smi nvlink --status # are all 18 links active and at full speed?
nvidia-smi nvlink -e # error/CRC counters per link
nvidia-smi -q -d ECC # memory errors that degrade silently

An NVLink link that negotiates at half speed or accumulates CRC errors degrades the all-reduce without raising any error: the system “works”, it is just slower. These counters are the difference between diagnosing in five minutes and chasing a ghost for days. They are integrated in DCGM (knob/stack: observability).

Summary table

#	Knob	Variable / command	Phase it helps
1	Topology diagnosis	`NCCL_DEBUG=INFO` + `SUBSYS=GRAPH`	always, first
2	Collective algorithm	`NCCL_ALGO` (NVLS/Tree/Ring)	depends on phase; auto usually wins
3	Protocol	`NCCL_PROTO` (LL/LL128/Simple)	LL=decode, Simple=prefill
4	NVLink SHARP	`NCCL_NVLS_ENABLE=1`	prefill; frees SMs (requires NVSwitch)
5	Channels (SMs)	`NCCL_MIN/MAX_NCHANNELS`	+prefill bandwidth / −SM theft in decode
6	Buffer	`NCCL_BUFFSIZE`	bandwidth-bound prefill
7	NVLink P2P	`NCCL_P2P_LEVEL=NVL`	critical; disable only if it hangs
8	GPUDirect RDMA	`NCCL_NET_GDR_LEVEL`	multi-node
9	Fabric NIC	`NCCL_IB_HCA`/`SOCKET_IFNAME`	multi-node (avoids mgmt NIC)
10	Driver + telemetry	`nvidia-smi -pm 1` / `nvlink -e`	jitter + silent diagnosis

How it connects with the rest of the stack

The interconnect is not an island; it touches almost every layer above.

With vLLM, the custom all-reduce. vLLM does not always use NCCL: for the tiny messages of decode (world_size==2 or a fully-connected NVLink topology, below a certain max_size) it uses its own all-reduce kernel that beats NCCL on latency, exactly the decode bottleneck we saw in the maths. It falls back to NCCL for large messages and for topologies without NVLink (where its custom kernel “adds little over NCCL”). The flag --disable-custom-all-reduce / VLLM_DISABLE_CUSTOM_ALL_REDUCE turns it off; it is the patch for hangs on multi-NUMA PCIe. Translation: the most effective decode latency knob is sometimes not an NCCL one, it is choosing well between vLLM’s custom kernel and NCCL.

With TP vs replicas. Everything in one big replica or many small ones rests on this: high TP is only viable inside the NVLink domain. The boundary of “TP=4 or 4 replicas at TP=1?” is drawn by the cable: crossing NVLink with TP means paying for the all-reduce at PCIe prices.

With disaggregated serving. In disaggregated prefill/decode, the KV cache generated in the prefill pool has to travel to the decode pool. That transfer is another consumer of the interconnect (NVLink intra-node, GPUDirect RDMA inter-node) and competes with the all-reduces. Designing the disaggregation without counting the cost of transferring the KV is the classic trap.

With MoE. Mixture-of-Experts models add expert parallelism: an all-to-all (not an all-reduce) that routes each token to its expert, possibly on another GPU. It is a different communication pattern and heavier on bandwidth; MoE in inference lives or dies by the same cable, with an even more demanding collective.

With GPU observability. The NVLink counters (nvidia-smi nvlink -e, TX/RX bytes per link, CRC errors) and NVSwitch utilisation are exposed via DCGM and land in Prometheus/Grafana. The question “is the interconnect healthy and saturated?” is answered there, alongside the rest of GPU observability with DCGM. A slow all-reduce shows up sooner in an NVLink error counter than in API latency.

With capacity planning. The inference sizing that assumes “TP=4 scales almost linearly” only holds inside NVLink. Outside it, scaling efficiency collapses and the capacity plan lies. The cable is a parameter of the capacity model, not a detail.

Traps and things that are not what they look like

“More NVLink bandwidth = faster decode.” False for a single sequence. Decode is latency-bound; bandwidth is barely touched by 16 KB messages. What speeds up decode is batching (amortising the fixed latency) and lowering latency per collective (LL, custom kernel, NVLS). Bandwidth rules in prefill.

“I have 4 H100s, so I have NVLink between all four.” Not necessarily. There are configurations where the GPUs are on different boards joined by PCIe, or with NVLink bridges only in pairs. Confirm it with nvidia-smi nvlink --status and knob 1 before planning TP=4. A TP=4 over P2P-over-PCIe performs far worse than the brochure says.

Forcing NCCL_ALGO/NCCL_PROTO “to go faster”. NCCL picks well by size in most cases. Forcing an algorithm without measuring usually makes one of the two phases worse. The correct sequence is: knob 1 (see what it is doing) → measure → touch only if there is evidence.

Disabling P2P/custom all-reduce by default. They are robustness patches for broken topologies (multi-NUMA PCIe, certain Blackwell). Leaving them on “for stability” on a node with healthy NVLink throws performance in the bin.

Stretching TP over the network. TP=8 crossing two nodes over InfiniBand because “there is bandwidth” ignores that the per-layer all-reduce now pays network latency ×160 per token. To cross nodes, pipeline parallel (which communicates once per micro-batch, not per layer) almost always wins. The communication pattern, not just the bandwidth, decides.

Ignoring the NVLink error counters. A degraded link does not raise an exception: the system works, it is just slow. Without watching nvlink -e and ECC, you chase a performance ghost that a counter would have pointed out in five minutes.

Conclusion

Tensor parallelism sells a simple promise, split the model, multiply the VRAM, serve models that do not fit in one GPU, but the small print is that every layer forces the GPUs to come together and add up, twice, dozens of times per token. That all-reduce is the real hidden protagonist of performance, and it lives in the cable: NVLink does it on the shared bench of the NVSwitch at 900 GB/s, or PCIe drags it through the CPU’s reception desk 14× slower. Of the ten knobs, the first one, looking with NCCL_DEBUG at what is really happening, solves half the problems, because half of the “NVLink is slow” cases are “NVLink is not being used”. The rest are refinements that only mean something if you know which phase you are in: prefill wants bandwidth (NVLS, Simple, channels, buffer), decode wants latency (LL, vLLM’s custom kernel, batching). And above all, an idea that reorders the intuition: in on-premise inference, the interconnect is not plumbing you install and forget, it is a hot path, a capacity parameter and, when it degrades silently, the root cause that no API dashboard will point out to you unless you look at the counters of the cable itself.

References

NVIDIA, NVIDIA Hopper Architecture In-Depth (NVLink 4, 900 GB/s): https://developer.nvidia.com/blog/nvidia-hopper-architecture-in-depth/.
NVIDIA, Introducing NVIDIA HGX H100 (4× NVSwitch, all-to-all): https://developer.nvidia.com/blog/introducing-nvidia-hgx-h100-an-accelerated-server-platform-for-ai-and-high-performance-computing/.
NVIDIA, NCCL Environment Variables (all the knobs in this post): https://docs.nvidia.com/deeplearning/nccl/user-guide/docs/env.html.
NVIDIA, The NVLink-Network Switch (Hot Chips 2022, NVLink SHARP): https://hc34.hotchips.org/.
vLLM, Why does vLLM use a custom all-reduce method? (discussion #6159) and custom_all_reduce.py: https://github.com/vllm-project/vllm/discussions/6159.
NVIDIA, NCCL Multi-Node NVLink Tuning Guide: https://docs.nvidia.com/multi-node-nvlink-systems/multi-node-tuning-guide/nccl.html.

Langfuse from the inside: the sorting centre that must not become the bottleneck it came to observe

Sat, 06 Jun 2026 06:00:00 +0200

This post closes a trilogy of the Observe layer: in LLM tracing with OpenTelemetry GenAI the SDK → Collector → backend pipeline was set up and Langfuse was treated as a black box that receives spans; in Prompt versioning with Langfuse and MLflow its prompt management layer was used. Here we open the box: what is inside Langfuse, why v3 stopped being a monolith on Postgres, and how it is operated so it holds the traffic of an inference cluster without becoming the problem.

TL;DR

Langfuse v3 (stable since December 2024) is not an application, it is six services: two containers of its own (Web and Worker) and four stateful dependencies (Postgres, ClickHouse, Redis/Valkey and an S3-compatible blob store). The key architectural change compared with v2, which was a Next.js monolith on Postgres, is the asynchronous ingestion pipeline: traces are received in batches, written immediately to S3, only a reference is queued in Redis, and a Worker ingests them into ClickHouse in the background. This decouples reception speed (limited only by Redis write latency, ~1-5 ms) from the cost of persisting and merging into the analytical database. The result: the Web container sustains hundreds of events per second without a spike blocking the client that serves the inference. But that design only performs with the right settings. This post covers the architecture, its interaction with the rest of the on-premise stack, and ten backend knobs, from batching to ClickHouse to queue sharding, from the FINAL modifier to system log table hygiene, that decide the real throughput and the storage cost. And it marks where the async design hides data loss windows worth knowing about before promising “full traceability”.

You are here: OBSERVE (and the layer that holds up the rest)

The analogy: the postal sorting centre

Picture the central mail sorting office of a large city at rush hour. Lorries loaded with sacks (batches of letters) arrive at a rate that does not stop. If the counter clerk had to open every sack, read every letter, decide its destination and file it before accepting the next lorry, the queue of lorries would go round the block in ten minutes. No serious sorting centre works like that.

What they do is decouple reception from processing:

The reception counter accepts the sack, stamps a receipt on it, leaves it in a pigeonhole in the warehouse and drops a ticket on a conveyor belt. Time per sack: seconds. The counter never blocks.
Further back, in the sorting room, a team of operators picks tickets off the belt, retrieves the sack from its pigeonhole, opens it, sorts the letters and files them in the permanent archive, ordered, indexed, searchable.

Langfuse v3 is exactly this sorting centre:

Postal centre	Langfuse v3	Function
Reception counter	Web container (ingestion endpoint)	Accepts event batches, gives an immediate receipt (HTTP 207)
Pigeonhole warehouse	S3 / Blob store (MinIO on-prem)	Stores the raw sack (the full event)
Ticket on the belt	Redis / Valkey (BullMQ queue)	Only the reference to the object in S3, not the content
Sorting room	Worker container	Takes tickets, reads S3, transforms and files
Indexed permanent archive	ClickHouse (OLAP)	Traces, observations and scores, queryable by project+time
Administrative register	Postgres (OLTP)	Users, projects, API keys, prompts, datasets, config

The thesis of the whole post follows from this analogy: the value of Langfuse lies in the counter never blocking the client that serves the inference. An observability tool that adds latency or outages to the token-serving path is worse than having no observability, because it degrades precisely the system it set out to look after. The whole of the v3 design, and every knob in this post, exists to keep that promise under load.

The mechanism itself: six services, two planes

Langfuse v3 separates two planes that in v2 were fused together:

Ingestion and query plane (the two containers of its own, stateless, horizontally scalable): Web and Worker.
State plane (four dependencies, each with its own load profile): Postgres (transactional OLTP), ClickHouse (analytical OLAP), Redis/Valkey (queue + cache), Blob store (raw objects).

What to take away from this diagram:

Web and Worker are interchangeable and stateless. They store nothing locally. You can run 1 or 20 replicas of each; the state lives in the four dependencies. This is what allows scaling by load without choreography.
Redis never carries the event content, only the reference to the object in S3. That is why Redis holds the spike: a Redis write is ~1-5 ms and moves bytes, not kilobytes. The bottleneck of the Web container is, literally, Redis write speed.
Postgres and ClickHouse have opposite profiles. Postgres is OLTP: many small transactional reads and writes (is this API key valid? which version does the production label point to?). ClickHouse is OLAP: few enormous batch writes and analytical queries over billions of rows (give me the p95 TTFT of project X over the last 7 days). Putting traces in Postgres, which is what v2 did, works until it does not: at production volume, Postgres drowns in a workload it was not designed for. That was the reason for the redesign.

The ingestion flow step by step (and the maths of the decoupling)

The heart of the design is the ingestion route. Seen in detail, a POST /api/public/ingestion request with a batch of events does this:

The mathematical point is the early ACK. The latency the client perceives when sending traces is:

$$ t_{\text{client}} = t_{\text{S3 write}} + t_{\text{Redis enqueue}} \approx 10\text{–}40\,\text{ms} $$

whereas the real cost of persisting, reading S3, transforming, merging against the previous version, inserting into ClickHouse, letting the background merges compact, happens off that path, in the Worker, and can take hundreds of ms or seconds without the client caring. The decoupling turns a system whose throughput would be limited by ClickHouse speed into one limited by Redis speed. And Redis, on modest hardware, sustains on the order of 50,000 operations/second.

This has an important sizing consequence. If your inference load generates $E$ events/second (a chat with RAG + 2 tool calls easily produces 6-10 spans = events per request), the Web container absorbs them while $E \ll 50{,}000$. The Worker, by contrast, scales with the cost of processing: that is the component to watch and replicate, and the first knob of the post.

Honest scepticism. The early ACK has a flip side: between the HTTP 207 and persistence in ClickHouse there is a potential loss window. If the event is in S3 and the reference in Redis, and Redis goes down without persistence (AOF/RDB) before the Worker processes it, the reference is lost; the data is still in S3 but nobody claims it any more. More subtly: the Worker buffers writes to ClickHouse in memory and flushes them in batches; a Worker crash with a full buffer loses that batch. There is a reported bug where the ClickhouseWriter discards rows after exhausting flush retries with no dead-letter queue. For observability this is usually tolerable (losing 0.01 % of the traces breaks nothing). For regulatory auditing, where the trace is evidence, it is not, and Langfuse is better treated as “best-effort” and not as a ledger. We will come back to this at the close.

Interaction with the rest of the stack: Langfuse in the example 4×H100 cluster

Langfuse does not live in isolation. In the seven-layer stack it occupies the LLM-aware observability layer, and it relates to almost every other one. On the generic reference cluster we use throughout the blog, 4×H100 SXM 80 GB (320 GB aggregate VRAM), NVLink, 640 GB of system RAM, NVMe-oF, 25/100 GbE network, the telemetry flow looks like this:

Three ideas from this topology:

Langfuse receives from the OTel Collector, not from the application directly (in the recommended pattern). The app SDK or vLLM emit spans with the gen_ai.* semantic conventions; the Collector does batch, tail-sampling (preserving 100 % of errors and high latencies, sampling the rest) and enriches with its own attributes (tenant_id, priority_tier); and it distributes: LLM traces go to Langfuse, infrastructure spans to Tempo, metrics (GPU DCGM, vLLM metrics) to Prometheus. Langfuse is one more exporter, not the only destination. This is covered in detail in the OTel tracing post.
Langfuse runs off the GPUs. It is a consumer of CPU, RAM, disk and network (ClickHouse wants memory, MinIO wants disk, Redis wants CPU for networking) but it does not touch VRAM. On the 4×H100 cluster, Langfuse lives on a CPU node (or on the GPU nodes but with nodeSelector/taints keeping it away from the vLLM pods). Mixing ClickHouse with vLLM on the same node without resource limits is asking for an ingestion spike to steal memory bandwidth from inference. Isolation by design.
The telemetry route is “cold” and the data route is “hot”. The data plane (left) serves tokens with a millisecond latency budget; the telemetry plane (right) tolerates seconds. The early ingestion ACK is what keeps these two clocks apart: the app does not wait for Langfuse to file anything before returning the response to the user.

The 10 backend knobs that move the needle most

These are, in approximate order of impact/frequency, the settings that decide whether your self-hosted Langfuse ingests 50 events/s or 5,000, and whether your disk grows sustainably or explodes in three weeks. They are all environment variables or config injected into the Web and Worker containers (except the ClickHouse ones, which go in its server-side config). The canonical detail is in the Langfuse scaling doc.

Knob 1 — Scale the Worker by load (the first lever, always)

The Worker is the component that saturates first, because it does the expensive work: reading S3, transforming, merging, inserting into ClickHouse. The Langfuse operational rule is simple: a 2-CPU Worker container above 50 % CPU use is saturated; add replicas. Better than CPU, the Worker publishes via statsd the metric langfuse.queue.ingestion.length (ingestion queue length), which is the direct signal for autoscaling: if the queue grows without draining, there are not enough Workers.

# The ideal autoscaler watches queue depth, not just CPU.
# (KEDA ScaledObject over the statsd metric → Prometheus)
triggers:
 - type: prometheus
 metadata:
 query: langfuse_queue_ingestion_length
 threshold: "10000" # if the queue goes past 10k refs, scale

In AWS deployments there is ENABLE_AWS_CLOUDWATCH_METRIC_PUBLISHING=true to push these metrics to CloudWatch. On-premise, the path is statsd → Prometheus → KEDA, slotting into the autoscaling on Kubernetes with KEDA we already covered for vLLM. Always start here: most “Langfuse is slow” problems are simply insufficient Workers, not fine tuning.

Knob 2 — Separate the ingestion deployment from the UI one

When ingestion is heavily loaded, UI queries and the public API become slow because they share the same Web container. The solution is to split langfuse-web into two identical deployments and route by path: everything under /api/public/ingestion*, /api/public/media* and /api/public/otel* goes to the ingestion deployment; the rest (UI, read API) to the interface one.

# Ingress / gateway rule
location ~ ^/api/public/(ingestion|media|otel) {
proxy_pass http://langfuse-web-ingest; # replicas dedicated to writing
}
location / {
proxy_pass http://langfuse-web-ui; # replicas dedicated to reading
}

It is the same idea as the read/write separation of any system with mixed loads: a storm of writes should not starve whoever is trying to look at the dashboard right in the middle of the incident, which is precisely when they need it most.

Knob 3 — Batching writes to ClickHouse (interval + batch size)

ClickHouse hates small, frequent inserts: every INSERT creates a part on disk that then has to be merged, and thousands of tiny inserts generate thousands of parts and a storm of background merges that saturates the disk. The defence is to accumulate in an in-memory buffer in the Worker and flush in large batches:

# Worker: fewer flushes, larger batches → fewer parts, fewer merges
LANGFUSE_INGESTION_CLICKHOUSE_WRITE_INTERVAL_MS=1000 # raise e.g. to 2000-5000
LANGFUSE_INGESTION_CLICKHOUSE_WRITE_BATCH_SIZE=10000 # raise if there is throughput

Raising the interval and the batch size reduces the flush frequency and improves sustained throughput. The trade-off is direct and has to be understood: larger, less frequent batches mean more data in the Worker’s volatile buffer, that is, a bigger loss window if the Worker goes down (a knob coupled to the scepticism at the close). Langfuse also uses ClickHouse async_insert, which accumulates server-side before confirming; it adds another layer of buffering to keep in mind.

Knob 4 — Skip the ClickHouse read-before-write in ingestion

By default, when ingesting an event the Worker reads the existing event from ClickHouse and merges it with the incoming one (necessary when legacy SDKs send partial events: a start, then an end, then an update of the same observation). That per-event read loads ClickHouse on the write path and limits total throughput.

If your projects are not migrated from an old version, because the full history already lives in S3, you can disable that read:

# Date earlier than the creation of your first project
LANGFUSE_SKIP_INGESTION_CLICKHOUSE_READ_MIN_PROJECT_CREATE_DATE=2025-01-01

With modern Langfuse SDKs or with ingestion via OpenTelemetry, this does not affect you negatively and removes one read per event. A warning from the doc itself: if you combine this with aggressive deletion (lifecycle) rules in S3 plus late event updates, you can generate duplicates in the history. Know that before enabling it.

Knob 5 — Write concurrency to S3/Blob storage

In high-throughput scenarios, the S3 client can exhaust its sockets and start queueing and throttling writes. The symptom is unmistakable in the logs of the Web container that handles ingestion:

@smithy/node-http-handler:WARN - socket usage at capacity=150
and 387 additional requests are enqueued.

…accompanied by a rise in memory in that container (queued requests pile up in RAM). The cure is to raise the concurrent write limit from its default of 50:

LANGFUSE_S3_CONCURRENT_WRITES=100 # raise gradually from 50

Each additional socket has a small memory cost, so the official advice is to raise it gradually while observing the behaviour, not to jump to 1000 in one go.

Knob 6 — Redis queue sharding + concurrency per shard

If Redis goes past 90 % CPU, first the obvious: an instance with at least 4 CPUs (so Redis spreads networking and background tasks across different cores) and Redis Cluster mode enabled. If CPU still does not come down, you can shard the queues Langfuse uses:

# Advanced: only if Redis is genuinely drowning and you already did the above
LANGFUSE_INGESTION_QUEUE_SHARD_COUNT=6 # ~2-3× the number of Redis cluster shards
LANGFUSE_TRACE_UPSERT_QUEUE_SHARD_COUNT=6
# Concurrency counts PER SHARD; target ~20 per worker
LANGFUSE_INGESTION_QUEUE_PROCESSING_CONCURRENCY=3 # 6 shards × ~3 ≈ 18
LANGFUSE_TRACE_UPSERT_WORKER_CONCURRENCY=3

Two traps the doc underlines and that are worth tattooing: once you shard, do not reduce the number of shards (it breaks the distribution); and concurrency counts per shard, not globally, so if you have 10 shards and want concurrency 20 per worker, set 2, not 20. It is an advanced knob: most on-premise deployments never need it.

Knob 7 — The `FINAL` modifier for OTel-only projects

Langfuse stores observations in a ClickHouse ReplacingMergeTree and, by default, adds the FINAL modifier to API queries so the last version of each row wins at read time. FINAL is necessary when ingestion produces several versions of the same observation (legacy SDKs with their start/end/update events), but it adds merge work to every read and slows it down.

Projects that ingest exclusively over OpenTelemetry write each observation as a single immutable row, so FINAL is redundant for them:

# Recommended in mixed deployments: per-project, flagged in Redis with a 24h TTL
LANGFUSE_SKIP_FINAL_FOR_OTEL_PROJECTS=true

# Only if ALL projects are OTel-only: global, no Redis lookup
LANGFUSE_API_CLICKHOUSE_DISABLE_OBSERVATIONS_FINAL=true

Since on the example cluster the instrumentation is 100 % OTel (gen_ai.* via Collector), this knob is free money in dashboard read latency. Careful with the global version: do not enable it if some project is still using legacy ingestion, or reads may return duplicate or stale rows.

Knob 8 — Separate analytical reads from the write path (compute-compute)

Heavy dashboard queries (percentiles over millions of spans) compete with ingestion inserts and with background merges on the same ClickHouse. If your deployment supports compute-compute separation (ClickHouse Cloud or BYOC), you can route reads to a read-only compute group:

CLICKHOUSE_URL=http://clickhouse-primary:8123 # writes, migrations, ingestion
CLICKHOUSE_READ_ONLY_URL=http://clickhouse-reader:8123 # UI + public API reads

Critical nuance for on-premise, and here it is time to be sceptical about the usefulness of this knob in our context: on a single-node ClickHouse or on a self-managed cluster without compute separation, this variable adds nothing, because the read endpoint would be the same as the write one. It is a knob for cloud architectures with storage separated from compute. On an on-premise 4×H100 cluster with ClickHouse on one node, the real alternative is to scale ClickHouse vertically (the doc recommends ≥16 GiB of RAM for large deployments; ClickHouse scales vertically well) and to make sure every query filters by projectId and time, which is how the tables are indexed. Without a time filter, even the fattest ClickHouse suffers.

Knob 9 — Data retention: TTL in ClickHouse + lifecycle in S3

Disk is the cost that grows on its own. LLM traces carry whole inputs and outputs (sometimes prompts of tens of KB), and ClickHouse also accumulates its own system tables. The first-order lever is a retention policy that nightly deletes traces, observations, scores and media older than N days, coordinating ClickHouse and blob storage. Where the retention feature is not available, it is done by hand:

-- ClickHouse: TTL over the tracing tables
ALTER TABLE traces MODIFY TTL toDateTime(timestamp) + INTERVAL 90 DAY;
ALTER TABLE observations MODIFY TTL toDateTime(start_time) + INTERVAL 90 DAY;
ALTER TABLE scores MODIFY TTL toDateTime(timestamp) + INTERVAL 90 DAY;
ALTER TABLE event_log MODIFY TTL toDateTime(timestamp) + INTERVAL 30 DAY;

# S3/MinIO: lifecycle rule, e.g. 30 days for the raw event bucket
# WATCH OUT! Do NOT apply retention to the MEDIA bucket:
# - it breaks the files referenced in traces
# - it breaks future uploads (state is tracked by hash in Postgres)

Two operational parameters that avoid nasty surprises on large deletions:

LANGFUSE_CLICKHOUSE_DELETION_TIMEOUT_MS=600000 # default 10 min; raise it if deletions time out
# ClickHouse 25.7+: less mutation pressure on bulk deletions
CLICKHOUSE_LIGHTWEIGHT_DELETE_MODE=lightweight_update
CLICKHOUSE_USE_LIGHTWEIGHT_UPDATE=true

The mental rule: short retention for raw events (S3, 30 days is usually enough, they are recoverable/recomputable), retention by business value for the ClickHouse tables (90 days, 180, whatever compliance asks for), and never touch the media bucket with blind lifecycle rules.

Knob 10 — Hygiene of the ClickHouse system log tables (the silent disk killer)

This is the knob nobody configures and that fills the disk without showing up in any Langfuse metric, because it is not Langfuse data: they are the system tables of ClickHouse itself (trace_log, text_log, opentelemetry_span_log, asynchronous_metric_log, metric_log, latency_log). By default they have no TTL, and the query profiler writes to system.trace_log continuously. On a ClickHouse with traffic, these tables can dominate disk usage while you look for the problem in your traces. Langfuse does not read from them, so they can be trimmed without fear. Two options:

<!-- Option A — disable the ones Langfuse never reads
 (file in /etc/clickhouse-server/config.d/) -->
<clickhouse>
 <trace_log remove="1"/>
 <text_log remove="1"/>
 <opentelemetry_span_log remove="1"/>
 <asynchronous_metric_log remove="1"/>
 <metric_log remove="1"/>
 <latency_log remove="1"/>
</clickhouse>
<!-- Keep query_log, part_log and error_log: useful for debugging and small -->

-- Option B — aggressive TTL + turn off the profiler, if you want to keep them for debugging
-- (in config: query_profiler_real_time_period_ns = 0)
SET max_table_size_to_drop = 0;
TRUNCATE TABLE system.trace_log;
ALTER TABLE system.trace_log MODIFY TTL event_date + INTERVAL 7 DAY;
-- repeat for each log table to cap

To identify which table is eating the disk, the golden query:

SELECT table, formatReadableSize(sum(bytes)) AS size, sum(rows) AS rows
FROM system.parts WHERE active GROUP BY table ORDER BY sum(bytes) DESC;

If you take only one knob from this post to your first real deployment, make it this one: the difference between a ClickHouse that grows 2 GB/day of useful data and one that grows 20 GB/day of system logs nobody looks at.

Summary table of the 10 knobs

#	Knob	Variable / action	When
1	Scale Worker	replicas by CPU>50 % / `langfuse.queue.ingestion.length`	always, first
2	Separate ingestion/UI	route `/ingestion`,`/media`,`/otel*` to a dedicated replica	slow UI under load
3	Batching to ClickHouse	`LANGFUSE_INGESTION_CLICKHOUSE_WRITE_INTERVAL_MS` / `_BATCH_SIZE`	high throughput
4	Skip CH read-before-write	`LANGFUSE_SKIP_INGESTION_CLICKHOUSE_READ_MIN_PROJECT_CREATE_DATE`	non-migrated projects
5	S3 concurrency	`LANGFUSE_S3_CONCURRENT_WRITES` (def. 50)	“socket usage at capacity”
6	Redis queue sharding	`LANGFUSE__QUEUE_SHARD_COUNT` + `_CONCURRENCY` (per shard)	Redis CPU >90 %
7	Remove `FINAL` (OTel)	`LANGFUSE_SKIP_FINAL_FOR_OTEL_PROJECTS=true`	100 % OTel instrumentation
8	CH read/write split	`CLICKHOUSE_READ_ONLY_URL` (cloud/BYOC only)	compute-compute available
9	Retention + TTL	TTL in CH + S3 lifecycle + `LANGFUSE_CLICKHOUSE_DELETION_TIMEOUT_MS`	always (disk cost)
10	CH system log hygiene	`<trace_log remove="1"/>` or aggressive TTL	always (hidden disk)

How to maximise Langfuse on the example 4×H100 cluster

With the architecture and the knobs clear, this is a concrete sizing to get the most out of Langfuse on the generic reference cluster (4×H100 SXM, 320 GB VRAM, 640 GB RAM, NVMe-oF, 25/100 GbE), without stealing a single GB of VRAM from inference.

Component distribution

Langfuse is 100 % CPU/RAM/disk/network load, so its natural place is off the GPU nodes or, if it cohabits, with taints/nodeSelector confining it away from the vLLM pods. Suggested distribution:

nodo-cpu-01 (control + observability, no GPU)
├── langfuse-web-ingest ×3 (2 CPU / 4 GiB each) ← ingestion, scales with load
├── langfuse-web-ui ×2 (2 CPU / 4 GiB each) ← dashboard/read API
├── langfuse-worker ×4 (2 CPU / 4 GiB each) ← the one that scales most
├── redis/valkey ×1 (4 CPU / 4 GiB, cluster mode)
└── postgres ×1 (2 CPU / 8 GiB, replica for HA)
nodo-storage-01 (heavy state, local NVMe)
├── clickhouse ×1 (8 CPU / 32 GiB / NVMe) ← ≥16 GiB is the minimum; 32 is comfortable
└── minio (S3) ×1 (4 CPU / 8 GiB / HDD+NVMe cache)
nodo-gpu-01..02 (4×H100 SXM each) → inference ONLY
└── vLLM, embeddings, reranker, guardrails (emit spans, do not host Langfuse)

Sizing by real load

Let us put numbers on an example load. Suppose the cluster serving 300 requests/second of chat-with-RAG, where each request generates on the order of 8 spans (request, retrieval, rerank, 2× tool, guardrail in, llm, guardrail out):

$$ E = 300\,\tfrac{\text{req}}{\text{s}} \times 8\,\tfrac{\text{spans}}{\text{req}} = 2,400\ \text{events/s} $$

Against the Redis ceiling (~50,000 ops/s), $E = 2{,}400$ leaves the reception counter at ~5 % of its capacity: enormous headroom. The component to watch is the Worker. With a target of ~20 concurrency per Worker and batches of 10,000 events every ~1-2 s, 4 Workers drain 2,400 ev/s with margin; the langfuse.queue.ingestion.length metric should stay flat near zero. If it grows, knob 1 (more Workers) is the answer before any fine tuning.

Tail-sampling is the multiplier that changes the economics. If the Collector preserves 100 % of errors/high latencies but samples normal traffic at, say, 10 %, the 2,400 ev/s you store in ClickHouse drop to ~240-300 effective ev/s without losing the signal that matters. The rule: sample in the Collector, not in Langfuse. Langfuse should receive, already filtered, what deserves to be persisted. This is developed in the OTel tracing post; here it is enough to note that upstream sampling is, in effect, the knob 0 that multiplies all the others.

Storage estimate

An LLM observation with full input+output weighs, compressed in ClickHouse, on the order of 1-3 KB (ClickHouse compresses text very well, 5-10×). With sampling at 10 % over 2,400 ev/s:

$$ 240\,\tfrac{\text{ev}}{\text{s}} \times 2\,\text{KB} \times 86,400\,\tfrac{\text{s}}{\text{day}} \approx 41\ \text{GB/day (raw)} \;\xrightarrow{\text{compression}}\; \sim 5\text{–}8\ \text{GB/day in CH} $$

At 90 days of retention (knob 9), the permanent archive settles at around 500-700 GB in ClickHouse, comfortable on the NVMe of the storage node, plus the raw events in MinIO with a 30-day lifecycle. Without the system log hygiene (knob 10), add easily as much again in rubbish nobody queries. The two disk knobs together are the difference between planning storage once a year and fighting a full disk every month.

“Maximum use” checklist

Sampling in the Collector (tail: 100 % errors + N % normal), before touching anything in Langfuse.
Workers scaled by queue length via KEDA (knob 1), not fixed.
Ingestion separated from UI (knob 2) so the dashboard responds during incidents.
SKIP_FINAL_FOR_OTEL_PROJECTS enabled (knob 7) because the instrumentation is 100 % OTel.
Generous CH batching (knob 3) tuned to the throughput, accepting the loss window.
Retention + TTL + system log hygiene (knobs 9 and 10) configured on day 1, not when the disk screams.
ClickHouse with ≥16 GiB and every query filtering by projectId+time (knob 8 in its on-premise version: vertical scaling).
Langfuse isolated from the GPUs by taints/nodeSelector: not a MB of VRAM, no memory bandwidth contention with vLLM.

Traps and things that are not what they look like

“Langfuse guarantees me full traceability.” No: the design is high-performance best-effort, not a ledger. Between the HTTP 207 and the row in ClickHouse there are volatile buffers (Redis without hard persistence, the Worker’s in-memory buffer, ClickHouse’s server-side async_insert). There is a known bug where the writer discards rows with no dead-letter queue after exhausting retries. For operational observability, losing 0.01 % of spans is irrelevant. For ENS/EU AI Act audit evidence, where the trace is the proof, Langfuse should not be the only record; the regulatory audit log needs durability guarantees this pipeline does not promise. A distinction covered in the technical controls for ENS/42001/EU AI Act.

Raising ClickHouse batching “to go faster” and nothing else. Knob 3 improves throughput at the cost of enlarging the loss window and the latency of data appearing in the dashboard. Batches of 50,000 every 10 s perform wonderfully… until the Worker restarts with 50,000 events in the buffer. Tune with awareness of the trade-off, not by blind maximisation.

Putting ClickHouse on the same node as vLLM with no limits. ClickHouse is voracious with memory bandwidth during merges. Sharing a node with vLLM without resources.limits or NUMA isolation means an ingestion spike can degrade inference TTFT, exactly the original sin this whole architecture wanted to avoid. Isolate.

Forgetting the time filter in your own queries. The ClickHouse tables are indexed by projectId and time. A custom dashboard or an API query without a time filter scans the whole history and takes performance down for everyone. It is not Langfuse “being slow”: it is a badly written query.

Applying lifecycle to the media bucket. It breaks the files referenced in traces and blocks future uploads (state is tracked by hash in Postgres). The media bucket is managed only with the Langfuse retention feature, never with blind S3 rules.

Treating queue sharding as a routine optimisation. It is an advanced knob for a genuinely drowning Redis, irreversible (do not reduce shards) and with per-shard concurrency semantics that are easy to misread. In the vast majority of on-premise deployments it is not needed; if you enable it “just in case”, you complicate your life without gaining anything.

Conclusion

Langfuse v3 solved the structural problem of LLM observability, that the observer should not suffocate the observed, by moving from a monolith on Postgres to a six-service sorting centre with asynchronous ingestion. That design is what allows a cluster serving thousands of tokens per second to be instrumented end to end without the app ever waiting for a trace to be filed. But the design is a necessary condition, not a sufficient one: it performs if the right levers are tuned. Of the ten knobs, three decide almost everything in a typical on-premise deployment, scaling Workers by queue length (1), retention + TTL (9), and system log hygiene (10); the rest are refinements that show up when the load bites. And above them all lives knob 0, which is not a Langfuse one: sampling in the Collector, which decides how much reaches the pipeline before any internal setting matters. Maximising Langfuse on the 4×H100 cluster is not about squeezing its peak throughput: it is about putting it off the GPUs, feeding it already-sampled traffic, sizing the Worker by the queue, and configuring retention on day one, so that the tool that came to tell the story does not end up being the chapter about the incident.

References

Langfuse, Scaling Langfuse Deployments (doc oficial de sizing y todos los env vars de este post): https://langfuse.com/self-hosting/configuration/scaling.
Langfuse, Self-host Langfuse y Configuration via Environment Variables: https://langfuse.com/self-hosting · https://langfuse.com/self-hosting/configuration.
Langfuse, ClickHouse (self-hosted): https://langfuse.com/self-hosting/deployment/infrastructure/clickhouse.
Langfuse, From Zero to Scale: Langfuse’s Infrastructure Evolution (el porqué del rediseño v2→v3): https://langfuse.com/blog/2024-12-langfuse-v3-infrastructure-evolution.
Langfuse, Migrate v2 to v3 (self-hosted): https://langfuse.com/self-hosting/upgrade/upgrade-guides/upgrade-v2-to-v3.
ClickHouse, Langfuse and ClickHouse: A new data stack for modern LLM applications: https://clickhouse.com/blog/langfuse-and-clickhouse-a-new-data-stack-for-modern-llm-applications.
Langfuse, issue #13468 — ClickhouseWriter drops rows after max flush attempts with no DLQ (la ventana de pérdida documentada): https://github.com/langfuse/langfuse/issues/13468.
ClickHouse, TTL for tables and columns: https://clickhouse.com/docs/guides/developer/ttl.
OpenTelemetry, Semantic Conventions for Generative AI (gen_ai.*): https://opentelemetry.io/docs/specs/semconv/gen-ai/.

Function calling and tool-augmented retrieval: the detective who knows which file to ask for

Thu, 04 Jun 2026 10:00:00 +0200

This post goes deeper into the agentic reasoning mechanism that extends the RAG described in RAG with reranker and hybrid retrieval. The retriever invoked when the LLM chooses vector_search is exactly the pipeline in that article. The JSON Schema that defines each tool call is structured output applied to the tool interface. And the agent’s requests pass through the L7 inference gateway before reaching the model.

TL;DR

A naive RAG always queries the same source. Function calling gives the LLM the ability to decide which tool to invoke (vector store, SQL, web search) according to what the query actually needs. The ReAct pattern chains those invocations in a reasoned loop until there is enough evidence. A 3-iteration pipeline with Llama-3.1-70B on on-premise hardware takes ≈ 1.1 s against the ≈ 300 ms of a single-step RAG; the gain is not in speed but in queries that a naive RAG simply cannot answer. The critical eval metric is tool selection accuracy: the percentage of turns in which the model picks the right tool, measured over a synthetic eval set.

The analogy: the detective who knows which file to ask for

A detective in a novel does not go to the same filing cabinet regardless of the case that lands on his desk. When he takes a case, he reasons first: what kind of evidence do I need? If there are witnesses, he asks for the statements (vector search over unstructured documents). If there are financial transactions, he asks the bank for the accounting records (SQL over the structured database). If the suspect has recent activity that the company cannot possibly have indexed, he goes to the press archive (web search). He does not consult all three sources at once on every case: he picks the one the evidence requires, receives the result, reasons again if he needs more, and only writes the report when he has enough evidence.

A bad detective always goes to the same filing cabinet. A naive RAG is that bad detective: it vectorises the query, goes to the vector store, and returns whatever it finds even if the question was “how many contracts?”, something no PDF chunk can answer better than a COUNT(*) in SQL.

Function calling is giving the LLM the ability to reason about which source to ask for, and to invoke it in a structured way. The analogy has three edges worth keeping in mind:

The detective does not improvise the file he asks for: there is a catalogue of available sources with a description of what each one contains. The tool description in the system prompt does that job.
The detective can ask for several pieces of evidence at once if they are independent (parallel tool calling).
The detective knows when to stop: if after N rounds he reaches no conclusion, he declares that he does not have enough evidence. The agent has an iteration limit for the same reason.

What function calling is: the anatomy of a tool call

Function calling, also called tool use, is a mechanism by which the LLM, instead of generating free text as its answer, generates a structured JSON object representing a tool invocation. The system intercepts that JSON, executes the real tool, and returns the result as a message with the tool role in the conversation.

Defining tools in the system prompt

Each tool is defined through a JSON Schema that specifies name, description and parameters. This JSON Schema is exactly the same mechanism described in structured output, applied here to the tool interface:

{
 "tools": [
 {
 "name": "vector_search",
 "description": "Search internal company documents about policies, contracts and procedures. Use when the query requires unstructured text, document context or semantic similarity.",
 "parameters": {
 "type": "object",
 "properties": {
 "query": {"type": "string", "description": "Natural language search query"},
 "top_k": {"type": "integer", "default": 5}
 },
 "required": ["query"]
 }
 },
 {
 "name": "sql_query",
 "description": "Query the SQL database for structured metrics, counts, aggregations and financial data. Use when the query requires exact numbers, filters, sums or joins over structured records.",
 "parameters": {
 "type": "object",
 "properties": {
 "query": {"type": "string", "description": "Parameterized SQL query with $1, $2 placeholders"},
 "params": {"type": "array", "items": {}, "description": "Parameter values for the placeholders"}
 },
 "required": ["query"]
 }
 },
 {
 "name": "web_search",
 "description": "Search public web for real-time information, recent news or current prices. Use only when data is public and not covered by internal sources.",
 "parameters": {
 "type": "object",
 "properties": {
 "query": {"type": "string"}
 },
 "required": ["query"]
 }
 }
 ]
}

The life cycle of a tool call

When the LLM decides to invoke a tool, the message it generates in place of free text has this structure (OpenAI-compatible format, the same one vLLM supports):

{
 "role": "assistant",
 "tool_calls": [
 {
 "id": "call_01",
 "type": "function",
 "function": {
 "name": "sql_query",
 "arguments": "{\"query\": \"SELECT COUNT(*), SUM(amount) FROM contracts WHERE amount > $1 AND year = $2 AND provider_region = $3\", \"params\": [100000, 2025, \"EU\"]}"
 }
 }
 ]
}

The system executes the tool and returns:

{"role": "tool", "tool_call_id": "call_01", "content": "{\"count\": 47, \"total\": 8300000}"}

The LLM receives that message as a continuation of the conversation and decides whether it needs more information or can generate the final answer.

Support in OSS models

In 2026, native function calling support (not emulated via system prompt) is available in:

Llama 3.1 / 3.3+: native tool call format, supported in vLLM with --enable-auto-tool-choice --tool-call-parser llama3_json
Qwen 2.5+: native support with --tool-call-parser hermes
Mistral NeMo / Mistral 7B Instruct v0.3+: native format with --tool-call-parser mistral
Qwen3 (2025+): extended native support with parallel tool calling

Without --enable-auto-tool-choice, vLLM can force the tool call format via guided decoding (JSON Schema), but the ability to decide when to invoke a tool versus generating free text requires specific model training.

The ReAct pattern: Reasoning + Acting

ReAct (Yao et al., 2022) is the loop that turns an LLM with tools into an agent with reasoning. The central idea is that the model interleaves thoughts (explicit reasoning about what to do) with actions (tool invocations) and observations (results received). This loop repeats until the model reckons it has enough information to answer, or until the iteration limit is reached.

Thought → Action → Observation → Thought → Action → Observation → ... → Response

A full ReAct trace: a real production query

Query: “How many contracts above 100k€ did we sign in 2025 with EU providers, and which special conditions recur?”

This query requires two different sources: the exact numbers live in SQL, and the context of special conditions lives in the contract PDFs indexed in the vector store.

Iteration 1

Thought: The query asks for aggregated data (count and sum) with exact filters over structured fields (amount, year, region). This is a SQL query, not a semantic search.

Action: sql_query

SELECT COUNT(*) AS num_contratos, SUM(amount) AS total_eur
FROM contracts
WHERE amount > $1 AND year = $2 AND provider_region = $3

params: [100000, 2025, "EU"]

Observation: {"num_contratos": 47, "total_eur": 8300000}

Iteration 2

Thought: I have the numbers. Now I need the special conditions that recur in these contracts. That kind of information is in the free text of the PDFs, not in the structured table. I go to the vector store.

Action: vector_search

"EU provider contracts 2025 amount above 100k special conditions recurring clauses"

top_k: 5

Observation:

[chunk_1] Provider A contract — clause 8.3: late-delivery penalty of 2% per month on the invoiced amount...
[chunk_2] Provider C contract — clause 7.1: price revision linked to the euro area CPI capped at 4% per year...
[chunk_3] Provider B contract — clause 8.3: late-delivery penalty of 2% per month...

Thought: I have enough data from both sources. Clause 8.3 on late-delivery penalties appears in at least two of the retrieved contracts. I can synthesise the answer.

Response: “In 2025 you signed 47 contracts with EU providers above 100,000 €, for a cumulative total of 8.3 M€. The special condition that appears most often in the higher-value contracts is the late-delivery penalty clause (8.3 in the contract templates of Provider A and B): 2% per month on the invoiced amount. The price revision linked to the euro area CPI, capped at 4% per year (clause 7.1), also recurs.”

This trace has 2 iterations. A more complex query (comparing with the previous year, cross-referencing with payment status) would add further iterations. Common practice is to set a limit of 5–10 iterations maximum to avoid loops.

The three sources and when to use each one

The choice of source is neither arbitrary nor left solely to the LLM: the architecture defines which tools exist and how they are described. The table below summarises the selection criteria:

Characteristic	Vector store (Qdrant)	SQL / structured (PostgreSQL)	Web search
Data type	Free text, documents, PDFs	Tables with a fixed schema	Public pages, news
Structure	Unstructured	Highly structured	Semi-structured
Temporality	Static index (periodic refresh)	Real time (transactional)	Real time (crawl)
Typical latency	5–50 ms	10–200 ms	200–2,000 ms
Privacy	Internal data, full sovereignty	Internal data, full sovereignty	Public data only
Natural query	Yes (natural language → embedding)	No (parameterised SQL)	Yes
Exact aggregations	No	Yes (`COUNT`, `SUM`, `GROUP BY`)	No
When to use	Document context, semantics, PDFs	Metrics, counts, exact filters, joins	Data that does not exist internally and is public

The most important rule of thumb: if the question contains words like “how many”, “sum”, “total”, “more than X”, “in year Y” and the data is in a structured table, the right answer is sql_query. If the question asks for context, explanations, clauses, procedures or examples from documents, the answer is vector_search. If it asks for the current price of something or recent news about a third party, web_search, but only if no data sovereignty is involved.

Tool routing: how the LLM picks the right tool

The description of each tool in the system prompt is the most critical factor for routing accuracy. An LLM with good function calling capabilities can still choose badly if the descriptions are ambiguous or overlap.

Descriptions that work versus those that do not

Weak description (leads the LLM to use the wrong tool):

"search_docs" — Searches for information in the available sources.
"query_data" — Gets data from the system.

Strong description (delimits precisely when to use each one):

"vector_search" — Search internal company documents about policies, contracts and procedures.
Use when the query requires unstructured text, document context or semantic
similarity. NOT for counts, sums or exact filters.
"sql_query" — Query the SQL database for structured metrics, counts, aggregations and
financial data. Use when the query requires exact numbers, filters, sums or
joins over structured records. NOT for finding document context.

The difference lies in two elements: (1) positive use-case examples, and (2) explicit exclusions with NOT for. Both reduce the semantic overlap between tools and improve tool selection accuracy.

Parallel tool calling

When two tools are independent of each other, that is, when the result of one does not affect the query of the other, the LLM can invoke them simultaneously in the same turn:

{
 "tool_calls": [
 {"id": "call_01", "function": {"name": "sql_query", "arguments": "..."}},
 {"id": "call_02", "function": {"name": "vector_search", "arguments": "..."}}
 ]
}

The system executes both in parallel and returns both observations before the LLM’s next turn. This cuts total latency when the queries are independent: instead of 2 sequential iterations (2 × tool_latency), you pay 1 × max(sql_latency, vector_latency). For the detective example: if he needs both the accounting records and the witness statements to answer, he can ask for them at the same time.

Tool selection accuracy: the eval metric

Tool selection accuracy is the percentage of turns in which the LLM picks the right tool given a set of evaluated queries:

\[ \text{TSA} = \frac{\text{turns with correct tool chosen}}{\text{total turns with expected tool call}} \]

It is measured over a synthetic eval set built with (query, expected_tool, expected_args) triples. An example of a minimal eval set:

Query	Expected tool	Failure indicator
“How many orders in March?”	`sql_query`	LLM uses `vector_search`
“What does the returns policy say?”	`vector_search`	LLM uses `sql_query`
“What is the price of copper today?”	`web_search`	LLM uses `vector_search`
“Sum the Q3 contracts”	`sql_query`	LLM uses `vector_search`

A TSA < 0.85 in a production agent is a sign that the tool descriptions need revising before the model does. For more detail on how to build these evals, see LLM evals.

Error handling in tool calls

SQL injection via prompt

The most serious risk in tool-augmented retrieval is that the LLM generates malicious SQL, either because a user induced it via prompt injection, or because the model hallucinated a destructive query. This attack vector is covered in detail in LLM Guard, but the minimum rules on the tool side are:

Rule 1: parameterised queries always, never direct interpolation.

# NEVER this:
cursor.execute(f"SELECT * FROM contracts WHERE provider = '{llm_output}'")

# Always this:
cursor.execute("SELECT * FROM contracts WHERE provider = $1", [llm_output])

Rule 2: DB user with minimum permissions. The user the agent runs SQL as must have SELECT on the necessary tables and nothing else. No DROP, INSERT, UPDATE or DELETE.

Rule 3: table allowlist. The system validates that the query generated by the LLM only references tables on an allowlist before executing it.

Rule 4: per-query timeout. Queries that block for too long, potentially induced to DoS the database, are cancelled with a configured timeout.

Rate limits, timeouts and tool errors

When a tool fails, the error is returned to the LLM as an observation:

{"role": "tool", "tool_call_id": "call_01", "content": "{\"error\": \"timeout after 5s\", \"tool\": \"web_search\"}"}

The system prompt must instruct the LLM on what to do in this case:

If a tool returns an error or is unavailable, acknowledge the limitation in your response.
Do not retry more than once. If web_search is unavailable, state that real-time data
is not accessible at this moment and answer with available internal sources only.

This stops the agent from entering retry loops and handles graceful degradation: if web_search is unavailable, it answers with what it has in the internal sources.

Diagram of the ReAct loop with the three sources

Latency maths of the ReAct pipeline

Each iteration of the ReAct loop has three latency components:

\[ T_{\text{iter}} = \text{TTFT}_{\text{LLM}} + T_{\text{tool}} + \Delta_{\text{context}} \]

where:

TTFT_LLM: time to the LLM’s first token (dominated by the prefill of the accumulated context)
T_tool: tool execution time
Δ_context: overhead of the growing context window (each iteration adds the previous output to the context)

Reference values: Llama-3.1-70B on 4×H100 SXM (320 GB, NVLink)

With Llama-3.1-70B in FP8 on a node with 4×H100 SXM (320 GB HBM3, NVLink 900 GB/s), the typical values in production are:

Component	Value
TTFT with context < 4k tokens	≈ 150 ms
TTFT with context 8k tokens	≈ 220 ms
`sql_query` (simple query, indexed)	≈ 50 ms
`vector_search` (top-5, Qdrant in RAM)	≈ 20 ms
`web_search` (external API)	≈ 600 ms

Total latency by number of iterations

1-iteration pipeline (simple query, a single tool):

\[ T_1 = 150 + 50 = 200 \text{ ms} + \text{final synthesis} \approx 200 + 300 = 500 \text{ ms} \]

2-iteration pipeline (SQL + vector_search in sequence):

\[ T_2 = (150 + 50) + (180 + 20) + 400 = 800 \text{ ms} \]

The context in the second iteration already includes the result of the first, so TTFT rises slightly to ≈ 180 ms.

3-iteration pipeline (the most common case in complex queries):

\[ T_3 = (150 + 50) + (180 + 20) + (200 + 50) + 450 \approx 1,100 \text{ ms} \]

Parallel tool calling (SQL + vector_search in parallel, a single iteration):

\[ T_{\text{parallel}} = 150 + \max(50, 20) + 400 = 600 \text{ ms} \]

When the two queries are independent, parallel tool calling trims latency from ≈ 800 ms to ≈ 600 ms: a 25% improvement over the 2 sequential iterations case.

Comparison with a naive RAG

Configuration	Latency	Queries it can answer
Naive RAG (1 retriever, 1 step)	≈ 300 ms	Document context queries
ReAct 1 iteration (SQL)	≈ 500 ms	Structured aggregation queries
ReAct 2 iterations (SQL + vector)	≈ 800 ms	Hybrid numeric + context queries
ReAct 3 iterations	≈ 1,100 ms	Complex multi-source queries
ReAct with web_search	≈ 1,500 ms	Queries requiring real-time data

The practical conclusion: multi-hop ReAct is 3–5× slower than a single-step naive RAG. The gain is not in speed but in query coverage: a naive RAG cannot answer “how many contracts?” because that answer is in no text chunk at all. For applications with a strict latency SLO (< 500 ms), you have to work out whether the use case really needs ReAct or whether a well-configured RAG with hybrid retrieval covers 90% of the queries.

On-premise hardware for ReAct agents

A ReAct agent with Llama-3.1-70B in production has different requirements from a naive RAG, because the context grows with each iteration and prefill throughput is more critical.

Recommended configuration: 4×H100 SXM (320 GB HBM3, NVLink 900 GB/s)

Llama-3.1-70B in FP8: fits in 2×H100 (70B params × 1 byte/param ≈ 70 GB + KV cache). With 4×H100 you can serve it with tensor parallelism TP=4, cutting prefill TTFT by ≈ 2×.
Qdrant instance: it can sit on the same node (if the collection fits in RAM) or on a dedicated node. For collections < 50M vectors of 768 dims: ≈ 150 GB, which fits in the RAM of a dual-socket server.
PostgreSQL: separate node or managed instance. The agent adds no unusual load to SQL; the queries are simple and bounded by timeout.
vLLM with --enable-auto-tool-choice --tool-call-parser llama3_json --max-model-len 16384: a 16k token context comfortably covers the 5–10 turns of a ReAct pipeline.

Minimum viable configuration: 2×H100 SXM (160 GB)

Llama-3.1-70B in FP8 with TP=2. TTFT ≈ 250–300 ms for 4k token contexts (a 60–100% increase over TP=4).
Good enough for internal workloads with < 20 concurrent requests.
Not advisable for an SLO < 1 s with more than 5 concurrent users and a long context.

What we have not covered

Agentic retrieval loops with planning. ReAct is the simplest agent pattern. When a query requires decomposition into sub-tasks with dependencies, you need orchestration frameworks such as LangGraph (state graphs), smolagents (Hugging Face, agents with Python code as actions) or llama-index Agents (planning + retrieval pipeline). These introduce a planning step before the execution loop.

MCP (Model Context Protocol). Anthropic’s emerging standard, with OSS implementations, for defining tools portably across frameworks and hosts. Instead of defining each tool’s JSON Schema separately in every application, MCP centralises those definitions in an MCP server that any compatible client can discover and invoke. Adoption in 2025–2026 is fast among OSS frameworks (LangChain, smolagents, OpenWebUI).

Tool caching. If the same tool call (same arguments, same tool) is going to be invoked several times within the same context or in very similar contexts, the result can be cached. The mechanism is analogous to the semantic cache described for RAG: before executing the tool, the hash of the arguments (or its embedding, for semantic matching) is compared against a cache with a TTL. Especially valuable for sql_query with frequent queries and data that changes little.

Multi-agent. When an orchestrating agent delegates sub-tasks to specialised agents, one for SQL, one for document retrieval, one for code generation, you enter multi-agent system territory. Each sub-agent can have its own tool set and its own LLM (possibly smaller and more specialised). Coordination between agents introduces additional tracing and observability complexity.

References

Yao, S. et al. (2022). ReAct: Synergizing Reasoning and Acting in Language Models. arXiv:2210.03629. https://arxiv.org/abs/2210.03629
vLLM documentation. Tool calling. https://docs.vllm.ai/en/stable/features/tool_calling.html
Qdrant documentation. Search. https://qdrant.tech/documentation/concepts/search/
OpenAI. Function calling. https://platform.openai.com/docs/guides/function-calling
Meta AI. Llama 3.1 Model Card. https://github.com/meta-llama/llama-models/blob/main/models/llama3_1/MODEL_CARD.md
Qwen Team (Alibaba). Qwen2.5 Technical Report. arXiv:2412.15115. https://arxiv.org/abs/2412.15115
Anthropic. Model Context Protocol. https://modelcontextprotocol.io
OWASP. LLM Top 10 for Large Language Model Applications. LLM01: Prompt Injection. https://owasp.org/www-project-top-10-for-large-language-model-applications/

Ontologies and knowledge graphs in LLMOps: the Linnaean nomenclature that holds up the six pipeline stages

Wed, 03 Jun 2026 03:30:00 +0200

This post runs through the six stages of the LLMOps pipeline from a cross-cutting perspective: the common nomenclature that makes the stages share a vocabulary. It connects directly to RAG corpus curation (the corpus is curated against an ontology), embeddings (the ontology enriches the embedding with typed metadata), hybrid retrieval (the KG is a fourth channel alongside dense/sparse/multi-vector), evals (golden sets are stratified by ontological class), structured output (JSON Schemas derive from OWL/SHACL), and the three frameworks of ISO 42001, ENS and EU AI Act (each one is a control ontology).

TL;DR

The conversation about ontologies and LLMs has swung between two equally wrong positions over the last three years: either “LLMs already extract knowledge on their own, ontologies belong to the last century”, or “all of RAG has to go and we should build a knowledge graph on top”. The operational reality of mid-2026 is more sober: the ontology is not a replacement for RAG but its common nomenclature, without which the six stages of the LLMOps pipeline work with different vocabularies without knowing it. The corpus is curated without knowing which entity classes exist; the embeddings drill through documents without being enriched with typed metadata; the evals report a global accuracy that hides whole class gaps; the guardrail blocks on word lists instead of formal classification; incident response groups badly because each alert names “the affected asset” its own way; compliance cannot map its controls because ENS, ISO 42001 and the EU AI Act are three ontologies and the system has none. This post takes apart what an ontology is in practical terms (TBox and ABox, RDF and SPARQL, the four OWL 2 profiles: EL for enormous terminologies such as SNOMED, QL for OBDA, RL for rule-based reasoning, DL for the full description logic, plus SHACL for shape validation, SKOS for thesauri, JSON-LD as a viable serialisation), walks the six LLMOps stages showing where the ontology changes operations, reviews the GraphRAG field in 2026 with verifiable data (Microsoft GraphRAG v2 Oct 2025, LightRAG dual-level with incremental updates, HippoRAG 2 with Personalized PageRank, KAG on OpenSPG ontology-grounded), inventories the vertical ontologies actually deployed in production (FIBO, SNOMED CT, schema.org, IEC 81346, GS1, Wikidata, ENS Annex I-II of RD 311/2022, EU AI Act Annex III), settles the viable open source on-prem stack with its licensing caveats (Neo4j Community is GPLv3 with AGPL implications in some features, KuzuDB upstream archived Oct 2025, forks bighorn and ryugraph), describes the five LLM × ontology integration patterns and closes with seven operational traps. The rule of thumb: the knowledge graph is not the answer; shared formalised nomenclature is.

The analogy: Carl Linnaeus, 1735

In 1735 Carl von Linné published the first edition of Systema Naturae. Before Linnaeus, European naturalists had an operational problem: the same species could appear in five treatises under five different Latin names, each a polynomial description of the type “Felis cauda elongata cum maculis nigris in dorso et lateribus”, and two naturalists exchanging letters took months to realise they were arguing about the same animal. Biology was a field of lexicographic noise: impossible to compare observations, impossible to verify replication, impossible to build cumulative theory.

Linnaeus did not discover biology. What he discovered was that the field needed a common nomenclature with three properties:

Strict hierarchy. Kingdom → Phylum → Class → Order → Family → Genus → Species. Each level is a class with well-defined subclasses. A property of Felis (a carnivorous diet) is inherited automatically by Felis catus and Felis silvestris without being redeclared.
Unambiguous naming. Each species has a single binomial name (Genus + specific epithet) and a single type specimen anchored in a museum. “Felis silvestris” means exactly the same thing in Madrid, Stockholm and Calcutta.
Priority rules. If two botanists publish the same genus under different names, the first to register it validly wins. The naming convention is not debated in every paper: there is an explicit meta-level of governance.

After Linnaeus, comparative biology becomes possible. Mendel can talk about Pisum sativum and a Polish botanist knows exactly which plant to grow to replicate him. Darwin can compare Galápagos finches with finches from other islands with no confusion about what “the same kind of bird” means. The change is not one of instrumentation; the microscope had existed since Hooke (1665). The change is one of shared formal vocabulary.

An ontology in computing is exactly this:

Linnaeus (1735)	Ontology (2026)
Kingdom → … → Species hierarchy	Class hierarchy (`Person ⊑ Agent ⊑ Thing`), the TBox
Type specimen in a museum	Instance anchored with a unique IRI, the ABox
Binomial name	A unique IRI / URI per concept
Priority rules	Ontology axioms plus governance
“Felis silvestris” means the same in Madrid and Stockholm	`<http://example.org/ont/Felis_silvestris>` means the same in any system

When an LLMOps team today says “our corpus is curated, the embeddings are bge-m3 and the evals measure recall@5”, but the question “what proportion of queries about high-category ENS assets are well covered?” has no answer, because no formal class “high-category ENS asset” exists in the system, the problem is pre-Linnaean: the field has not yet given itself the nomenclature that makes each stage comparable.

The ontology runs through the six stages as shared vocabulary. Without it, each stage has its own definition of "customer", "sensitive document" or "incident".

What an ontology is in operational terms

The word “ontology” carries an unavoidable philosophical family resemblance (Aristotle, Kant’s categories, Quine) that confuses at first. In LLM infrastructure it does not matter: an ontology is a directed graph with types, formally described, over which you can reason, validate and query. What matters are six practical concepts.

TBox and ABox

The distinction used every day. The TBox (from terminology) is the schema: classes, subclass hierarchy, properties, axioms. The ABox (from assertions) is the instances.

# TBox — schema
:Person rdfs:subClassOf :Agent .
:Employee rdfs:subClassOf :Person .
:worksFor rdfs:domain :Employee ; rdfs:range :Organization .

# ABox — instances
:alice a :Employee .
:alice :worksFor :acme .

A reasoner checks that the ABox is consistent with the TBox: if you declare :alice :worksFor :acme but :alice is not an :Employee, the reasoner flags the inconsistency. That is the lever: automatic validation of knowledge, which no system based on dense embeddings alone can provide.

RDF and the unit of information

The atomic unit of the Semantic Web is the RDF triple (subject, predicate, object). Every piece of data is expressed as a collection of triples. This gives the paradigm’s most useful operational property: two graphs merge trivially by union. If your system indexes the medical corpus with SNOMED CT and the legal corpus with FIBO, both in RDF, merging them for a query that crosses both domains is literally g1 ∪ g2. In a property graph (Neo4j) this takes more surgery.

The four OWL 2 profiles

People new to the field assume OWL is one thing. It is four profiles with different trade-offs, all W3C Recommendations:

Profile	Expressiveness	Reasoning cost	Use cases
OWL 2 EL	restricted (subclass, intersection, properties)	polynomial in ontology size	enormous terminologies — SNOMED CT (350k+ concepts)
OWL 2 QL	subset that maps to SQL/UCQ	LOGSPACE in data	OBDA (ontology-based data access) over relational DBs
OWL 2 RL	subset implementable as rules (Datalog)	scalable, without full DL	production reasoning with rule engines
OWL 2 DL	full SROIQ (the “full ontology”)	decidable but NEXPTIME in the worst case	academic ontologies, deep validation

Operational rule: if your team is not going to read a description logics paper every month, do not use OWL 2 DL. Almost all the value is in EL/QL/RL. For large medical terminologies, EL. To reason over existing relational data, QL. For business rules, RL.

SHACL — the validation you actually operate

OWL does reasoning (“given these axioms, what can be deduced?”). SHACL does validation (“given this concrete graph, does it satisfy these shapes?”). In production, SHACL wins because its semantics are closer to the type checking a developer already understands:

:PersonShape a sh:NodeShape ;
 sh:targetClass :Person ;
 sh:property [
 sh:path :nombre ;
 sh:minCount 1 ;
 sh:datatype xsd:string ;
 ] ,
 [
 sh:path :nif ;
 sh:pattern "^[0-9]{8}[A-Z]$" ;
 ] .

Validating an incoming graph against this shape catches :alice :nombre 42 (wrong type), :alice :nif "12345678X9" (wrong format) or :alice a :Person with no name (min count violated). It is JSON Schema for graphs, conceptually. The SHACL 1.2 spec is a W3C draft from 2025; SHACL 1.0 has been in production since 2017.

SKOS — the lightweight thesaurus

Not all knowledge deserves OWL. For controlled vocabularies (thesauri, taxonomies, glossaries) there is SKOS:

:Mamifero a skos:Concept ;
 skos:prefLabel "Mamífero"@es , "Mammal"@en ;
 skos:broader :Animal ;
 skos:narrower :Felino , :Canido .

SKOS does not express formal axioms: skos:broader is not rdfs:subClassOf. It serves to classify content with no pretension of reasoning, which covers 80% of corporate cases. Start with SKOS: most enterprise “ontologies” are in fact thesauri that were promoted to OWL out of fashion and drag unnecessary complexity along.

JSON-LD and SPARQL — the practical surfaces

JSON-LD 1.1 (W3C Rec 2020) is the serialisation that actually gets used in real systems: ordinary JSON with an @context field mapping the keys to IRIs. The schema.org microformat on web pages is JSON-LD. For an LLMOps team, JSON-LD is the natural exchange format with tools and APIs.

SPARQL 1.1 (W3C Rec 2013; 1.2 in draft 2025) is SQL for graphs:

SELECT ?empleado ?empresa WHERE {
 ?empleado a :Employee ;
 :worksFor ?empresa ;
 :pais "España" .
 ?empresa :sector "fintech" .
}

Every modern triple store speaks it. The federation features let a single query touch several endpoints: SNOMED CT plus your own corporate ontology.

Why it matters for an LLM in production

The romantic promise of 2023-2024 was: “now that we have LLMs we do not need ontologies; the model understands natural language and extracts knowledge”. The operational reality of mid-2026 is more nuanced and rests on four observations anyone with a RAG in production has already made:

The LLM has semantic memory but no declared schema. If you ask “which entities of type Person appear in this document?”, it answers something reasonable. If you ask “which people appear and which of them are employees of the customer?”, the answer depends on how the model interprets “employee of the customer” in that context. Without an external schema saying “Employee is a subclass of Person and relates to Organization via worksFor”, coherence between two calls to the same LLM is not guaranteed.
Quality varies by domain without the system knowing why. Your RAG has a global accuracy of 78% but fails systematically on queries about derivative financial instruments. Since you have no formal classification of queries by category, the problem is invisible until a customer complains.
Compliance demands formal nomenclature. ENS classifies assets along five dimensions (Confidentiality, Integrity, Availability, Authenticity, Traceability) with three levels each. The EU AI Act lists eight high-risk areas in Annex III. Without a formal mapping between your assets and those categories, you cannot audit what you cannot name. The auditor asks “which corpus chunks touch specially protected personal data?” and your system does not have that column.
Interoperability between components demands types. Your retrieval returns “relevant chunks”. Your reranker reorders them. Your guardrail filters out the sensitive ones. If each component has its own definition of what a “sensitive chunk” is, the chain breaks at every interface. A shared ontology is the pipeline’s type contract.

The operational consequence: the ontology does not replace RAG. It types it. It makes it auditable, comparable and debuggable. The right question is not “do I need a knowledge graph?” but “at which pipeline stages do I gain if I introduce a shared formal nomenclature?”.

The six LLMOps stages × ontology

Let us walk the six pipeline stages asking what changes in each one when there is an ontology. This is the spine of the post: the lever is not “install Neo4j”, it is introducing types where there was plain text before.

Stage 1 — Data

Corpus curation becomes ontology-driven curation:

Each chunk is not just “text + embedding”, it also carries chunk:tipoDocumento, chunk:nivelClasificacion, chunk:categoriaENS, chunk:contienePII.
These types come from an explicit corporate ontology, not from ad-hoc strings written by whichever data engineer was on the morning shift.
Rule 4 of corpus curation, anti-contamination, benefits: chunks in the golden eval set carry dataset:goldenEval=true declared as a triple; any reindexing that filters on goldenEval=true becomes trivial.
The PII detector stops being a regular expression and becomes a classifier against the personal data categories thesaurus: identifier, contact, financial, health, biometric. The chunk:pii column is no longer boolean but a list of SKOS categories.

# Ingestion with ontological typing
chunk = {
 "@context": "https://ontology.fibercli.es/v1/context.jsonld",
 "@id": f"chunk:{uuid4()}",
 "@type": "Chunk",
 "tipoDocumento": "ContratoComercial",
 "nivelClasificacion": "ConfidencialMedio",
 "categoriaENS": ["Disponibilidad-M", "Confidencialidad-A"],
 "contienePII": ["IdentificadorFiscal", "Contacto"],
 "embedding": [...],
 "text": "...",
}

Under the JSON-LD context, all those keys resolve to IRIs and are queryable via SPARQL.

Stage 2 — Train / Adapt

Continuous fine-tuning and retrain gain two levers:

Datasets stratified by class. When production feedback turns into a training dataset, each example comes labelled with the ontological class of the incident that produced it. This lets you sample n examples per class instead of n examples globally, which corrects the model’s coverage gaps.
Ontology-guided synthetic generation. For classes with few examples in the real corpus, synthetic data is generated against the schema: “generate 50 questions about FIBO:DerivativeInstrument that a trader might ask”. The output goes through structured output validated against the schema’s SHACL shape before entering the dataset.

Stage 3 — Eval

The evals layer changes more than any other. Without an ontology, the eval reports a global accuracy that hides everything:

accuracy = 0.78

With an ontology, it reports a coverage matrix by class:

 accuracy n_queries covered_in_corpus
ContratoComercial 0.82 142 yes
EmpleadoENS-Alto 0.31 18 partial
DerivadoFinanciero 0.74 67 yes
SOAP_3.0_Endpoint 0.05 9 no

The EmpleadoENS-Alto row with accuracy 0.31 exposes a problem that is invisible without stratification. The SOAP_3.0_Endpoint row with accuracy 0.05 and covered_in_corpus=no shows that the class has no corpus at all: before touching the model you have to touch ingestion. A single metric hides; a per-class metric drives action.

This is the rule that LLM-as-judge and evals should always implement wherever an ontology exists: the golden eval set is labelled by class and every metric is reported stratified.

Stage 4 — Deploy

In the LLM inference router the ontology enables:

Semantic routing by class. Queries that, after a first classification, fall under FIBO:Securities are routed to the adapter fine-tuned on finance; queries under SNOMED:ClinicalFinding go to the medical adapter. Without an ontology, this routing rests on ad-hoc classifiers or fragile lexical heuristics.
Typed tool calling. The tools the agent can invoke declare their arguments against ontology classes. The cliente_id argument is not a string; it is a :ClienteCorporativo. Before running the tool, the arguments are validated with SHACL. This sharply cuts errors from badly populated arguments.
Feature flags with a class. The canary becomes “the new model receives 10% of queries in class X” instead of an undifferentiated 10%: it isolates the blast radius.

Stage 5 — Observe

This is where the absence of an ontology hurts fastest in operations. The incident response runbooks require:

A formal incident taxonomy. IncidenteSeguridad ⊑ Incidente, IncidenteIA ⊑ Incidente, FugaDatos ⊑ IncidenteSeguridad. Without this taxonomy, the five events from last month labelled “model issue”, “data drift”, “pii leak”, “prompt injection” and “hallucination” are neither groupable nor comparable. Keep + Kafka apply deduplication against that taxonomy.
Typed lineage in the KG. GPU observability plus tracing emits spans with attributes. If those attributes are typed against the ontology (span.input.classification = :ConfidencialMedio), finding every request that touched class ConfidencialAlto in the last hour is a trivial SPARQL query; without an ontology, it is a grep over unstructured logs.

Stage 6 — Govern

Where the ontology becomes unavoidable. Every regulatory framework is an ontology:

ENS RD 311/2022 Annex I: defines five dimensions (C, I, A, A, T) × three levels (Low, Medium, High). It is an asset classification scheme. Annex II lists 73 control measures with an organisational / operational / protection hierarchy. The ENS technical controls map each control to stack components, and that mapping is a relational ontology.
ISO 42001 Annex A: lists grouped controls (A.5 policies, A.6 internal organisation, A.7 resources for AI, A.8 assessment, A.9 operation). The AIMS over an on-premise LLM formalises them.
EU AI Act Annex III: eight high-risk areas. The technical file mappings are a translation from the legal ontology to the system’s technical ontology.

Without an ontology mapping your inventory of assets, datasets, models and endpoints to the classes of those three frameworks, compliance is manual, reactive and breaks with every change to the stack. With the ontology, a model change automatically triggers which controls are affected.

The GraphRAG field in 2026

GraphRAG is the generic name for a family of techniques that build a knowledge graph from a corpus and use it as an additional retrieval layer complementary to the dense / sparse / multi-vector we saw in embeddings. The motivation is that some queries (“what are the dominant themes in this corpus”, “which entities appear connected to customer X in the last six months”) are not answered well by cosine similarity between vectors.

Microsoft GraphRAG

microsoft/graphrag (July 2024, v1.0 Dec 2024, v2.x Oct 2025; any reference to v3 needs checking against GitHub releases before citing). The canonical pipeline:

Extraction. An LLM reads the corpus chunk by chunk and extracts entities and relations, so the TBox emerges from the data instead of being declared.
Graph construction. The extracted entities are disambiguated, merged and connected through the relations.
Community detection with the Leiden algorithm. The graph is partitioned into hierarchical communities.
Per-community summaries. For each community, the LLM generates a summary.
Local vs global search. Local: neighbourhood traversal for queries about specific entities. Global: map-reduce over community summaries for thematic queries.

The price: building the KG costs on the order of 5-20× more tokens than an embedding pass over the same corpus. For a corpus of 1 million chunks with bge-m3 embeddings (a day of compute on an RTX 4090), a pure GraphRAG typically requires 1-3 weeks of LLM-extractor compute (Qwen2.5-72B or similar). The LazyGraphRAG variant (mid-2025) defers summary generation to query time and cuts the construction cost by an order of magnitude.

LightRAG

HKUDS/LightRAG (HKU, arXiv:2410.05779, October 2024, EMNLP 2025). Practical improvements over canonical GraphRAG:

Dual-level retrieval. Each query generates both low-level keywords (specific entities) and high-level keywords (themes). The system searches by both and fuses the results. It captures factual and thematic questions in the same pipeline.
Incremental updates. Insertion of new chunks without rebuilding the whole graph. Canonical GraphRAG requires periodic rebuilds.
Reported cost: comparatively cheaper than GraphRAG for serving similar queries.

It is the operationally most reasonable GraphRAG when the corpus mutates.

HippoRAG 2

OSU-NLP-Group, arXiv:2502.14802 (Feb 2025; the original HippoRAG at NeurIPS'24). Inspired by the hippocampal indexing model of human memory:

It builds an open KG and also keeps the original chunks.
For each query it extracts entities and runs Personalized PageRank over the graph seeded by those entities: the PageRank “marks” the relevant nodes and, transitively, the associated chunks.
Reported +7% on associative memory tasks over SOTA embedders, with significantly lower indexing cost than GraphRAG, RAPTOR and LightRAG.

It is the most efficient GraphRAG for corpora where “which chunks are relevant to which entities” matters more than “what is the semantic structure of the corpus”.

KAG / OpenSPG

Ant Group + OpenKG, arXiv:2409.13731 (Sep 2024). The key difference from the others: KAG is ontology-grounded. It does not let the LLM invent the TBox; the TBox is declared by the domain (FIBO, SNOMED, corporate ontology) and the LLM only populates the ABox in accordance with that schema. Four pillars:

LLM-friendly representation — the schema is exposed in a format the LLM can consume as context.
Mutual index between the KG and chunks — each KG node links to the chunks it appears in.
Hybrid logical-formal reasoning — it combines the LLM with a declarative rule engine.
Semantic alignment — entity disambiguation against the ontological catalogue.

Reported +19.6% F1 on 2WikiMultiHopQA, +33.5% on HotpotQA over the RAG baseline. Deployed in Ant’s e-government and e-health Q&A in production.

KAG is the GraphRAG that works when the domain has a stable ontology (finance, health, government). Canonical GraphRAG wins when the corpus is exploratory and no prior TBox exists.

Others in the landscape

nano-GraphRAG: a lightweight Python port of GraphRAG; ideal for prototypes.
Think-on-Graph (ToG) / GraphReader: agents that plan hop traversals over the KG instead of single-shot retrieval. Better at multi-hop QA.
Neo4j LLM Graph Builder plus LangChain integration: the path of least resistance for companies already running Neo4j.

Vertical ontologies that really get used in production

Three ontologies cover 90% of vertical cases in mid-2026:

FIBO — Financial Industry Business Ontology

EDM Council + OMG, MIT license, OWL DL. The Q1/2026 production release contains 2,446 classes spread across Foundations, Business Entities, Securities, Derivatives, Loans and so on. Used in production for:

KYC entity resolution: disambiguating legal organisations (fibo-be-le-fbo:FormalBusinessOrganization).
Classification of financial instruments (fibo-sec-sec-bsk:Basket, fibo-der-drc-cds:CreditDefaultSwap).
Regulatory reporting: mapping fields against the canonical schema.

For a corporate RAG in finance, FIBO is the type schema any extraction has to satisfy. Without FIBO, two chunks that talk about a “swap” might mean an interest rate swap or a currency one.

SNOMED CT

IHTSDO/SNOMED International. Monthly releases (the May 2026 International Edition published on 15 May). Roughly 350,000+ active concepts in OWL 2 EL. Free licence in member countries (Spain is a member through the CSI / Ministry of Health), commercial outside. In production:

Assisted clinical coding: the LLM proposes SNOMED codes and the system validates them against the ontology.
Cross-lingual search in records: Diabetes mellitus type 2 and Diabetes mellitus tipo 2 resolve to the same concept (73211009).
HIPAA / GDPR health compliance: traceability of what type of clinical data each component handles.

schema.org

CC-BY-SA, ~800 types, native JSON-LD. The ontology of the web. Used in any RAG over public crawls to type Product, Article, Person, Organization from the microformats the corpus already carries embedded.

The others worth keeping on the radar

Ontology	Domain	Licence	When to use it
IEC 81346	industrial systems (designation =K1-Q1)	IEC proprietary	CMDB-as-graph, industrial plant
GS1	supply chain (GTIN, GLN, SSCC)	membership; free web vocab	EUDR traceability, retail
NIEM	US gov interoperability	CC0	gov-to-gov integration
Wikidata	universal KB (~115M items)	CC0	universal entity linking
ENS RD 311/2022 Annex I-II	ESP public sector security	BOE public	asset classification in any ENS deployment
EU AI Act Annex III	8 high-risk areas	EU regulation	EU compliance tagging

For a Spanish public sector customer with AI systems, the minimum ontology worth formalising is the union of ENS Annex I + EU AI Act Annex III + ISO 42001 Annex A. That mapping is generated once, kept as a versioned artefact in the AI governance repo and linked from the lineage of every deployed model.

Open source on-prem stack 2026

The implementation landscape splits into RDF triple stores, property graphs and auxiliary tooling.

RDF / SPARQL triple stores

Stack	Licence	Operational notes
Apache Jena Fuseki	Apache 2.0	The open reference. TDB2 storage. Quarterly releases. The reasonable default.
Eclipse RDF4J	EDL/BSD-like	Java framework plus server (Sesame-derived). Mature.
Virtuoso Open Source	GPLv2	High performance. The Community edition does not include clustering.
Ontotext GraphDB Free	proprietary EULA, free up to 2 concurrent queries	Strong OWL 2 RL reasoning. Operational cap on concurrency.
Stardog	proprietary	No genuine free production tier in 2026, only developer.
Blazegraph	discontinued	Wikidata is migrating to Qlever / others. Do not start a new project on it.

Property graphs (Cypher / Gremlin)

Stack	Licence	Operational notes
Neo4j Community Edition	GPLv3 (with a historical Commons Clause on some artefacts); Enterprise closed	Native vector index since 5.11. Cypher 25 adds the `SEARCH` clause. Cypher AI procedures (Dec 2025) integrate LLM calls and embedding generation into the query. AGPL implication: if you redistribute a SaaS exposing Neo4j Community functionality it may require source disclosure — check with legal.
Memgraph	BSL → Apache after 4 years	In-memory, Cypher. Faster than Neo4j for query-intensive workloads.
NebulaGraph	Apache 2.0	Distributed. For large sizes.
ArangoDB	Apache 2.0 (Community); features moved to Enterprise post-3.12	Multi-model (graph + document).
KuzuDB	MIT	Kùzu Inc. archived the upstream repo in Oct 2025. Community forks: `bighorn` (Kineviz), `ryugraph`. Treat upstream as unmaintained.

Hybrid vector + graph

Neo4j 5.x with native HNSW: the vector as a node property, search from inside Cypher. The most integrated option.
Memgraph + pgvector: two stacks, two operating points.
Qdrant with a graph payload: not a real graph, but it allows basic k-hop style filters over the payload.

Editors and tooling

Protégé (Stanford, BSD): the de facto ontology editor. A suite with the HermiT, Pellet and ELK reasoners.
TopBraid Composer: commercial; useful if it is already in the organisation.
Atomgraph: LGPL web editor.

Building the KG with an LLM

GLiNER / GLiREL (Apache 2.0): zero-shot NER and relation extraction. Far cheaper than an LLM-extractor (10-100× fewer tokens).
REBEL (MIT): joint entity + relation extraction based on BART. SOTA for years, today beaten by LLM-extractors but still reasonable as a baseline.
LLM-extractor with structured output: vLLM + XGrammar or Outlines enforcing a JSON Schema derived from SHACL. XGrammar has been the default backend in vLLM / SGLang / TensorRT-LLM since March 2026, with <40 µs/token of overhead.

SPARQL clients

rdflib (Python, BSD), the Apache Jena CLI, Comunica (MIT, JS, native SPARQL federation).

Five LLM × ontology integration patterns

Almost everything useful fits into five repeatable patterns.

1. Schema-guided extraction

The LLM emits JSON conforming to a schema derived from the ontology, validated in the decoder with structured output. The output is typed ABox ready to insert as triples:

schema = derive_json_schema_from_shacl("PersonShape.ttl")
# The LLM can only emit tokens that keep the output valid.
extracted = llm.generate(prompt=document, schema=schema)
graph.add_triples(jsonld_to_rdf(extracted))

Cost: practically zero overhead per token with XGrammar; effective elimination of “outputs that do not validate”.

2. Text-to-SPARQL with a semantic firewall

The LLM generates SPARQL; a semantic firewall validates every predicate and class against the TBox before running the query:

sparql_text = llm.generate(prompt=user_query, context=ontology_summary)
query = parse(sparql_text)
for predicate in query.predicates:
 if predicate not in ontology.declared_predicates:
 raise UnknownPredicate(predicate)
result = endpoint.execute(query)

It catches the classic pattern of the LLM inventing a plausible predicate that does not exist in the ontology, before touching the triple store.

3. Hybrid dense + sparse + KG retrieval with RRF

The reranker and hybrid retrieval is extended with a fourth channel: traversal in the KG seeded by the entities extracted from the query. The rankings of the four channels are fused with Reciprocal Rank Fusion:

\text{RRF}(d) = \sum_{c \in \{\text{dense}, \text{sparse}, \text{colbert}, \text{kg}\}} \frac{1}{k + \text{rank}_c(d)}

with k=60 typical. The KG channel covers exactly the queries that break the other three: queries with named entities that dense misreads or that appear rarely in the corpus.

4. Reranking by graph distance

Among the candidates from the first retrieval committee, chunks whose entities lie within k hops in the KG of the query entities are preferred. A practical implementation: add a graph_distance score and fuse it into the reranker:

def graph_distance_score(chunk, query_entities):
 chunk_entities = chunk["entities"]
 distances = [
 shortest_path_length(kg, qe, ce)
 for qe in query_entities for ce in chunk_entities
 ]
 return 1 / (1 + min(distances))

5. Typed tool calling + stratified evals

Tools declare their arguments as ontology classes. Before invocation, the arguments go through SHACL validation. This avoids the classic bug of the agent calling buscar_cliente(cliente_id="the customer who complained yesterday"), a free string where an IRI was expected.

Stratified evals by rdf:type or skos:Concept: each golden set query carries its ontological class as a label, the metrics are reported per class, and global accuracy is complemented by per-class coverage. It is the mechanism evals recommends and the ontology makes operational.

Implications for on-premise inference

The triple store or property graph does not eat GPU: it runs on CPU plus NVMe. What does compete for GPU is the LLM-extractor that builds and maintains the KG.

On the RTX 4090 (24 GB)

A reasonable setup for a PoC and small sites:

GPU 24 GB ┐
 ├─ TEI bge-m3 (dense + sparse + colbert) │ ~6 GB VRAM
 ├─ vLLM Qwen2.5-7B-Instruct AWQ Q4 (main LLM) │ ~8 GB VRAM
 └─ Occasional load: vLLM Qwen2.5-7B-Instruct for overnight extraction │ shares VRAM in another window
CPU/RAM ┐
 ├─ Apache Jena Fuseki (TBox + ABox of the corporate KG) │ ~2 GB RAM per M triples
 ├─ Qdrant (dense + sparse + colbert) │ ~3 GB RAM per M chunks
 └─ GLiNER + REBEL for fast batch extraction │ CPU-only

For corpora of up to a few million chunks, an RTX 4090 does the job by combining GLiNER/REBEL on CPU for bulk extraction (cheap but less accurate) and the LLM on GPU for critical cases.

On the 4×H100 80 GB cluster

H100 #1 (80 GB) ── vLLM Qwen3-72B-Instruct AWQ + Qwen2.5-7B speculative │ main LLM
H100 #2 (80 GB) ── vLLM gte-Qwen2-7B-instruct (embedding 32k ctx) │ large embedder
H100 #3 (80 GB) ── vLLM Qwen2.5-32B-Instruct (dedicated KG extractor) │ KG construction + maintenance
H100 #4 (80 GB) ── Hold-out for canary and offline evals │ see the canary post

Apache Jena Fuseki cluster (3 CPU nodes + NVMe RAID)
 ├─ Corporate ontology (TBox)
 ├─ ABox (hundreds of millions of triples)
 └─ FIBO / ENS / EU AI Act as named graphs

Qdrant cluster (3 CPU nodes + NVMe)
 ├─ Chunks indexed with triples in the payload
 └─ Lineage towards KG nodes

The H100 dedicated to the KG extractor is the real price of the GraphRAG approach. If the corpus is stable, that H100 can go to offline evals or speculative decoding. If the corpus mutates daily, it is busy keeping the graph up to date online.

The seven operational traps

Promoting SKOS to OWL DL out of academic ego. Most “corporate ontologies” are taxonomies that need no description logic reasoning. A SKOS with skos:broader/skos:narrower and skos:prefLabel per language covers 80% of cases. OWL DL only makes sense when there are consistency axioms the reasoner has to verify. Start with SKOS, move up to OWL EL/RL if you need to, avoid OWL DL unless the need is proven.
Building a KG over the whole corpus. Canonical GraphRAG applied to 100 million chunks costs as much as training a small model. The correct alternative is HippoRAG 2 / LightRAG / KAG depending on the case, or GraphRAG over the critical subset of the corpus only. The rule: if the construction cost exceeds the annual cost of serving the model, you have picked the wrong tool.
A TBox created by an LLM with no governance. Microsoft GraphRAG generates an emergent TBox from the data. For an exploratory corpus that works; for a regulated domain (finance, health, government) the TBox is not discovered, it is declared: FIBO, SNOMED, ENS. KAG is the right choice in those cases.
Forgetting KG maintenance when the corpus changes. New chunks introduce new entities. Without an entity reconciliation process (disambiguation, merging), the graph accumulates duplicates of the same entity under different IRIs and quality collapses quietly within six months. LightRAG has primitives for this; canonical GraphRAG requires periodic rebuilds.
A JSON Schema out of sync with the SHACL. If the ontology lives in RDF/SHACL and the structured outputs come from a hand-written JSON Schema, they drift apart. The right approach is to generate the JSON Schema from the SHACL with tools such as shacl-to-json-schema and regenerate it in CI every time the shape changes.
Neo4j Community licensed badly. GPLv3 means any modification you distribute has to be released under the same licence. If you are going to redistribute a product embedding Neo4j Community, check with legal or use an alternative with a more permissive licence (Memgraph BSL, Apache Jena for RDF, the Kùzu fork bighorn).
A compliance ontology not linked to the technical stack. Your ENS / ISO 42001 / EU AI Act mapping lives in a spreadsheet owned by the governance team. Your inventory of models, datasets and endpoints lives in another system. With no formal link between them, no stack change triggers the corresponding compliance review. The mapping goes into the graph, not into the spreadsheet.

Conclusion

An ontology is not an alternative to RAG; it is the nomenclature that makes its pieces comparable. Without it, the corpus is curated with ad-hoc categories, the embeddings drill through documents without being enriched, the evals measure the mean instead of the variance by class, the guardrail blocks on lists instead of types, incident response groups badly because each alert names things its own way, and compliance is a spreadsheet out of sync with the system. The six LLMOps stages are all better when they share a vocabulary, and sharing a vocabulary means formalising a small corporate ontology, aligned with the relevant vertical frameworks (FIBO, SNOMED, schema.org, ENS, EU AI Act), serialised in JSON-LD so the code consumes it without friction, validated with SHACL at every interface and queried with SPARQL when reasoning is needed. GraphRAG in its 2026 variants (Microsoft v2, LightRAG, HippoRAG 2, KAG) is a complementary lever, not the main course: the main course is the shared formal nomenclature. The rest, Neo4j vs Jena, OWL DL vs SKOS, GLiNER vs LLM-extractor, are technical decisions that resolve better once there is clarity about which nomenclature is needed. Linnaeus discovered this in 1735 and biology has not gone back; the LLM field is discovering it in 2026 and will not go back either.

References

W3C. RDF 1.1 Concepts and Abstract Syntax. https://www.w3.org/TR/rdf11-concepts/
W3C. OWL 2 Profiles (EL, QL, RL, DL). https://www.w3.org/TR/owl2-profiles/
W3C. SHACL — Shapes Constraint Language. https://www.w3.org/TR/shacl/
W3C. SKOS Reference. https://www.w3.org/TR/skos-reference/
W3C. JSON-LD 1.1. https://www.w3.org/TR/json-ld11/
W3C. SPARQL 1.1 Query Language. https://www.w3.org/TR/sparql11-query/
Edge et al. From Local to Global: A Graph RAG Approach to Query-Focused Summarization. Microsoft Research, 2024. https://arxiv.org/abs/2404.16130
Microsoft GraphRAG. https://github.com/microsoft/graphrag
Guo et al. LightRAG: Simple and Fast Retrieval-Augmented Generation. arXiv:2410.05779, 2024. https://arxiv.org/abs/2410.05779
Gutiérrez et al. HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models. NeurIPS 2024. https://arxiv.org/abs/2405.14831
Gutiérrez et al. From RAG to Memory: Non-Parametric Continual Learning for Large Language Models (HippoRAG 2). arXiv:2502.14802, 2025. https://arxiv.org/abs/2502.14802
Liang et al. KAG: Boosting LLMs in Professional Domains via Knowledge Augmented Generation. arXiv:2409.13731, 2024. https://arxiv.org/abs/2409.13731
OpenSPG / KAG. https://github.com/OpenSPG/openspg
EDM Council. Financial Industry Business Ontology (FIBO). https://spec.edmcouncil.org/fibo/
SNOMED International. https://www.snomed.org/
schema.org. https://schema.org/
Real Decreto 311/2022, de 3 de mayo, por el que se regula el Esquema Nacional de Seguridad. BOE-A-2022-7191. https://www.boe.es/eli/es/rd/2022/05/03/311
Reglamento (UE) 2024/1689 (EU AI Act). https://eur-lex.europa.eu/eli/reg/2024/1689
ISO/IEC 42001:2023 — Artificial Intelligence Management System. https://www.iso.org/standard/81230.html
Apache Jena. https://jena.apache.org/
Neo4j Cypher and AI procedures. https://neo4j.com/docs/
Protégé. https://protege.stanford.edu/
GLiNER. https://github.com/urchade/GLiNER
REBEL. https://github.com/Babelscape/rebel

Embeddings in 2026: the three families (dense, sparse, multi-vector), the model zoo and the decision that matters in production

Wed, 03 Jun 2026 03:00:00 +0200

This post opens the data sub-saga inside the six-stage LLMOps pipeline by going into the piece that holds up three-layer retrieval: the embedder. If the librarian of curation decided what enters the index and the faculty committee decided what reaches the model’s face, this post looks at the one in the middle: the cartographer who draws the map you search over.

TL;DR

The conversation about embeddings has been simplified in production to the point where “which embedder do you use” always gets the same answer: OpenAI text-embedding-3-large in the demo, bge-m3 in the “sovereign-ready” version. That simplification hides the fact that an embedder is three different models at once, dense single-vector, learned sparse (SPLADE) and multi-vector late-interaction (ColBERT), and that in 2026 the leading models do not compete within the same family: gte-Qwen2-7B-instruct and NV-Embed-v2 break MTEB in dense single-vector, SPLADE-v3 and the sparse head of bge-m3 dominate the learned lexical descriptor, Jina-ColBERT-v2 and ColNomic-7B are the strongest in multilingual multi-vector, and Snowflake Arctic Embed L 2.0 has slipped in as the small multilingual favourite with decent Matryoshka. This post takes apart the three families with their maths (InfoNCE with τ, MaxSim, SPLADE’s FLOPS regularization, Matryoshka Representation Learning), goes through the open source model zoo with the dimension, licence and niche of each one, poses the specific problem of multilingual Spanish, which cuts the list of viable models to fewer than six, gives the real storage cost per million chunks with int8 / binary / TurboQuant, describes how they are served on-premise with TEI, Infinity and vLLM --task embed, sets the minimum hardware at an RTX 4090 and the good hardware at a 4×H100 cluster, lists the seven operational traps that sink quality without warning (corpus drift, forgotten normalisation, badly chosen Matryoshka dimension, missing hard negatives, chat template slipped into the embedder, tokenizer drift, MTEB overfit) and closes with a license-clean stack for sovereign production.

The analogy: three librarians cataloguing the same book

A technical library receives a new book and, before putting it on the shelves, has to generate a searchable identifier for it. Three librarians with three different trades live in the library, and all three catalogue the same book at once:

Librarian A, the thematic one. He reads the whole book and gives it a single rich RFID tag. That tag is a vector of 1,024 numbers where each coordinate encodes a latent semantic axis (one the librarian never verbalises: he has learned it by reading a hundred million previous books). Two books about Kubernetes in production will end up with RFID tags very close together in the space even if one talks about Linkerd and the other about Cilium, because they share thematic axes. To search, you compare the question’s tag with the book’s and return the nearest ones by cosine. It is fast, it scales to millions, and it loses fine nuance. This is the dense single-vector embedder: bge-m3 in dense mode, gte-Qwen2-7B-instruct, Snowflake Arctic Embed L 2.0, multilingual-e5-large-instruct.

Librarian B, the lexical one. He summarises nothing. What he does is write on a card the weighted list of relevant terms from the book, expanded with synonyms from the field. The Kubernetes book carries on its card “kubernetes 4.2, linkerd 3.8, cilium 3.7, service-mesh 4.1, sidecar 3.2, mtls 2.9, ebpf 2.6, k8s 4.0…”, each term with a weight. The trick is that the model does the expansion: if your question says “service mesh” and the original book only said “Linkerd”, librarian B’s card did record “service-mesh 4.1” because he understood the relationship. To search, you intersect the question’s card with the book’s the old-fashioned way: inverted index, posting lists. It is decisive when the reader writes few, very specific words (product names, errors, jargon). This is the learned sparse embedder: SPLADE-v3 or the sparse head of bge-m3. It is the modern successor to BM25, not its rival; we will see why.

Librarian C, the copyist. He gives up on summarising. He takes every word on every page of the book and gives it a 128-dimension mini-RFID. He ends up with a 30,000-token book turned into 30,000 mini-RFIDs. When you search for something, the librarian compares every word of your question with every word of the book and keeps the maximum for each word of the question, adding them up. It captures nuance the other two lose by construction (proper nouns, numbers, specific phrasings), but its filing system is an order of magnitude larger in disk space. This is ColBERT-v2 / Jina-ColBERT-v2 / ColNomic-7B: late interaction, MaxSim.

And then there is the all-rounder librarian who does all three jobs in a single pass: bge-m3. A single 568 M parameter model that simultaneously returns the thematic RFID tag (1,024-d), the weighted list of terms (sparse) and the set of per-token mini-RFIDs (128-d). That is why it has settled in as the standard embedder for multilingual on-premise RAG: a single model.encode(chunk) produces the three outputs that feed hybrid retrieval without orchestrating three different models.

The same chunk indexed by three librarians. The all-rounder bge-m3 runs all three in a single 568 M parameter pass.

What an embedding really is

A text embedding is a function f : text → ℝᵈ trained so that two “semantically similar” texts produce nearby vectors in that space. The important part is not “vector”, any fixed-length hash is one, but what closeness means. Closeness is defined implicitly by the loss the model is trained with.

Almost every modern embedder is trained with InfoNCE (also called Multiple-Negatives Ranking Loss in sentence-transformers):

\mathcal{L}_{\text{InfoNCE}} = -\log \frac{\exp(\text{sim}(q, d^+)/\tau)}{\sum_{d \in \mathcal{B}} \exp(\text{sim}(q, d)/\tau)}

For each (query, doc⁺) pair the model has to assign the positive more similarity than all the other documents in the batch B, where the other documents act as free in-batch negatives. The temperature τ (typically 0.02–0.07, almost always 0.05) controls how sharp the distribution gets: lower τ means the model becomes more demanding about the positive but less stable. A large batch size |B| means many more negatives per gradient, so a better informed model. That is why embedders are trained with batch ≥ 1,024 on H100 clusters with AllGather across nodes to stack every negative in the cluster into a single effective batch.

On top of the in-batch negatives comes hard negatives mining: documents deliberately picked because they are “almost in the answer” (typically the next 10-100 neighbours from a previous BM25 / dense retrieval). Without hard negatives the model learns to discriminate the trivial and real quality on BEIR / MTEB drops 5-10 points.

This matters because the embedder family depends on what you feed into sim(·,·):

Family	`sim(q, d)`	Model output
Dense single-vector	dot product of two normalised 1024-d vectors	`f(q), f(d) ∈ ℝ^d`
Learned sparse	dot product of two 30,522-d vectors (≈80 non-zero each)	`f(q), f(d) ∈ ℝ^V`, `V =
Multi-vector	`Σᵢ maxⱼ ⟨qᵢ, dⱼ⟩` (MaxSim)	`f(q) ∈ ℝ^{

The loss is the same in all three cases, InfoNCE, but the geometry of the space changes, and with it quality, storage cost and search latency.

The three families in detail

Dense single-vector, the cartographer

The embedder reads the full chunk, passes it through an encoder transformer (XLM-RoBERTa, BERT, a Mistral decoder with a prompt) and aggregates the token representations into a single vector by:

CLS pooling: uses the embedding of the [CLS] token. Standard in BERT-base.
Mean pooling: plain average of the embeddings of all tokens. Standard in multilingual-e5-large-instruct, bge-m3 dense, Snowflake Arctic Embed L 2.0.
Last-token pooling: for decoder-LLM based embedders (e5-mistral-7b-instruct, gte-Qwen2-7B-instruct, NV-Embed-v2) that take the final token as the aggregate. It works because the model is causal and the last token “has seen” the whole context.
Latent-attention pooling: new in NV-Embed-v2. A learnable attention layer that weights tokens instead of averaging them. +2-3 MTEB points over mean pooling.

After pooling the vector is normalised to norm 1: v ← v / ‖v‖₂. With normalised vectors, cosine and dot product coincide:

\cos(q, d) = \frac{q \cdot d}{\|q\|\|d\|} = q \cdot d, \qquad \|q - d\|^2 = 2 - 2\,(q \cdot d)

That is why almost every vector DB indexes by inner product and leaves it to the user to normalise beforehand (Qdrant, Faiss IP, Milvus). If you forget to normalise the query vector but the corpus vectors are normalised, retrieval degrades silently: the query’s magnitude distorts the ranking. It is the number one bug in RAG in production.

A minimal numerical example, two normalised 4-d vectors:

q = [0.5, 0.5, 0.5, 0.5] ‖q‖ = 1
d₁ = [0.6, 0.4, 0.5, 0.5] ‖d₁‖ = 1.005, normalised [0.597, 0.398, 0.498, 0.498]
q · d₁ = 0.5·0.597 + 0.5·0.398 + 0.5·0.498 + 0.5·0.498 = 0.995

Closeness of nearly 1, as expected. If d₁ had not been normalised, q · d₁ = 1.0 and it would appear closer than any d that is perfectly aligned but with a norm < 1.005. Normalisation is not a detail: it is the contract of the space.

Learned sparse, the lexical descriptor

SPLADE-v3 (Naver, March 2024) has consolidated the modern version of the lexical librarian. Internally it is a small BERT (~110 M parameters, DistilBERT/BERT base) that produces, for each input token, a distribution over the whole vocabulary (30,522 dimensions in BERT WordPiece), and then does max-pool over the tokens:

w_j = \max_{i \in \text{seq}} \log\bigl(1 + \text{ReLU}(W_{ij})\bigr)

where Wᵢⱼ is the logit of input token i for vocabulary term j. The log(1+ReLU) saturates high logits (it stops a single word from dominating the vector) and the ReLU cuts the negatives. The result is a 30,522-dimension vector of which typically 50-200 entries stay non-zero.

The key part is the FLOPS regularization added to the loss during training:

\mathcal{L}_{\text{FLOPS}} = \lambda \cdot \sum_{j=1}^{V} \bar{w}_j^2, \qquad \bar{w}_j = \frac{1}{|B|}\sum_{i \in B} w_{ij}

It penalises the expected cost of the posting lists: if a vocabulary word appears on average in many documents, adding it to a new document costs double. The model learns to generate sparse vectors by construction.

What does this do to the text “Linkerd 3.8 brings mTLS by default”? The model does not only write the literal terms, it also writes, with a smaller but non-zero weight, “service-mesh”, “kubernetes”, “tls”, “sidecar”, “envoy”, “istio” (its competitor, also semantically related), “encryption”, “k8s”. That semantic expansion of the document is what sets SPLADE apart from BM25. BM25 only knows what was literally in the text; SPLADE knows what an expert would add as a descriptor.

In practice SPLADE-v3 beats BM25 by 3-6 MRR@10 points on MS MARCO and dominates BEIR zero-shot. The cost is ~2-4× the query latency of BM25 over the same inverted index, which static pruning can mitigate.

For the multilingual case, bge-m3 with its sparse head is the only maintainable option: SPLADE-v3 is trained in English and the multilingual ports are in an experimental state.

Multi-vector, the copyist

ColBERT-v2 (Stanford, NAACL 2022) introduced the late interaction paradigm. Instead of compressing the document to a single vector, it leaves it as a matrix (|d|, k) with a k-dimension vector per token. The similarity between query and document is computed token by token and aggregated with MaxSim:

s(q, d) = \sum_{i \in q} \max_{j \in d} \langle q_i, d_j \rangle

What is being computed: for each word of the query, find the document token that fits it best and add that similarity. The sum is over the query, not over the document. This lets a 30,000-token document compete fairly with a 200-token one, because the query always adds |q| terms.

Why does it give more quality than dense single-vector? Because compressing to one vector loses information about where each idea was. If the query is “which version brought mTLS by default in Linkerd”, the dense summary of the document only knows that the chunk is about “Linkerd and mTLS”; ColBERT’s copyist can match “which version” with “3.8” because it stores the embedding of the 3.8 token separately. On BEIR / out-of-domain, late interaction beats single-vector by between +2 and +6 nDCG@10 with the same backbone.

The price is storage. Per document, a 1024-dimension dense single-vector in fp16 takes 1024 × 2 = 2 KB. ColBERT-v2 with 128-dimension tokens for a 256-token chunk takes 256 × 128 × 2 = 65,536 B ≈ 64 KB: 32× more space. With ColBERT-v2’s nbits=2 residual compression it drops to ~16 KB (8×). Jina-ColBERT-v2 adds Matryoshka over the token dimensions (truncatable to 128 / 96 / 64), cutting another 50%.

For 1 million chunks:

Family	Per doc	Total
Dense fp32 (1024-d)	4,096 B	4.0 GB
Dense fp16 / halfvec	2,048 B	2.0 GB
Dense int8 (SQ)	1,024 B	1.0 GB
Dense binary (1 bit/d)	128 B	128 MB
SPLADE (≈ 80 terms × 8 B)	~640 B	~640 MB
ColBERT fp16 (256 tok × 128 d)	65,536 B	64 GB
ColBERT residual `nbits=2`	~16,000 B	~16 GB
Jina-ColBERT-v2 (MRL token 64)	~8,000 B	~8 GB

ColBERT in on-premise production is reserved for corpora of up to a few million chunks, or applied only as a reranker over the top-100 from the first committee (dense + sparse), as described in the reranker post.

Matryoshka, the truncatable dimension

An operational lever that changed the conversation about embeddings between 2024 and 2026 is Matryoshka Representation Learning (Kusupati et al., NeurIPS 2022). The trick: during training, on top of the loss over the full D-dimension vector, the same loss is computed over prefixes of the vector:

\mathcal{L}_{\text{MRL}} = \sum_{k \in \{64, 128, 256, 512, 1024\}} \alpha_k \cdot \mathcal{L}_{\text{InfoNCE}}\bigl(\text{emb}[:k]\bigr)

The first k dimensions of the embedding are trained to be, on their own, a valid embedding. At inference time, if you want a cheaper embedding, you truncate the vector: the first quarter is already a usable embedding. Without Matryoshka, truncating wrecks the geometry: the first 256 dimensions of an embedding trained only at 1024-d encode nothing coherent.

Typical degradation in MTEB nDCG@10 when truncating an MRL embedder:

Truncation	Average loss
1024 → 512	-1 to -2 points
1024 → 256	-3 to -5 points
1024 → 128	-5 to -8 points
1024 → 64	-8 to -12 points

Native MRL models in 2026 (the ones that let you pick the dimension at runtime without retraining):

jina-embeddings-v3 (1024 → 32, fine steps, CC-BY-NC-4.0)
jina-embeddings-v4 (2048 → 128, multimodal text+image, CC-BY-NC-4.0)
nomic-embed-text-v2-moe (768 → 256, Apache 2.0)
Snowflake-arctic-embed-l-v2.0 (1024 → 256, Apache 2.0)
mxbai-embed-large-v1 and mxbai-embed-2d-large-v1 (the latter also truncatable in layer depth)
Stella_en_1.5B_v5 (multiple steps 512 / 768 / 1024 / 2048 / 4096 / 6144 / 8192, MIT, English only)
text-embedding-3-large (OpenAI, 3072 → 256, API only)
voyage-3 family (1024 → 256 / 512 / 1024 / 2048, API only)

Production recommendation: always use an MRL model even if you do not truncate at the start, because it simplifies the future quantisation decision. And above all, evaluate truncation on your real corpus: the average MTEB degradation of “-3 points” becomes “-12 points” in a niche domain.

The 2026 open source model zoo

What follows is a spec sheet per model. Verified as of 2026-06: HuggingFace cards + reference papers + MTEB / MMTEB leaderboard.

Dense single-vector

Model	Params	Dim	Tokens	Languages	Licence	Distinctive
`BAAI/bge-m3` (dense)	568 M	1024	8192	100+	MIT	Tri-mode (dense + sparse + colbert) in one forward. De facto standard for multilingual on-prem.
`Snowflake/snowflake-arctic-embed-l-v2.0`	568 M	1024 (MRL → 256)	8192	~100	Apache 2.0	Trained for multilingual + English without degrading either. MIRACL 55.8.
`intfloat/multilingual-e5-large-instruct`	560 M	1024	512	~100	MIT	Veteran multilingual baseline. Short window.
`intfloat/e5-mistral-7b-instruct`	7.1 B	4096	4096	English	MIT	First decoder-as-embedder to break MTEB. English.
`Alibaba-NLP/gte-Qwen2-7B-instruct`	7 B	3584	32,768	100+	Apache 2.0	The only one with 32k context. Top MTEB-en, strong multilingual.
`nvidia/NV-Embed-v2`	7.85 B	4096	32,768	English	CC-BY-NC-4.0	Latent-attention pooling. Top quality. Licence blocker in prod.
`Linq-AI-Research/Linq-Embed-Mistral`	7 B	4096	4096	English	CC-BY-NC-4.0	Top MTEB retrieval Aug 2024. Non-commercial.
`NovaSearch/stella_en_1.5B_v5`	1.54 B	8192 (multiple MRL)	8192	English	MIT	Small + rich MRL. English.
`BAAI/bge-multilingual-gemma2`	9 B	3584	8192	100+	Gemma	High quality, the Gemma licence restricts redistribution.
`BAAI/bge-en-icl`	7 B	4096	8192	English	MIT-style	In-context learning from examples in the prompt.
`mixedbread-ai/mxbai-embed-large-v1`	335 M	1024 (MRL)	512	English	Apache 2.0	MRL + native binary. Short window.
`jinaai/jina-embeddings-v3`	570 M	1024 (MRL → 32)	8192	89	CC-BY-NC-4.0	Per-task LoRA. Non-commercial without a licence.
`nomic-ai/nomic-embed-text-v2-moe`	475 M / 305 M active	768 (MRL → 256)	512	~100	Apache 2.0	First general-purpose MoE in embeddings.

Learned sparse

Model	Params	Vocab	Tokens	Languages	Licence	Distinctive
`naver/splade-v3`	110 M	30,522	512	English	CC-BY-NC-SA-4.0	SOTA learned sparse. Non-commercial.
`BAAI/bge-m3` (sparse head)	568 M	XLM-R vocab	8192	100+	MIT	The only license-clean multilingual option.

Multi-vector (late interaction)

Model	Params	Dim/token	Tokens	Languages	Licence	Distinctive
`colbert-ir/colbertv2.0`	110 M	128	512	English	MIT	The original paper, the basis for everything.
`jinaai/jina-colbert-v2`	560 M	128 / 96 / 64 (MRL)	8192	89	Apache 2.0	The license-clean multilingual multi-vector.
`nomic-ai/colnomic-embed-multimodal-7b`	7 B	3584 (Qwen2-VL)	—	~100	Apache 2.0	Multimodal text+image multi-vector. Vidore-v2 open SOTA.

The leaderboard, with caution

MTEB / MMTEB (Massive Multilingual Text Embedding Benchmark, Enevoldsen et al., arxiv 2502.13595) is the standard thermometer. Top retrieval on MMTEB in mid-2026 is dominated by Qwen3-Embedding-8B (~70.6 multilingual avg) and Llama-Embed-Nemotron-8B. Below them, the 7B models (gte-Qwen2-7B, NV-Embed-v2) and the 568M ones (bge-m3, Snowflake-Arctic-L-2.0) compete task by task.

Trap: MTEB has started to saturate through dataset contamination. The higher the ranking, the more likely it is that the model saw subsets of the evaluation datasets during training. The rule in production: the leaderboard is for discarding bad models, not for picking the best one. The final decision is made on your own domain eval set, generated with the LLM-as-judge or evals recipe.

The specific problem of multilingual Spanish

For a Spanish customer serving corporate, legal or support documentation in Spanish (and often with Catalan / Portuguese / English mixed in), the embedder zoo narrows to fewer than six viable models. The operational exclusions:

English-only models: e5-mistral-7b-instruct, stella-en-1.5B-v5, Linq-Embed-Mistral, mxbai-embed-large-v1, NV-Embed-v2, SPLADE-v3. They cut Spanish performance below an acceptable level: translating the query into English before searching is one route, but it introduces latency, tokenisation drift and another model dependency.
Models with a non-commercial licence: jina-embeddings-v3, jina-embeddings-v4, NV-Embed-v2, Linq-Embed-Mistral. They work for a PoC, but commercial production needs an explicit agreement with the vendor. Unless you have the licence signed, they have to be excluded.
Models with a Gemma licence: bge-multilingual-gemma2. Allowed for internal use, awkward to redistribute weights to a customer.

The ones left, ordered by practical choice in sovereign production:

BAAI/bge-m3 — MIT, 568 M, 100+ languages (including Spanish and Catalan, explicitly trained), 8,192 tokens, tri-mode dense+sparse+colbert. A reasonable default. Fits in an RTX 4090. TEI and Infinity serve it natively.
Snowflake/snowflake-arctic-embed-l-v2.0 — Apache 2.0, same size, explicit Matryoshka, better MIRACL/CLEF than bge-m3 on some multilingual tasks, no sparse head. If the priority is pure MMTEB in Spanish.
intfloat/multilingual-e5-large-instruct — MIT, 560 M, veteran baseline. Its 512-token window is the big limitation: long documents have to be split first. If what you already have in production works, do not migrate out of fashion.
Alibaba-NLP/gte-Qwen2-7B-instruct — Apache 2.0, 32k context, high quality in Spanish (Qwen2 is well trained on Spanish). If the chunks are long (more than 4k tokens) and you have the GPU to serve it (it does not fit in a 4090; it does in an H100). It fits alongside an LLM on an H100 80GB with care.
nomic-ai/nomic-embed-text-v2-moe — Apache 2.0, 305 M active, MRL, ~100 languages. If latency and cost per token rule: the MoE gives it disproportionate throughput for its quality.
jinaai/jina-colbert-v2 — Apache 2.0, multilingual multi-vector, as a reranker or as the main retrieval on a small corpus (< 1 M chunks). The only license-clean multi-vector in Spanish.

The rule of thumb: bge-m3 as dense+sparse on the front line, jina-colbert-v2 as a third reranking layer when the use case warrants it, and Snowflake Arctic L 2.0 as the alternative if the corpus-specific eval prefers its geometry.

Serving embeddings on-premise

Three engines split the on-prem embedding serving landscape in 2026, with different profiles.

Text Embeddings Inference (TEI), the standard

huggingface/text-embeddings-inference is a server written in Rust with a Candle / ONNX backend, integrated FlashAttention and dynamic batching by tokens. It exposes an OpenAI-compatible /v1/embeddings API and supports the three bge-m3 modes simultaneously from version 1.5 onwards. For multilingual production it is the obvious default.

# values.yaml — TEI serving multilingual bge-m3 on an RTX 4090
image: ghcr.io/huggingface/text-embeddings-inference:1.5
args:
 - --model-id=BAAI/bge-m3
 - --pooling=cls
 - --max-batch-tokens=16384
 - --max-concurrent-requests=512
 - --dtype=float16
resources:
 limits:
 nvidia.com/gpu: 1

Indicative throughput with bge-m3, fp16, 512-token sequence, batch 32:

RTX 4090 (24 GB): ~8–15 k tokens/s
A100 80 GB: ~60 k tokens/s sustained
H100 80 GB: ~40–80 k tokens/s, with fp8 about 50% more

(The ranges are approximate and depend on the real batch, the average sequence length and compilation with FA2/FA3.)

Infinity, the flexible one

michaelfeil/infinity (MIT) is a multi-model FastAPI server able to load bge-m3, Snowflake Arctic, Jina-v3, Nomic, ColPali, CLAP and rerankers simultaneously behind the same OpenAI-style API. Backend PyTorch + Optimum (ONNX/TensorRT) or CTranslate2. Useful when you need to serve several different embedders (one for text, another for code, another for images) behind a single endpoint, or when the model does not yet have TEI support.

vLLM `--task embed`, for the 7B embedders

When the embedder is really an LLM decoder turned into an embedder (e5-mistral-7b-instruct, gte-Qwen2-7B-instruct, NV-Embed-v2, Stella-1.5B), the natural place to serve it is vLLM, which already has the PagedAttention and continuous batching stack in production:

vllm serve Alibaba-NLP/gte-Qwen2-7B-instruct \
 --task embed \
 --dtype bfloat16 \
 --max-model-len 32768 \
 --trust-remote-code

vLLM detects the correct pooling (last-token on the Qwen / Mistral based ones) and exposes an OpenAI-compatible /v1/embeddings. For inference clusters already running vLLM with an LLM on another port, it is the natural way to serve the embedder without standing up another stack.

fastembed, the lightweight one

qdrant/fastembed loads bge-small, MiniLM, ColBERT and BM25/SPLADE sparse in ONNX-CPU. It is not competitive on throughput against TEI/Infinity with a GPU, but it is the right option when you have to serve embeddings on a NUC node without a GPU (see mixed NVIDIA + Intel environments) or when the embedder is part of the client (a preview in a UI, prior scoring at the edge).

Storage, quantisation and the corpus arithmetic

The embedding does not stay in the embedder’s memory: it lives in the vector DB index and is materialised every time you ingest a new chunk. The storage cost calculation is what decides the final dimension, not MTEB quality. For 1 million chunks with a dense embedder at 1024-d:

fp32 : 1,024 dims × 4 B × 1 M = 4,096 MB ≈ 4.0 GB
fp16/halfvec: 1,024 dims × 2 B × 1 M = 2,048 MB ≈ 2.0 GB
int8 (SQ) : 1,024 dims × 1 B × 1 M = 1,024 MB ≈ 1.0 GB
binary : 128 B × 1 M ≈ 128 MB

The quantisation options in order of real use (mid-2026):

halfvec (fp16): the default in pgvector 0.7+ and in any serious vector DB. Zero MTEB loss, 2× compression. Always turn it on.
Scalar Quantization int8 (SQ): each vector component is mapped to int8 with a global min/max. Typical recall@10 loss: 0–1%. 4× compression. The Qdrant default, supported in Milvus and Weaviate.
Binary quantisation: bit = sign(v_i - μ_i). 32× raw compression. Cold loss 5–15%. Mitigated with a Hadamard / TurboQuant pre-rotation (Qdrant 1.18, Dec 2025): it pre-multiplies by a random orthogonal matrix that spreads the energy across dimensions before binarising. After TurboQuant the loss drops to 1–3%. It also combines with rescoring over the original fp16 for the top-100 candidates.
Product Quantization (PQ): the FAISS classic. Up to 64× compression, 2–5% loss. More complex to operate (it needs a trained codebook); in 2026 it has given ground to binary + rescoring.

A corpus of 100 million chunks (a real figure for a large corporate RAG) with dense bge-m3:

Format	Total
fp32	400 GB
fp16	200 GB
int8	100 GB
binary + Hadamard	12.5 GB

The difference between 200 GB and 12.5 GB is the difference between needing a dedicated vector DB node with 8 NVMe in RAID and fitting in the RAM of a single node. For large corpora, quantisation is no longer an optimisation: it is the only way to operate.

The vector DB integration

The vector DBs of 2026 have become hybrid DBs that index all three types at once. The quick map:

Vector DB	Native hybrid	Multi-vector / ColBERT	Quantisation
Qdrant ≥1.10	RRF/DBSF in `query_points` with dense + sparse + colbert in one collection	Yes, native (one-shot MaxSim)	SQ int8, binary, TurboQuant 1.18
Weaviate	`hybrid(alpha=0.75)` BM25 + dense, multi-target named vectors	Yes, as a multi-vector named vector	PQ, SQ, rotational 8-bit BBQ
Milvus ≥2.4	Multi-vector + sparse in the schema; 2.5 adds native BM25 full-text	Multi-vector field, MaxSim orchestrated from the client	SQ, PQ, CAGRA GPU
pgvector 0.7+/0.8	`halfvec`, `sparsevec`, `bit`; HNSW for all three	Not native (separate-table workaround)	`binary_quantize()`, halfvec, rescoring with exact `<#>`
Elasticsearch / OpenSearch	`sparse_vector` (ELSER, SPLADE) + `dense_vector` HNSW; RRF	OpenSearch 3.x yes	ES 9: `int8_hnsw` by default, BBQ binary quantization

For sovereign on-prem production in Spanish, the easiest combination to operate in 2026 is Qdrant + bge-m3: a single collection indexes the three modes of the same model, the hybrid query with RRF is done in one call, TurboQuant quantisation brings the corpus down to manageable levels, and the operator is a Go binary with simple backups to S3/MinIO. pgvector + bge-m3 is the other reasonable option when you already have Postgres with HA and do not want to add a second DB to the operational inventory; you lose native multi-vector, but you gain cross-cutting SQL over the chunks.

The HNSW parameters you absolutely have to touch:

M: connections per node in the graph. 16–32 typical. Higher means more recall, more RAM. For small corpora (<1 M) M=16; for medium corpora (10 M) M=24; for large corpora M=32+IVF-PQ or M=32+binary.
ef_construction: search width during construction. 100–400. Higher means a better graph, slower construction. Build with ef_construction=400 even if it is slow; you pay it once.
ef_search: width during the query. 50–200. The main knob for the recall/latency trade-off at runtime. Start at 64 and measure.

Implications for on-premise inference

The embedder does not share hardware with the LLM as comfortably as it might seem. The numbers:

bge-m3 (568 M) takes about 568 × 2 = 1,136 MB in fp16 for the weights, plus the batch KV cache, plus temporary activations. In practice it is served comfortably in 6–8 GB of VRAM even at a high batch. It fits alongside a 7B-Q4 LLM on an RTX 4090.
gte-Qwen2-7B-instruct needs ~14 GB fp16 for weights alone. It does not fit next to a 7B LLM on a 4090; on an H100 80 GB it does, with care over simultaneous batching.
jina-colbert-v2 (560 M) takes ~1.1 GB of weights, but the storage of the multi-vector index is the real cost: 8 GB per million chunks even with Matryoshka and compression.

On the RTX 4090 (24 GB)

A realistic minimum stack for a Spanish RAG with a corpus under 1 M chunks:

GPU 24 GB ┐
 ├─ TEI bge-m3 (dense + sparse + colbert) │ ~6 GB VRAM, ~12 k tok/s
 └─ vLLM Qwen2.5-7B-Instruct AWQ Q4 │ ~8 GB VRAM, ~80 tok/s
CPU/RAM ┐
 ├─ Qdrant with bge-m3 dense + sparse + colbert │ ~3 GB RAM per M chunks
 └─ FastAPI gateway (LiteLLM)

It serves a few tens of RAG QPS with decent multilingual quality. It is the PoC configuration and the deployment for a small site.

On the 4×H100 80 GB cluster

For the production case with several million chunks and an SLO of p99 < 500 ms:

H100 #1 (80 GB) ── vLLM Qwen3-72B-Instruct AWQ + Qwen2.5-7B speculative ┐
H100 #2 (80 GB) ── vLLM gte-Qwen2-7B-instruct (embedding 32k ctx) │ LLM + large embed
H100 #3 (80 GB) ── TEI bge-m3 multi-tenant + jina-colbert-v2 reranker │ medium embed
H100 #4 (80 GB) ── Hold-out for canary / shadow │ see canary post
 ┘
Qdrant cluster (3 CPU nodes + NVMe) ── 100 M chunks indexed (binary + TurboQuant + rescoring)

This configuration separates the large LLM from the large embedder (they share the Qwen2 architecture but compete for VRAM if put on the same GPU) and leaves a whole H100 for variants in canary. bge-m3 fits with room to spare alongside the reranker on a single H100, serving tens of thousands of requests/min.

The seven operational traps of the embedder

Not normalising the query vector. Cosine and dot product coincide only when both vectors are unit vectors. If you forget v ← v / ‖v‖₂ in the client, the results are “almost right”, the top-1 is still correct on trivial queries, the top-10 no longer is, and nobody notices until RAG quality drops 8 points. Fix: bake normalisation into the embedder adapter, not into the client.
Chat template slipped into the embedder. Some LLM-based embedders (e5-mistral-7b-instruct, gte-Qwen2-7B) expect a specific instruction prompt before the text to embed ("Instruct: Retrieve relevant passages\nQuery: ..."). Forgetting it leaves performance ~5 MTEB points lower. Fix: read the usage_template in the model card and put it into the embedder wrapper.
Badly chosen Matryoshka dimension. The default in many Qdrant / pgvector clients is dim=768. If your embedder is natively MRL at 1024 → 768, fine. If it is 1024 without MRL, truncating to 768 wrecks the space (typical loss -8 MTEB points). Fix: use the model’s native dim and truncate only when storage demands it, and only on MRL models.
Missing hard negatives in fine-tuning. When the embedder is fine-tuned with your own data (which should be standard practice for corporate RAG), if the mini-batch only carries positives and in-batch negatives from the same domain, the model learns that anything outside the domain is negative, but inside the domain it does not discriminate. Fix: mine hard negatives with BM25 / dense from your own corpus before fine-tuning.
Corpus drift without reindexing. When you retrain or replace the embedder but only apply the new model to new chunks, you end up with the index mixing two incompatible geometries. Chunks from the old and the new model are not comparable by cosine. Fix: every change of embedder is a full reindexing of the corpus, planned as an operational retrain.
Tokenizer drift between client and model. The Python client that prepares the queries uses its own tokenizer (sometimes tiktoken by default) and truncates at 8,192 tokens. The embedder uses XLM-R with sentencepiece and truncates at 8,192 of its own tokenizer. Long queries are truncated differently; the corpus embeddings are consistent but the query ones are not. Fix: use the model’s tokenizer in the client or in the wrapper.
MTEB overfit as a selection guide. The MTEB leaderboard has become a contaminated metric: there is evidence that leading models saw subsets of the evaluation datasets during training. The model that is +0.5 points above the second is not necessarily better for your domain. Fix: your own domain eval set (100-300 labelled query-doc pairs) run with the evals recipe decides.

License-clean stack for sovereign production

Let us write down the recommended ending. For a Spanish organisation serving corporate RAG on-prem under ENS / ISO 42001 / EU AI Act, with a corpus of 1-50 M chunks in Spanish + English + Catalan:

Layer	Component	Licence	Rationale
Dense embedder	`BAAI/bge-m3`	MIT	Robust multilingual, 8k tokens, license-clean, served by TEI
Sparse embedder	`bge-m3` sparse head	MIT	Same pass as the dense one, no second model needed
Layer 2 reranker	`BAAI/bge-reranker-v2-m3`	MIT	Multilingual cross-encoder from the same team
Layer 3 reranker (optional)	`jinaai/jina-colbert-v2`	Apache 2.0	License-clean multilingual multi-vector
Embed server	`TEI` + `Infinity` for multi-model	Apache 2.0 / MIT	Supported stack
Vector DB	`Qdrant` (preferred) or `pgvector 0.8`	Apache 2.0 / PostgreSQL	Native hybrid + quantisation
Quantisation	`int8 SQ` + `binary + TurboQuant + rescoring`	Apache 2.0	Cuts the corpus 16×–32× with < 3% loss
Minimum hardware	RTX 4090 24 GB	—	For PoC and small sites
Production hardware	4×H100 80 GB cluster	—	For RAG with an SLO of p99 < 500 ms

The alternative stack, if explicit MMTEB in Spanish weighs more than the tri-modality of bge-m3: replace bge-m3 with Snowflake/snowflake-arctic-embed-l-v2.0 (Apache 2.0, MRL → 256) and explicitly add SPLADE-v3 or plain BM25 for the sparse layer. It loses the elegance of the single forward and gains 1-2 points on Spanish MIRACL.

Conclusion

The embedder is the easiest piece of a RAG to oversimplify and the one that decides real quality the most once everything else is in place. The three families (dense, sparse, multi-vector) are not three options to choose between but three trades that bge-m3 performs in a single pass and that hybrid retrieval consumes in parallel. The maths that matters is modest, InfoNCE with τ, MaxSim, FLOPS regularization, MRL, but the operational traps are many and quiet: forgotten normalisation, missing chat template, badly chosen Matryoshka dimension, tokenizer drift. For sovereign production in Spanish the list of viable models fits in fewer than ten, and the real decision comes down to “bge-m3 or Snowflake Arctic L 2.0”, with jina-colbert-v2 added as layer three when the fine-grained quality justifies the cost. The license-clean stack fits on an RTX 4090 for a PoC and scales to a 4×H100 cluster for real production.

References

Chen et al. M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation. arXiv:2402.03216. https://arxiv.org/abs/2402.03216
Sturua et al. Jina Embeddings v3: Multilingual Embeddings With Task LoRA. arXiv:2409.10173. https://arxiv.org/abs/2409.10173
Günther et al. Jina Embeddings v4: Universal Embeddings for Multimodal Multilingual Retrieval. arXiv:2506.18902. https://arxiv.org/abs/2506.18902
Nussbaum et al. Nomic Embed v2: Multilingual Mixture of Experts. arXiv:2502.07972. https://arxiv.org/abs/2502.07972
Wang et al. Improving Text Embeddings with Large Language Models (E5-Mistral). arXiv:2401.00368.
Yu et al. Arctic-Embed 2.0: Multilingual Retrieval Without Compromise. Snowflake, 2024-12. https://www.snowflake.com/blog/arctic-embed-2-multilingual/
Lee et al. NV-Embed: Improved Techniques for Training LLMs as Generalist Embedding Models. arXiv:2405.17428.
Khattab y Zaharia. ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT. SIGIR 2020.
Santhanam et al. ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction. arXiv:2112.01488.
Jha et al. Jina-ColBERT-v2: A General-Purpose Multilingual Late Interaction Retriever. arXiv:2408.16672.
Lassance et al. SPLADE-v3. arXiv:2403.06789.
Kusupati et al. Matryoshka Representation Learning. NeurIPS 2022, arXiv:2205.13147.
Enevoldsen et al. MMTEB: Massive Multilingual Text Embedding Benchmark. arXiv:2502.13595.
Hugging Face Text Embeddings Inference. https://github.com/huggingface/text-embeddings-inference
Michael Feil. Infinity. https://github.com/michaelfeil/infinity
Qdrant. TurboQuant 1.18 release notes. https://qdrant.tech/articles/turboquant-quantization/
pgvector. Release notes 0.7 / 0.8. https://github.com/pgvector/pgvector
Hugging Face. Embedding Quantization. https://huggingface.co/blog/embedding-quantization

Incident response runbooks for LLM inference: every alert to a concrete action with Kafka and Keep

Tue, 02 Jun 2026 04:30:00 +0200

This post closes the observability trilogy opened by GPU observability for LLM inference (which metrics) and Anatomy of the twelve DCGM and five vLLM metrics (which documented anomaly per metric). Here each anomaly gets its concrete action and is fitted into the incident management machinery that compliance demands.

TL;DR

The GPU observability alerts are useless without a codified procedure for each one; the operator who interprets them by hand every time is operating on intuition. The right combination has three indispensable pieces. (1) Runbook catalogue: for each of the six critical alerts (GpuHbmNearOom, GpuThermalOrPowerThrottle, GpuXidErrorDetected, GpuEccDoubleBit, VllmKvCachePoolNearFull, VllmTtftP95OutOfSlo), a severity, an immediate mitigation, the evidence to capture before remediating, a resolution action, a closure criterion and a postmortem trigger. (2) Reproducible pipeline: Prometheus + DCGM → Alertmanager → Kafka as event bus (topics gpu.alerts.enriched, incidents.lifecycle, audit.actions with WORM retention) → Keep as workflow engine (declarative YAML workflows versioned in git) → Kubernetes job / script / ChatOps executors. (3) Formal fit into incident management under the regulatory corpus: ISO/IEC 27035 phases identify → report → assess → respond → learn; ENS controls op.exp.7 (incident management), op.exp.8 (activity logging), op.exp.10 (notification to users); NIS2 art. 23 with an early warning at 24 h, a formal notification at 72 h and a final report at 1 month; EU AI Act art. 73 for a serious incident on a high-risk system, deadlines of 2 to 15 days depending on severity; ISO/IEC 42001 clause 10 (continual improvement of the AIMS). The action taxonomy is immediate mitigation (drain, throttle, scale-down: contains the damage in seconds) → diagnosis (evidence capture with nvidia-smi -q, dmesg, a vLLM /metrics snapshot, the related OTel trace; without this the postmortem is not defensible) → resolution (restart, reset, RMA, rollback) → postmortem (5-whys RCA, prevention plan, runbook update). Kafka contributes the immutable audit trail that ENS and the EU AI Act demand: every action executed by Keep or by a human is published as an event on audit.actions with timestamp, actor, decision and evidence, retained WORM for at least 6 months. Keep contributes workflows as code: this post includes three complete workflows (XID with drain plus Jira ticket, ECC DBE with immediate paging and node blocking in the scheduler, automatic canary rollback on TTFT P95 out of SLO). Four anti-patterns close the material: alerts without a runbook (most of them), a runbook without prior evidence capture (it perpetuates the incident because the root cause is lost), escalation by seniority rota instead of severity (a junior operator handling an ECC DBE), and the absence of a human gate for destructive actions (Keep running nvidia-smi --gpu-reset without confirmation). Applicable to a generic 4×H100 SXM cluster with Kafka and Keep already deployed.

You are here: OBSERVE → DEPLOY (incident response closes the loop)

The analogy: a nuclear reactor control room

In a nuclear plant control room, the operator on shift never decides what to do on seeing an alarm. The decision is pre-made and codified in a written procedure (SOP) covering every alarm on the panel: if X sounds, open book X, read steps 1-N, execute exactly, call the supervisor at step M, escalate to the plant director at step N+3. The reason is strict: critical alarms are rare but catastrophic if handled badly; an operator improvising in an emergency makes worse decisions than one applying a procedure reviewed by experts and validated by simulation.

The reactor does not expect the operator to be a genius. It expects them to know the procedures to the letter and the operations management system to hand them the right procedure at the right moment. If the procedures are not written, not versioned, or not integrated with the alarms that fire them, the control room is operating on intuition. The difference between the two operations, procedural versus intuitive, is the difference between a plant that runs 30 years without incidents and one that ends up on a blacklist.

Incident response on an LLM inference cluster works identically. The DCGM and vLLM alerts listed in the previous posts are the panel alarms. Each one needs its written SOP, versioned, integrated with the alert that fires it and reviewed after every incident. Without that codification, the operator on shift improvises in the middle of an ECC DBE failure at 4 in the morning; with it, they execute the nine steps of runbook 12 and the incident closes in 20 minutes.

The incident pipeline architecture

Prometheus + DCGM. Collects the metrics described in the two previous posts. PrometheusRules define the six critical alerts with for: <duration> to avoid noise.

Alertmanager. Receives raw alerts; deduplicates, groups by labels ({cluster, node, gpu, model}), routes. Instead of sending straight to PagerDuty or Slack, it sends to Kafka via a webhook receiver. That turns the alert into a bus event that multiple consumers process (Keep for action, the audit topic for compliance, dashboards for visualisation).

Kafka as event bus. Three canonical topics:

gpu.alerts.enriched — alerts with added context (tenant, model, version, namespace owner, effective severity). Retention: 7 days, replication factor 3.
incidents.lifecycle — incident cycle events: incident.opened, incident.acknowledged, action.proposed, action.executed, incident.escalated, incident.resolved, postmortem.attached. Retention: 90 days.
audit.actions — an immutable record of every action executed (by Keep automatically or by a human confirming). Retention: 6 months minimum with compaction off plus tiered storage, WORM storage. This is the topic ENS op.exp.8, EU AI Act art. 12 and NIS2 require you to keep.

Keep as workflow engine. Consumes from gpu.alerts.enriched, fires YAML workflows versioned in git, executes actions (HTTP calls, kubectl jobs, Slack messages, Jira tickets) and publishes the result to incidents.lifecycle plus audit.actions. Choosing Keep over Alertmanager alone (or over PagerDuty alone) is deliberate: Keep separates runbook declaration (readable, reviewable YAML) from notification delivery (PagerDuty). The runbook is versioned code; notifications are operational details.

Executors. What actually moves the cluster:

Kubernetes jobs: kubectl drain, kubectl cordon, kubectl rollout undo.
NVIDIA API: nvidia-smi --gpu-reset, dcgmi diag -r <level>.
ChatOps: human confirmations through Slack interactive messages before a destructive action runs.
External tooling: Jira ticket, PagerDuty notification, CMDB call.

The six critical alerts and their runbooks

For each alert: severity, immediate mitigation (seconds), evidence to capture before remediating, resolution action, closure criteria, postmortem trigger.

RB-01 · `GpuHbmNearOom` — HBM > 92 % sustained

Severity: WARNING. Risk of OOM on the next PagedAttention allocation.

Immediate mitigation. Reduce admission temporarily by lowering max_num_seqs on the affected engine via hot reload (if the engine supports it) or a staggered restart of replicas. Trigger an additional scale-out via KEDA if there are free GPU nodes. There is no need to drain the node.

Evidence to capture.

nvidia-smi --query-gpu=index,memory.used,memory.free,memory.total --format=csv
nvidia-smi -q -d ROW_REMAPPER | grep -i pending
curl http://vllm-pod:8000/metrics | grep -E "gpu_cache_usage|num_requests"
kubectl logs <pod> --tail=200 | grep -i "preempt\|swap"

Save the snapshot to audit.actions with a timestamp and an incident_id.

Resolution. If the cause is a traffic spike: let the autoscaler scale to a stable regime, monitor for 30 min. If the cause is a model regression (canary v2 consumes more KV cache than v1): roll the canary back (see RB-06). If it is a leak (the metric grows without traffic growing): restart the pod with a heap dump capture.

Closure. gpu_cache_usage_perc < 80 % sustained for 15 min AND num_requests_waiting == 0.

Postmortem. Not mandatory unless the incident lasted > 30 min or had SLO impact.

RB-02 · `GpuThermalOrPowerThrottle` — bit ≠ 0 and not Idle in CLOCK_THROTTLE_REASONS

Severity: WARNING (thermal) or CRITICAL (sustained HW Power Brake, PDU risk).

Immediate mitigation. Identify the bit (decode the bitmap). If it is 0x40 HW_THERMAL or 0x20 SW_THERMAL: drain the workload from the node to other replicas if the temperature does not drop within 2 min, and stop new pods landing on that node (kubectl cordon). If it is 0x80 HW_POWER_BRAKE: alert DC infrastructure immediately (likely an over-committed PDU, the Dell KB 000220508 / Lenovo HT514380 case) and lower the TDP of the rack’s GPUs via nvidia-smi -pl to a lower value to relieve load on the breaker.

Evidence.

nvidia-smi --query-gpu=index,temperature.gpu,temperature.memory,power.draw,clocks_throttle_reasons.active --format=csv
ipmitool sdr | grep -i "fan\|temp\|inlet"
# PDU data if they are instrumented (modbus / SNMP)

Resolution.

Thermal: review rack airflow, check the rear-door HX, T_inlet, DGX fans. An infra issue, not an engine one.
Power Brake: review the sizing of the PDU branch, the breaker, the 415 VAC distribution. Likely a redistribution of load to another branch or a temporary TDP cap.

Closure. CLOCK_THROTTLE_REASONS == 0x1 (Idle only) or 0x0 for 30 min under normal load.

Postmortem. Mandatory if it was a HW Power Brake: that implicates the DC electrical infrastructure.

RB-03 · `GpuXidErrorDetected` — `increase(DCGM_FI_DEV_XID_ERRORS[5m]) > 0`

Severity: CRITICAL.

Immediate mitigation. kubectl cordon the node (no more new pods). If the XID is 31/48/79/94/95 (hardware or cascade): drain the node’s existing pods. If the XID is 13/43 (possibly software): keep the pods but block new ones, capture the trace and the active workload.

Evidence.

# The specific XID from dmesg
dmesg | grep -i xid | tail -30
nvidia-smi -q -d ERROR
nvidia-smi -q -d PCIE
# State of the retired pages
nvidia-smi -q -d ROW_REMAPPER
# Workload that was running
kubectl get pods -o wide | grep <node>
kubectl logs <pod> --previous --tail=500

Resolution.

XID 13/43 (software exception / channel verif): if it recurs only with one specific model, it is a workload bug; raise an issue with the model team. If it is transient, restarting the pod is enough.
XID 31 (MMU fault): usually a cascade from a previous XID 48. Reset the GPU (nvidia-smi --gpu-reset -i <index>) or reboot the node if the reset does not resolve it.
XID 48 / 95 (DBE / uncontained ECC): see RB-04. The node goes into quarantine.
XID 79 (fallen off the bus): reboot the node. If it recurs after the reboot, open an RMA for the GPU. ByteDance reports 43 % co-occurrence with PCIe errors, so check the slot and the cable too.
XID 94 / 145 / 149: catalogued in NVIDIA’s Xid Catalog with a specific procedure.

Closure. Node smoke test passed (dcgmi diag -r 3), 24 h with no new XIDs, back into the pool.

Postmortem. Mandatory. Include the specific XID, the distribution of XIDs across the cluster, an updated MTBE.

RB-04 · `GpuEccDoubleBit` — `DCGM_FI_DEV_ECC_DBE_VOL_TOTAL > 0`

Severity: CRITICAL, data corruption in progress.

Immediate mitigation. Drain the node immediately without waiting for further evidence. Page (PagerDuty / OpsGenie) the primary ON-CALL. Mark the node unschedulable and failed. XID 48 has a 100 % probability of killing the running job according to the Story of Two GPUs dataset; any inference already in flight is compromised.

Evidence (in parallel with the mitigation).

nvidia-smi -q -d ECC
nvidia-smi -q -d ROW_REMAPPER # Pending: Yes expected
dmesg | grep -E "Xid.*48|DBE|double-bit" | tail -50
# Full capture of the GPU state
dcgmi diag -r 4 -i <gpu_index>

Resolution. Full GPU reset (nvidia-smi --gpu-reset) or a node reboot if the reset does not complete. The reset activates the row remap. After the reboot:

nvidia-smi -q -d ROW_REMAPPER # Pending: No expected
nvidia-smi -q -d ECC # volatile counters at 0

If RETIRED_DBE > 8 pages after the remap: schedule a GPU replacement in the next window, because silicon degradation is progressive. The published real case documents ~19 hours of downtime as typical.

Closure. Node back in the pool after 48 h with no new DBEs.

Postmortem. Mandatory. If the incident affected a request carrying personal or classified data, evaluate notification to the DPO under GDPR art. 33 (it is not necessarily a breach, but it has to be assessed).

RB-05 · `VllmKvCachePoolNearFull` — `gpu_cache_usage_perc > 95 %` sustained 3 min

Severity: WARNING (risk of preempt-on-OOM, not of a real OOM).

Immediate mitigation. Activate autoscaler scale-out by lowering the KEDA threshold temporarily (from 0.85 to 0.75) for 30 min. In recompute mode, preempts raise TTFT but do not break requests; acceptable in the short term. In swap mode latency goes through the roof, so it is better to cut new traffic (return 503 from the router) for 5 min.

Evidence.

curl http://vllm-pod:8000/metrics | grep -E "gpu_cache|num_requests|num_preemptions"
kubectl get hpa vllm-llama70b
kubectl logs <pod> --tail=200 | grep -i preempt

Resolution. If it recurs regularly: revisit capacity planning, possibly lowering max_num_seqs or raising the stable replica count. See Capacity planning.

Closure. Pool < 85 % sustained 30 min, no preempts in the last 15 min.

Postmortem. Not mandatory unless it recurs > 3 times / week.

RB-06 · `VllmTtftP95OutOfSlo` — TTFT P95 > 1.5 s for 5 min

Severity: CRITICAL (contractual SLO violation).

Immediate mitigation. Quick diagnosis of the regime (in order of likelihood):

If a v2 canary is active and the ratio ttft_p95(v2)/ttft_p95(v1) > 1.30: automatic rollback of the canary via Argo Rollouts (argo rollouts abort vllm-llama70b).
If num_requests_waiting > 5: scale out via KEDA.
If DRAM_ACTIVE > 90 % plus gpu_cache_usage_perc > 90 %: HBM bottleneck, reach for the quantisation lever or a context reduction.
If CLOCK_THROTTLE_REASONS != 0: see RB-02.

Evidence.

# Histogram snapshot
curl http://vllm-pod:8000/metrics | grep time_to_first_token
# Breakdown by version if a canary is running
# DCGM state at that moment
curl http://dcgm-exporter:9400/metrics | grep -E "PIPE_TENSOR|DRAM_ACTIVE|THROTTLE"
# Active traffic
kubectl top pods -n inference

Resolution. Depends on the diagnosis. Typical cases:

Canary regression → full rollback (see Canary).
Capacity saturation → scale replicas or accept a temporary 503 with Retry-After.
Prefill bound → enable or calibrate chunked prefill or disaggregated serving (see Disaggregated serving).

Closure. TTFT P95 inside SLO sustained 30 min.

Postmortem. Mandatory. Document the root cause and the lever applied; update the runbook.

Keep YAML workflows — three complete examples

Runbooks are only useful if they are codified in the workflow engine. Keep lets you declare them in YAML versioned in git.

Workflow 1 — `xid-detected.yaml`

workflow:
 id: xid-detected-drain
 name: "XID error detected — cordon node and capture evidence"
 description: "RB-03 implementation"
 triggers:
 - type: alert
 filters:
 - key: alertname
 value: GpuXidErrorDetected
 steps:
 - name: capture-evidence
 provider:
 type: bash
 with:
 command: |
 set -e
 NODE="{{ alert.labels.node }}"
 GPU="{{ alert.labels.gpu }}"
 INC_ID="{{ alert.fingerprint }}"
 mkdir -p /var/evidence/$INC_ID
 kubectl debug node/$NODE -it --image=nvcr.io/nvidia/cuda:12.4.0-base-ubuntu22.04 -- \
 bash -c "nvidia-smi -q -d ERROR,PCIE,ROW_REMAPPER > /host/var/evidence/$INC_ID/smi.txt"
 kubectl describe node $NODE > /var/evidence/$INC_ID/node.txt
 - name: cordon-node
 provider:
 type: kubernetes
 with:
 action: cordon
 name: "{{ alert.labels.node }}"
 if: "{{ alert.labels.severity == 'critical' }}"
 actions:
 - name: open-jira-ticket
 provider:
 type: jira
 config: "{{ providers.jira-prod }}"
 with:
 project: GPUOPS
 issuetype: Incident
 summary: "RB-03: XID {{ alert.annotations.xid_code }} on {{ alert.labels.node }}/{{ alert.labels.gpu }}"
 description: |
 Severity: {{ alert.labels.severity }}
 XID: {{ alert.annotations.xid_code }}
 Evidence: /var/evidence/{{ alert.fingerprint }}
 Runbook: https://runbooks.example.local/RB-03
 - name: notify-slack
 provider:
 type: slack
 config: "{{ providers.slack-gpu-incidents }}"
 with:
 message: |
 :warning: *RB-03 triggered*
 Node: `{{ alert.labels.node }}` GPU: `{{ alert.labels.gpu }}`
 XID: `{{ alert.annotations.xid_code }}`
 <{{ jira.url }}|Jira ticket>
 - name: emit-audit
 provider:
 type: kafka
 config: "{{ providers.kafka-audit }}"
 with:
 topic: audit.actions
 message:
 incident_id: "{{ alert.fingerprint }}"
 action: "cordon_node"
 actor: "keep-workflow"
 workflow_id: "xid-detected-drain"
 target: "{{ alert.labels.node }}"
 timestamp: "{{ now }}"

Workflow 2 — `ecc-dbe.yaml` — immediate paging

workflow:
 id: ecc-dbe-critical
 name: "ECC double-bit — page on-call and quarantine node"
 triggers:
 - type: alert
 filters:
 - key: alertname
 value: GpuEccDoubleBit
 steps:
 - name: cordon-immediately
 provider:
 type: kubernetes
 with:
 action: cordon
 name: "{{ alert.labels.node }}"
 - name: drain-workload
 provider:
 type: kubernetes
 with:
 action: drain
 name: "{{ alert.labels.node }}"
 options:
 ignore-daemonsets: true
 delete-emptydir-data: true
 grace-period: 120
 - name: page-oncall
 provider:
 type: pagerduty
 config: "{{ providers.pagerduty-critical }}"
 with:
 service_key: "{{ env.PD_SERVICE_KEY }}"
 severity: critical
 summary: "RB-04 ECC DBE on {{ alert.labels.node }}/{{ alert.labels.gpu }} — node drained"
 - name: emit-lifecycle
 provider:
 type: kafka
 config: "{{ providers.kafka-incidents }}"
 with:
 topic: incidents.lifecycle
 message:
 incident_id: "{{ alert.fingerprint }}"
 event: incident.opened
 severity: critical
 runbook: RB-04
 requires_postmortem: true
 - name: notify-dpo
 provider:
 type: email
 with:
 to: dpo@example.local
 subject: "ECC DBE on a production GPU — assessment required"
 body: |
 RB-04 ECC DBE incident detected on {{ alert.labels.node }}.
 Affected model: {{ alert.labels.model }}.
 Please assess whether personal/classified data was processed
 during the error window and whether GDPR art. 33 notification is needed.

Workflow 3 — `canary-rollback.yaml` — TTFT P95 out of SLO

workflow:
 id: canary-rollback-ttft
 name: "Rollback canary when TTFT P95 ratio v2/v1 > 1.30"
 triggers:
 - type: alert
 filters:
 - key: alertname
 value: VllmTtftP95OutOfSlo
 - key: canary_active
 value: "true"
 steps:
 - name: check-ratio
 provider:
 type: prometheus
 config: "{{ providers.prom-prod }}"
 with:
 query: |
 histogram_quantile(0.95, sum by(le)(rate(vllm:time_to_first_token_seconds_bucket{version="v2"}[5m])))
 /
 histogram_quantile(0.95, sum by(le)(rate(vllm:time_to_first_token_seconds_bucket{version="v1"}[5m])))
 condition: result > 1.30
 actions:
 - name: argo-rollback
 provider:
 type: kubernetes
 with:
 action: exec
 command:
 - kubectl
 - argo
 - rollouts
 - abort
 - "{{ alert.labels.rollout }}"
 - -n
 - "{{ alert.labels.namespace }}"
 - name: notify-and-audit
 provider:
 type: kafka
 config: "{{ providers.kafka-audit }}"
 with:
 topic: audit.actions
 message:
 incident_id: "{{ alert.fingerprint }}"
 action: canary_rollback
 ratio: "{{ steps.check-ratio.result }}"
 actor: keep-workflow
 timestamp: "{{ now }}"

Each workflow lives in repos/keep-workflows/ versioned in git, reviewed by pull request, validated by CI (keep workflow validate). The written runbook lives as docs/runbooks/RB-XX.md linked from the workflow YAML; the two always evolve together.

The canonical Kafka event schema

For the topics to be consumable by compliance, postmortem tooling and dashboards without each consumer having to guess the shape, the schema is fixed with Avro / Protobuf.

{
 "name": "IncidentLifecycleEvent",
 "type": "record",
 "fields": [
 { "name": "incident_id", "type": "string" },
 { "name": "event", "type": { "type": "enum", "symbols": [
 "incident.opened", "incident.acknowledged", "action.proposed",
 "action.executed", "action.failed", "incident.escalated",
 "incident.resolved", "postmortem.attached"
 ]}},
 { "name": "timestamp", "type": "string", "logicalType": "timestamp-millis" },
 { "name": "actor", "type": "string" },
 { "name": "severity", "type": { "type": "enum", "symbols": ["low","warning","critical"] } },
 { "name": "runbook", "type": ["null","string"], "default": null },
 { "name": "alert_name", "type": "string" },
 { "name": "labels", "type": { "type": "map", "values": "string" } },
 { "name": "annotations", "type": { "type": "map", "values": "string" } },
 { "name": "evidence_uri", "type": ["null","string"], "default": null },
 { "name": "requires_postmortem", "type": "boolean", "default": false }
 ]
}

For audit.actions (WORM), a separate and stricter schema with non-modifiable fields:

{
 "name": "AuditAction",
 "type": "record",
 "fields": [
 { "name": "incident_id", "type": "string" },
 { "name": "action", "type": "string" },
 { "name": "actor", "type": "string" },
 { "name": "actor_type", "type": { "type": "enum", "symbols": ["human","workflow","scheduler"] } },
 { "name": "workflow_id", "type": ["null","string"], "default": null },
 { "name": "target", "type": "string" },
 { "name": "command", "type": ["null","string"], "default": null },
 { "name": "result", "type": { "type": "enum", "symbols": ["success","failure","partial"] } },
 { "name": "timestamp", "type": "string", "logicalType": "timestamp-millis" },
 { "name": "evidence_uri", "type": ["null","string"], "default": null },
 { "name": "approver", "type": ["null","string"], "default": null }
 ]
}

The topic is configured with cleanup.policy=delete, retention.ms=15552000000 (6 months) and min.insync.replicas=2 with acks=all to guarantee durability. For longer retention without the Kafka cost, use tiered storage to Ceph RGW or an S3-compatible backend: new log in the hot tier, old log in the cold tier, transparently to the consumer.

Formal fit into incident management

Runbooks are not an isolated SRE practice. They fit into four regulatory frameworks that production LLM platforms touch daily.

ISO/IEC 27035 — information security incident management

It defines the formal cycle in five phases: plan & prepare → detect & report → assess & decide → respond → lessons learned. Each phase has outputs that must be documented. Translated to the stack:

Plan & prepare: runbooks RB-01 to RB-06 plus the Keep workflows are part of the Information Security Incident Management Plan. Versioned in git, reviewed annually.
Detect & report: the Prometheus alerts entering Kafka are the materialisation.
Assess & decide: the severity in gpu.alerts.enriched plus the Keep workflow logic.
Respond: execution of the workflow’s steps plus actions.
Lessons learned: a mandatory postmortem for the runbooks that flag it; the output documented in the postmortem repo plus the runbook update.

ENS (Esquema Nacional de Seguridad) — op.exp controls

op.exp.7 Incident management: the runbook catalogue plus the Keep / Kafka pipeline materialise the “organised and procedural response”.
op.exp.8 Activity logging: the audit.actions topic with WORM retention of 6 months (minimum for level ALTO).
op.exp.9 Incident management logging: the incidents.lifecycle topic with the full cycle of each incident.
op.exp.10 Protection of activity logs: WORM plus encryption at rest plus access control (compliance consumers read-only).

NIS2 — notification to the competent authority

For essential and important entities, art. 23 sets three deadlines from the detection of significant impact:

24 hours: an early warning to the national CSIRT (INCIBE-CERT in Spain).
72 hours: a formal notification with an initial assessment.
1 month: a final report with root cause, impact and corrective measures.

The data for those reports comes straight out of incidents.lifecycle plus audit.actions with a consumer that generates the dossier in the required format. Without the auditable pipeline, the NIS2 deadlines are unreachable.

EU AI Act — art. 73 (serious incident reporting)

Applicable to high-risk systems. Deadlines:

2 days: for incidents causing death or irreversible harm to people or critical infrastructure.
10 days: for incidents producing serious disruption of critical infrastructure.
15 days: for all other “serious incidents”.

The definition of “serious incident” includes systematic model failures, a fundamental rights breach, material or environmental harm. Runbooks must flag which alerts can lead to a serious incident (typically anything affecting the model’s output in a high-risk context) and fire a specific legal assessment sub-workflow.

ISO/IEC 42001 — AIMS clause 10 continual improvement

The mandatory post-incident postmortem feeds clause 10. Updating the runbook after every incident that reveals a new pattern is the “corrective action with verification of effectiveness” the standard demands. See ISO 42001 AIMS.

Four anti-patterns

Anti-pattern 1 — alerts without a runbook. The alert fires, the junior operator on call looks at the dashboard, searches Confluence, finds nothing up to date, calls the senior on Slack, waits 20 minutes. In that time the incident has grown. Rule: no alert goes to production without a published runbook and an approved Keep workflow. CI validates that every PrometheusRule with severity ≥ warning has its corresponding keep workflow.

Anti-pattern 2 — a runbook without prior evidence capture. The workflow runs nvidia-smi --gpu-reset as soon as the XID arrives, losing the state that would have diagnosed the root cause. The next identical XID forces the diagnosis to be redone from scratch. Rule: steps before actions; all evidence is captured first, destructive actions come after.

Anti-pattern 3 — escalation by seniority rota instead of severity. The junior operator on call handles an ECC DBE because “it is their turn”. They lack the context to understand row remap, retired pages or the data corruption risk. Rule: page by severity, not by rota: RB-04 and RB-03 page the senior primary ON-CALL with automatic escalation to infra/hardware if there is no acknowledgement within 10 min.

Anti-pattern 4 — no human gate for destructive actions. The workflow runs kubectl drain automatically on any alert marked CRITICAL. On the first false alarm (a transient that resolved itself in 30 s), Keep drained a production node during peak hour. Rule: destructive actions (drain, reset, RMA, full rollback) require human confirmation via a Slack interactive message, with a configurable timeout. A justified exception: ECC DBE confirmed by > 1 measurement, where the corruption risk outweighs the false alarm risk.

Applied to typical on-premise hardware

For a generic cluster of 4 nodes × 4×H100 SXM 80 GB with Kafka and Keep already deployed:

Kafka: a 3-broker cluster on non-GPU nodes of the K8s cluster; topics gpu.alerts.enriched, incidents.lifecycle, audit.actions configured with replication factor 3, min.insync.replicas 2. Audit with tiered storage to Ceph RGW for retention > 6 months without a brutal cost.
Keep: 2 operator replicas plus 1 worker replica in a keep namespace; connected to Prometheus (read provider), Kafka (read plus write provider), Slack, PagerDuty, Jira, Kubernetes (provider with a dedicated SA holding get/list/patch nodes and create jobs permissions).
Workflows: ~25-40 YAML files in the infra/keep-workflows/ repo, synchronised with the cluster via Flux or Argo CD. Validated by CI (keep workflow validate) on every PR.
Event volume: for 16 GPUs in normal operation with debounced alerts, ~50-200 events/day on gpu.alerts.enriched. In a typical incident, peaks of 500-2,000 events/day.
Compliance consumers: a python consumer in the compliance namespace that generates NIS2 / ENS / EU AI Act reports weekly, reading audit.actions and incidents.lifecycle.

What we have not covered (upcoming posts)

Postmortem playbooks — the mechanics of RCA with 5-whys, Ishikawa adapted to LLMs, integration with MLflow tracking for re-training if the postmortem produces an enriched dataset.
Chaos engineering for LLMs — controlled injection of XID errors, simulated ECC, artificial HBM latency to validate runbooks before the real incident.
Multi-cluster incident coordination — how to coordinate Keep across geographic clusters when an incident affects multiple regions.
CMDB and procurement integration — the RMA → ticket → ServiceNow → hardware replacement cycle automated via workflow.
LLM forensics — extracting the full OTel trace of a request affected by an incident, with redacted PII, kept in an evidence vault.

References

ISO/IEC 27035-1:2023 — Information security incident management — Principles and process.
ISO/IEC 27035-2:2023 — Information security incident management — Guidelines to plan and prepare for incident response.
ENS — Real Decreto 311/2022, Anexo II controles op.exp.7 a op.exp.10.
Directiva NIS2 (UE 2022/2555) — art. 23 (notificación de incidentes significativos).
Reglamento EU AI Act (UE 2024/1689) — art. 73 (reporting of serious incidents).
ISO/IEC 42001:2023 — AI management system — cláusula 10 (mejora continua).
Keep project — keephq.dev y github.com/keephq/keep (documentación de workflows YAML, providers).
Apache Kafka — Tiered Storage y cleanup.policy (docs.confluent.io / kafka.apache.org).
Confluent — Schema Registry y best practices para eventos lifecycle.
NVIDIA — Xid Errors Documentation y procedimientos de remediación.
Google SRE Book — Effective Troubleshooting y Postmortem Culture.
Atlassian — Incident Management Handbook (referencia para severity matrices).

Mixed NVIDIA + Intel environments for LLM inference: from the central H100 cluster to the NUC in the branch office

Tue, 02 Jun 2026 04:30:00 +0200

This post complements those on Capacity planning for on-premise LLM inference (which assumed a pure NVIDIA cluster), Seven layers of the stack (which also did not get into hardware heterogeneity) and The LLM inference router (where capability routing makes full sense once there is mixed hardware). It is the missing piece for talking about “hardware sovereignty” without reducing it to “which vendor do I pick”.

TL;DR

A production LLM inference cluster in 2026 can stop being monolithically NVIDIA if it accepts heterogeneity as an architectural decision. The motivation is not theory but three measurable operational advantages. (1) Cost: an Intel Xeon 6 with AMX (Advanced Matrix Extensions) delivers 7B INT4 at ~80 tok/s while serving embeddings and reranking at a fraction of the cost of dedicating an H100 to that task; the capacity planning adds up better with an Intel CPU handling the cheap work and an NVIDIA H100 the big LLM. (2) Sovereignty and supply chain diversification: NVIDIA holds roughly 94 % of the AI accelerator market (November 2025), single-vendor dependency with all its risks; Intel manufactures in Europe (Leixlip operational, Magdeburg planned) against a design-only NVIDIA with TSMC as foundry, which for a Spanish or European organisation subject to ENS / NIS2 / EU AI Act is a real hedging argument. (3) Edge: an Intel NUC with a Lunar Lake CPU (48 TOPS NPU) or Panther Lake (50 TOPS NPU + 120 TOPS Xe3 = 180 TOPS platform) runs 7B INT4 models at usable speed, which opens up the “branch office with local inference + central DC for complex cases” pattern. Relevant Intel hardware in June 2026: Intel Gaudi 3 (128 GB HBM2e, 1,835 TFLOPS BF16/FP8, 3.67 TB/s; a direct competitor to the H100, Intel claims +20 % on Llama 2 70B but Signal65 published H200 9× over Gaudi 3 on Llama 3.1 405B, you have to cite both; Falcon Shores cancelled in January 2025, Jaguar Shores 2026 as the restart bet, Gaudi 4 confirmed as never happening); Intel Xeon 6 with AMX (up to 288 E-cores on Sierra Forest or 86 P-cores on Granite Rapids, 1,024 FLOPS BF16/cycle/core with AMX, Intel claims 2.7× tok/s vs EPYC 9965 on the vLLM CPU backend); Intel Arc Pro B60 (Battlemage, 24 GB GDDR6, 456 GB/s, 197 TOPS INT8, launched September 2025, with a dual-GPU 48 GB variant and a “Battlematrix” rack with 8× = 192 GB VRAM); Intel NUC with NPU (Lunar Lake 48 TOPS, Arrow Lake similar, Panther Lake 50 TOPS at CES 2026; realistic for 7-13B INT4, not for the 30-70B Intel claims in its marketing). Software: OpenVINO 2025.3 with the GenAI API and vLLM-OpenVINO; IPEX-LLM with integrations into llama.cpp, vLLM, HF, LangChain; vLLM CPU backend with AMX; llama.cpp SYCL (better than Vulkan on Arc). Four canonical patterns: embeddings + reranker on Intel alongside the LLM on NVIDIA; guardrails + PII redaction on a near-edge NUC; speculative drafter on a NUC near the user and target on the H100; NUC dev workstations. Unified observability via DCGM + habana-metric-exporter + intel-gpu-exporter + Intel PCM federated in Prometheus. Pitfalls: tokenizer mismatch between engines, edge↔central round-trip latency, FP8 Hopper ≠ INT8 AMX in quality, version synchronisation. Applied to a generic cluster: central DC 4×H100 SXM + Xeon 6 AMX sidecar + 6-12 Intel NUCs in branch offices. Critical disclaimer: as of June 2026 there are no publicly verifiable cases of a mixed NVIDIA + Intel deployment in European banking or government; the pattern is emergent architecture and worth recommending, not established practice with an industrial track record.

You are here: DEPLOY (with heterogeneity as a decision)

The analogy: the factory with several different machines

A serious factory has several machines with different purposes, not one universal machine. A 200-tonne hydraulic press for heavy stamping; a bench lathe for turned parts; a 3D printer for quick prototypes; a pick-and-place robot for SMD. Each machine does what it does better than the others in its niche, and the plant manager sizes the mix according to the real product portfolio, not according to fashion. Buying three hydraulic presses because “they are the most impressive” when 60 % of the work is turned parts wastes capital: the lathe is cheaper, faster in its niche and frees the press for what genuinely needs it.

An LLM inference cluster with an NVIDIA H100 dedicated to computing embeddings for a RAG corpus is using a hydraulic press to drill bolts. The H100 is magnificent for a 70B LLM in BF16 with concurrency of 40+; for embeddings of an 800-token document with bge-m3, what you need is an Intel Xeon 6 with AMX at a fraction of the cost and power draw. A cluster that wants to serve light guardrails (Llama Guard 4 8B) on every request, with a 50 ms budget, does not need that guardrail on an H100 either: an Intel NUC with a 48 TOPS NPU covers the case with room to spare.

The heterogeneous factory is not theoretical elegance: it is maximising the useful utilisation of fixed capital. Nor is the heterogeneous LLM inference cluster.

Three operational reasons for heterogeneity

Reason 1 — cost

An H100 SXM 80 GB running 24/7 draws ~700 W (real measurement at the wall ~697 W with vLLM Llama 3.1 405B batch=4) and represents between 25,000 € and 35,000 € of amortised hardware. An Intel Xeon 6 with AMX (Granite Rapids 86 cores or Sierra Forest 288 E-cores) draws 350-500 W for the socket and costs a fraction. The operational split: the H100 is reserved for the big LLM (Llama 70B BF16 or FP8, where its HBM3 and FP8 tensor cores are worth their weight); the Xeon AMX absorbs embeddings (bge-m3, e5-large), reranking (bge-reranker-v2-m3), small models (Llama 3.2 1B / 3B INT4) and offline batch processing. It is the same logic as the capacity planning taken one step further: instead of budgeting KV cache VRAM only on the H100, budget each workload on the silicon where its arithmetic intensity fits best.

Reason 2 — sovereignty and supply chain diversification

As of November 2025, NVIDIA holds approximately 94 % of the AI accelerator market. That concentration is risk. For an organisation subject to ENS / NIS2 / EU AI Act, depending on a single supplier whose foundry is concentrated in Taiwan (TSMC) introduces supply chain vulnerabilities that recent regulations (NIS2, supply chain provisions) are starting to require you to document and mitigate. Intel diversifies: it has its own fabs in Europe (Leixlip operational in Ireland; Magdeburg planned in Germany, with EU Chips Act funding), which for a Spanish or European public sector client is a real contractual argument, not marketing.

Mandatory disclaimer: the Intel roadmap after Falcon Shores is unstable. Intel cancelled Falcon Shores in January 2025 and relegated Gaudi 4 to “will never exist”; the restart bet is Jaguar Shores in 2026 as a rack-scale platform, still with no confirmed public specs. Diversification is strategically correct, but assuming continuity of the Intel roadmap at NVIDIA’s level in 2026 would be naive. The operational strategy: Intel for workloads where lock-in is lower (CPU for embeddings, NUC for light edge, replaceable by AMD/Apple/SiFive if Intel pivots again), NVIDIA for the big LLM where the maturity of the software stack still has no rival.

Reason 3 — edge

The “everything travels to the central DC” pattern breaks in three cases: latency (branch office 100+ ms away from the DC, unacceptable for chat), data sovereignty (prompts with personal or classified data that must not leave the local perimeter), and offline operation (branch office with intermittent connectivity). The Intel NUC with a modern CPU (Lunar Lake / Arrow Lake / Panther Lake) brings 48-50 TOPS NPU + 100-180 TOPS Xe2/Xe3 iGPU in a box of 0.5-1.5 L volume drawing 30-65 W. 7B INT4 models run at usable speed; with more aggressive quantisation (Q3_K) a Llama 13B fits. For branch offices with RAG over a local corpus + a 7B LLM + guardrails, the NUC is perfect.

Relevant Intel hardware (June 2026)

Part	Memory	Key performance	Launch	Status
Intel Gaudi 3	128 GB HBM2e, 3.67 TB/s	1,835 TFLOPS BF16/FP8; 1,200 GB/s networking	Apr 2024	Active; successor Jaguar Shores 2026 (not Gaudi 4)
Intel Xeon 6 (Granite Rapids)	DDR5 + MRDIMM	86 P-cores, AMX 1,024 FLOPS BF16/cycle/core	2024-2025	Active
Intel Xeon 6 (Sierra Forest)	DDR5	288 E-cores	2024	Active
Intel Arc Pro B60 (Battlemage)	24 GB GDDR6, 456 GB/s	197 TOPS INT8; 12.28 TFLOPS FP32	Sep 2025	Active; dual 48 GB variant, rack 8× = 192 GB
Intel Data Center GPU Max	128 GB HBM	successor to Ponte Vecchio	discontinued	Discontinued Jan 2026
Intel NUC (Lunar Lake)	DDR5x	NPU 48 TOPS + Xe2 67 TOPS = 120 TOPS platform	2024	Active
Intel NUC (Arrow Lake)	DDR5	NPU 13 TOPS + Xe iGPU	2024	Active (less NPU than Lunar)
Intel NUC (Panther Lake)	DDR5x	NPU 50 TOPS + Xe3 120 TOPS = 180 TOPS platform	CES Jan 2026	Rolling out

Intel Gaudi 3 — the critical note on the marketing

Intel publishes that Gaudi 3 delivers +20 % throughput vs H100 on Llama 2 70B inference and 2× price/performance. The figure appears in the official whitepaper and in launch presentations. However, Signal65 (an independent firm) published in 2025 that the H200 beats Gaudi 3 by a factor of 9× on Llama 3.1 405B. The discrepancy matters: both numbers can be true for their specific benchmarks (Llama 2 70B FP16 vs Llama 3.1 405B FP8) but the operational conclusion changes radically depending on which one you keep.

This post’s recommendation: treat Gaudi 3 as a valid option for Llama-class 70B in BF16/FP8 where Intel claims parity or advantage, not for frontier 200B+ models where NVIDIA keeps a clear margin. And consider the roadmap risk: Gaudi 4 will never exist; the line’s successor is Jaguar Shores 2026 with a completely new rack-scale architecture, a discontinuity rather than an evolution.

Intel Xeon 6 with AMX — the CPU workhorse

The Advanced Matrix Extensions (AMX) are the non-obvious piece. Each Granite Rapids P-core executes up to 1,024 BF16 FLOPS per cycle via AMX, which turns a Xeon 6 with 64-86 cores into a respectable matrix accelerator for small and mid-sized models. Real reported figures: Llama 3.2 INT4 at ~57 tok/s with AMX vs 28 tok/s without AMX (a clean 2× factor). For serving 7B INT4 with the vLLM CPU backend + AMX, Intel claims 2.7× tok/s vs EPYC 9965, a figure with Intel bias but qualitatively corroborated by LMSYS in their DeepSeek R1 671B deployment on Xeon 6 + SGLang.

Operational use case: embeddings and reranking on a Xeon 6 sidecar next to the H100 cluster. Models such as bge-m3 (multilingual embedding) or bge-reranker-v2-m3 run at acceptable throughput on CPU AMX; they do not deserve a dedicated H100. Freeing the H100 for the 70B LLM raises the cluster’s effective RPS without buying more GPUs.

Intel Arc Pro B60 and Battlematrix

Launched in September 2025, the Arc Pro B60 (Battlemage) brings 24 GB GDDR6 with 456 GB/s of bandwidth and 197 TOPS INT8 at 200 W. Maxsun has a dual-GPU 48 GB variant. The “Battlematrix” rack configuration combines 8 units = 192 GB of aggregate VRAM. The interesting point: at a cost far below an H100 SXM 80 GB, which makes it a candidate for 30-70B INT4-INT8 LLMs served via OpenVINO or llama.cpp SYCL.

Phoronix verified that on SYCL the Arc Pro B70 reaches parity with the Radeon PRO W7900 (previous AMD generation) on DeepSeek R1 Llama 8B pp512. The Vulkan backend loses heavily (~1/4 of SYCL performance); for Arc Pro, always SYCL.

Intel NUC with NPU — the edge node

Intel NUCs with a Lunar Lake CPU (Core Ultra Series 2) bring NPU 4 with 48 TOPS and a platform total of 120 TOPS adding the Xe2 iGPU and CPU AVX. Panther Lake (CES January 2026) goes up to NPU 5 = 50 TOPS + Xe3 120 TOPS = 180 TOPS platform.

Intel states that Panther Lake “runs 30-70B models locally”. Reality check: it is marketing. A 30-70B INT4 fits in RAM (DDR5x 32-64 GB) but sustained speed with Q4_K_M quantisation on a NUC hovers around 2-8 tok/s; comfortable for occasional use, not for serving traffic. The NUC’s real sweet spot is 7B INT4 at 20-40 tok/s on the iGPU/NPU, perfect for a client branch office with casual queries.

Intel software — the relevant stack

OpenVINO 2025.3 (June 2026) is the central piece. It supports single-command deployment via the OVMS CLI with automatic download from the HF Hub; it integrates OpenVINO GenAI with a C++/Python API for generative pipelines; it exposes a vLLM v1-compatible API (vLLM-OpenVINO). GGUF model support: DeepSeek Distill, Qwen 2/2.5, Llama 3. Optimisations: Sage Attention (first token with long prompts), per-channel KV-cache compression.

Intel Extension for PyTorch (IPEX), XPU version 2.8.10+xpu, adds Intel backends to PyTorch. IPEX-LLM is the subproject that integrates with llama.cpp, Ollama, HuggingFace, LangChain, LlamaIndex, vLLM and DeepSpeed. May 2025: it ran DeepSeek V3/R1 671B and Qwen3MoE 235B on 1-2 Arc A770/B580 cards with FlashMoE.

vLLM CPU backend, the CPU branch of vLLM with AMX optimisations. For 7B INT4 on a 4th-gen Xeon with AMX: 12-50 tok/s; with a Xeon Gold 6530 + INT4: ~80 tok/s. Academic figures (arXiv 2410.04466).

llama.cpp SYCL, the recommended backend for Arc; Vulkan works but hovers around 1/4 of SYCL performance on an Arc B580. SYCL reaches parity with the previous AMD generation.

Habana SynapseAI, the Gaudi 3 stack. The PyTorch bridge habana_frameworks.torch registers the hpu device; it integrates with torch.compile. It is not a complete port to oneAPI but a partial integration via oneMKL. That implies the Gaudi ecosystem keeps a certain separation from Intel’s general oneAPI, relevant in view of the hypothetical Jaguar Shores and future unification.

The four canonical patterns

Pattern 1 — embeddings + reranker on Intel

The most mature and the easiest to adopt. In a typical RAG system, each user request invokes:

Query embedding (50 ms on an H100, 80 ms on a Xeon AMX, 30 ms on an Arc Pro B60).
Vector search (Qdrant / Milvus / Chroma; latency ~10-30 ms).
Reranker over the top-k candidates (60 ms on an H100, 100-150 ms on a Xeon AMX).
LLM over the augmented prompt (200-500 ms TTFT, 30-50 ms/token).

Steps 1 and 3 are memory-bound and relatively small (100M-1B models): a Xeon 6 with AMX (the Arc Pro B60 is faster but is already a dedicated GPU) does the job at a hardware cost that is a fraction of a dedicated H100. Step 4 stays on NVIDIA because that is where its tensor architecture + HBM3 + FP8 is worth what it costs.

Operational implication: a Xeon 6 sidecar (~40 cores, ~10-15 k€) serving embeddings + reranking frees the equivalent of 1-2 H100s of “cheap” load, recovering that capacity for the big LLM. Clear ROI in sizing.

Pattern 2 — guardrails + PII redaction on a near-edge NUC

A bank branch, a medical practice or a law office generates prompts with personal or classified data. Sending those prompts to the central DC (even if it is corporate on-premise) can clash with local retention policies or with specific compliance (GDPR, professional privilege).

The pattern: the NUC in the branch office runs two critical steps before the round-trip:

PII redaction with Presidio (CPU-only, fast) or Llama Guard 4 8B on the NUC’s NPU + iGPU. It replaces names, tax IDs and account numbers with placeholders.
Light guardrails (PromptGuard 2 86M, Llama Guard 4 8B) on the NPU + iGPU. It filters prompt injection, jailbreaks and prohibited content.

Only afterwards does the redacted prompt travel to the central DC so the big LLM can answer. The response comes back to the NUC, which rehydrates the placeholders with the real values before showing it to the user. The sensitive data never leaves the branch office.

Costs: a Panther Lake NUC is ~1,500-2,500 € per unit, scalable to dozens of branch offices with no additional central GPU cost. Latency: 50-150 ms for the edge step before the DC’s 200-500 ms round-trip.

Pattern 3 — speculative decoding drafter on a NUC

Speculative decoding uses a small drafter that proposes γ tokens and a large target that accepts or rejects them in a single forward pass. If the drafter is geographically close to the user (a NUC in the branch office) and the target is in the central DC, the client’s perceived latency falls further still.

Setup: drafter Llama 3.2 1B INT4 on a NUC + target Llama 3.1 70B FP8 on the central H100. The NUC generates γ=4 tokens in ~50 ms locally; the target verifies them in one pass (40-80 ms including round-trip); if the acceptance rate is > 60 %, effective TTFT falls ~50 % compared with Llama 70B without speculative decoding.

An important constraint: drafter and target must share a tokenizer. Llama 3.2 1B and Llama 3.1 70B have compatible tokenizers. Mixing a Llama drafter with a Qwen target breaks the pattern.

Pattern 4 — NUC dev workstations

Dev / CI iterating on prompts, evals and retrieval logic does not need a production GPU to validate correctness. A NUC with Llama 3.2 3B INT4 runs the end-to-end functional tests (including embeddings + retrieval + LLM + guardrails) at a tenth of the cost of iterating on an H100. Only the last pre-prod smoke test uses the production cluster.

A mature pattern in organisations with many developers and scarce production GPU. Iteration that is 10× faster and cheaper translates into feature delivery speed.

Unified observability in a heterogeneous cluster

The GPU observability post covered DCGM Exporter for NVIDIA. In a mixed cluster you need more:

Hardware part	Exporter	Key metrics
NVIDIA H100/A100	`nvidia/dcgm-exporter`	DCGM_FI_DEV_* + DCGM_FI_PROF_*
Intel Gaudi 3	`HabanaAI/habana-metric-exporter`	habana_hpu_utilization, habana_hbm_used
Intel Arc Pro	`intel/intel-gpu-exporter` (unofficial; alternatives exist)	xe_engine_utilization, xe_memory_used
Intel Xeon CPU + AMX	`prometheus/node-exporter` + Intel PCM	cpu_amx_utilization (via PCM)
Intel NUC (NPU+iGPU)	`intel/intel-gpu-exporter` + custom NPU exporter	npu_utilization, xe_iGPU

All federated into a single Prometheus + Grafana. The dashboards are organised by hardware family (NVIDIA, Intel server, Intel edge) plus an aggregate “heterogeneous cluster” view with per-tenant SLOs combining all four.

Cardinality: ~1.5-2× that of a pure NVIDIA cluster. Manageable with Thanos / Mimir for long retention.

Capability routing — from the L7 router to the heterogeneous one

The LLM inference router stops being a version selector for the same model and becomes a capability dispatcher:

models:
 - name: "llama-70b-chat"
 endpoint: "vllm-llama70b.inference.svc:8000"
 backend: nvidia-h100
 capabilities: [chat, tool_use, json_mode]
 - name: "embedding-multilingual"
 endpoint: "ipex-bge-m3.inference.svc:8080"
 backend: intel-xeon-amx
 capabilities: [embeddings]
 - name: "reranker-multilingual"
 endpoint: "ipex-bge-reranker.inference.svc:8080"
 backend: intel-xeon-amx
 capabilities: [reranking]
 - name: "guardrail-prompt-injection"
 endpoint: "openvino-llama-guard.edge-suc01.local:8080"
 backend: intel-nuc-edge
 capabilities: [guardrails, redact-pii]
 region: sucursal-01
 - name: "llama-3b-draft"
 endpoint: "openvino-llama-3b.edge-suc01.local:8080"
 backend: intel-nuc-edge
 capabilities: [speculative-drafter]
 region: sucursal-01
 target_model: "llama-70b-chat"

The router resolves model=embedding-multilingual → Intel Xeon; model=llama-70b-chat → H100; model=guardrail-prompt-injection with region=sucursal-01 → local NUC. If the branch office NUC goes down, failover to an equivalent replica in the central DC, accepting the latency cost.

LiteLLM Proxy, NVIDIA Dynamo and Envoy AI Gateway support this capability routing. The non-obvious piece: the router has to know the compatible tokenizer between drafter and target for pattern 3, which is modelled in extra catalogue metadata.

Specific pitfalls

Tokenizer mismatch between engines. OpenVINO with a Llama 3.2 GGUF and vLLM with the same nominal Llama 3.2 can use slightly different tokenizers (chat template, special tokens). Validate token identity with tokenizer.encode("hola") on both sides before assuming interchangeability. For speculative decoding, a single different token breaks the pattern.

Edge ↔ central round-trip latency. Patterns 2 and 3 assume the NUC and the DC are on the same corporate WAN with controlled latency. If the branch office is on 4G/5G with 100-200 ms of jitter, the speculative drafter compensates for nothing; on the contrary, it adds latency. Measure before promising.

FP8 Hopper ≠ INT8 AMX in output quality. The operator assumes a request that runs FP8 on an H100 and INT8 on a Xeon AMX will produce the same output. That is not true: the two quantisations have different degradation profiles. If the system expects idempotence (for example, evals with golden output), validate offline that the Intel version reproduces the expected behaviour within tolerance.

Model version synchronisation between sites. The model in the central DC gets updated, but the branch office NUCs keep the old version of the drafter or the guardrail for weeks. Result: divergent behaviour between branches with no easy diagnosis. Policy: the central model and the edge model advance together or within a documented window; the canary extends to the NUC fleet.

Unstable Intel roadmap. Falcon Shores cancelled, Gaudi 4 will never exist, Jaguar Shores 2026 still with no confirmed public specs. Buying Gaudi 3 today is reasonable if the use case justifies the 18-24 months of amortisation; committing an architecture for 5+ years on an Intel accelerator is a riskier bet than the NVIDIA equivalent, at least until Jaguar Shores materialises with a mature software stack.

Absence of public production deployments. As of June 2026, the confirmed Gaudi 3 deployments are IBM Cloud, Dell AI Factory and a handful of early adopters (Bharti Airtel, Bosch, Naver). There is no publicly verifiable case of a mixed NVIDIA + Intel cluster in European banking or government. This pattern is recommended emergent architecture, not practice with an industrial track record. The first adopter takes on a validation cost that a second adopter avoids.

Applied to a generic on-premise cluster

For an organisation with a generic heterogeneous LLM inference cluster:

Central DC: 4 nodes × 4×H100 SXM 80 GB with intra-node NVLink = 16 H100s. Serves the big LLMs (Llama 70B, Mixtral 8×22B, Qwen 72B) in BF16 or FP8.
Xeon 6 sidecar: 2-4 Xeon 6 servers (Granite Rapids 64-86 cores) with AMX, 512 GB DDR5, in the same rack as the H100 cluster. Serves embeddings (bge-m3), reranking (bge-reranker-v2-m3) and small models (Llama 3.2 1B/3B) on the vLLM CPU backend with AMX.
Arc Pro sidecar (optional): 1-2 servers with 4-8× Arc Pro B60 24 GB each (Battlematrix), for 13-30B INT8 models via OpenVINO. Useful if the cost per mid-sized LLM has to come down from the H100.
Edge NUCs in branch offices: 1-2 Panther Lake NUCs per branch, with a 50 TOPS NPU + Xe3 120 TOPS, serving Llama Guard 4 + Presidio + a Llama 3.2 1B INT4 drafter via OpenVINO. Corporate WAN connectivity with latency < 80 ms to the DC.

Estimated envelope: central cluster ~120 kW of GPU peak + ~10-15 kW of Intel sidecars. Edge: ~50 W per NUC, negligible compared with office costs.

Observability: federated Prometheus in the DC + pull scrape from the NUCs (via corporate VPN). Dashboards “NVIDIA GPU fleet”, “Intel server fleet”, “Intel edge fleet” plus a “consolidated SLO” view.

Router: LiteLLM Proxy or NVIDIA Dynamo in the DC, with an extended model catalogue to include Intel backends and regions (sucursal-01, sucursal-02, …). Documented edge→central failover.

What we have not covered (upcoming posts)

Reproducible benchmarks of Llama 70B on Gaudi 3 vs H100 SXM on equivalent hardware, the missing material for making decisions with your own data rather than Intel’s or Signal65’s.
AMD ROCm in the mix: how MI300X / MI355X fit into this heterogeneous pattern and what changes in the router catalogue.
Apple Silicon as edge: M3/M4 Max with a Neural Engine of ~38 TOPS + a 40-core GPU, hardware equivalent to the Panther Lake NUC but with a different software stack (MLX).
Energy cost optimisation: how nvidia-smi -pl 500W + an Intel TDP cap on Xeon 6 cuts the bill by 25-30 % with a 15-20 % throughput loss.
Model CI/CD for an edge fleet: how a rolling update of a Llama Guard reaches 50 branch office NUCs without any of them losing service.

References

Intel Gaudi 3

Intel — Gaudi 3 AI Accelerator White Paper. https://cdrdv2-public.intel.com/817486/gaudi-3-ai-accelerator-white-paper.pdf
Intel — Hot Chips 2024 Gaudi 3 deep dive. https://hc2024.hotchips.org/assets/program/conference/day1/60_HC2024.Intel.RomanKaplan.Gaudi3-0826.pdf
Signal65 / DataCenterDynamics — NVIDIA H200 outperforms Intel Gaudi 3 by factor of 9× across first Llama 3.1 405B benchmark test. https://www.datacenterdynamics.com/en/news/nvidia-h200-outperforms-intel-gaudi-3-by-factor-of-nine-across-first-llama-31-405b-benchmark-test-exclusive/
IEEE Spectrum — Intel Gaudi 3 review. https://spectrum.ieee.org/intel-gaudi-3
Tom’s Hardware — Intel cancels Falcon Shores GPU; Jaguar Shores to be successor. https://www.tomshardware.com/tech-industry/artificial-intelligence/intel-cancels-falcon-shores-gpu-for-ai-workloads-jaguar-shores-to-be-successor

Intel Xeon 6 + AMX

Intel — Xeon 6 (Granite Rapids) Product Brief. https://www.intel.com/content/dam/www/central-libraries/us/en/documents/2025-02/xeon-6-granite-rapids-product-brief.pdf
OpenMetal — Intel AMX AI Inference Performance. https://openmetal.io/resources/blog/intel-amx-ai-inference-performance/
LMSYS — Intel Xeon 6 + SGLang for DeepSeek R1 671B. https://www.lmsys.org/blog/2025-07-14-intel-xeon-optimization/
arXiv 2410.04466 — CPU-LLM benchmarks with AMX.
Intel community blog — Accelerating vLLM Inference on Intel Xeon 6 Processor.

Intel Arc Pro Battlemage

Intel — Arc Pro B60 Graphics Specifications. https://www.intel.com/content/www/us/en/products/sku/243916/intel-arc-pro-b60-graphics/specifications.html
StorageReview — Intel Arc Pro B60 Battlematrix Preview: 192GB VRAM for On-Premise AI. https://www.storagereview.com/review/intel-arc-pro-b60-battlematrix-preview-192gb-of-vram-for-on-premise-ai
Phoronix — Intel Arc Pro B-series review. https://www.phoronix.com/review/intel-arc-pro-b-series

Intel NUC / NPU

HotHardware — Intel CES 2026 Panther Lake is a Go. https://hothardware.com/news/intel-ces-2026-panther-lake-is-a-go
TechPowerUp — Intel Panther Lake Technical Deep Dive.
arXiv 2412.11053 — NITRO: LLM inference on laptop NPU.
Intel — AI PC brings larger LLM development to your desk.

Software

OpenVINO — Release Notes 2025.3. https://www.intel.com/content/www/us/en/developer/articles/release-notes/openvino/2025-3.html
HuggingFace — Deploy with OpenVINO. https://huggingface.co/blog/deploy-with-openvino
Intel — Intel Extension for PyTorch XPU 2.8.10. https://intel.github.io/intel-extension-for-pytorch/xpu/latest/tutorials/releases.html
IPEX-LLM — github.com/intel/ipex-llm.
Habana — SynapseAI PyTorch Theory of Operations. https://docs.habana.ai/en/latest/PyTorch/PyTorch_Gaudi_Theory_of_Operations.html

Market context

MLCommons — MLPerf Inference v6.0 benchmark results. https://www.spheron.network/blog/mlperf-inference-v6-benchmark-results-2026/
Intel newsroom — Gaudi 3 Expanded Availability. https://newsroom.intel.com/artificial-intelligence/intel-gaudi-3-expands-availability-drive-ai-innovation-scale

Sources: the full URLs are linked inline on each reference.

Anatomy of the twelve DCGM and five vLLM metrics: analogies, documented anomalies and real cases 2024-2026

Tue, 02 Jun 2026 04:00:00 +0200

This post goes deeper into the list of metrics presented in GPU observability for LLM inference. There each metric got its G/A/R threshold and PromQL query; here each one gets its explanatory analogy and the anomaly documented in production with a referenced public case. It is the post worth having open when an alert fires and you still do not know what to do with it; the next post on runbooks translates each anomaly into concrete action.

TL;DR

The twelve DCGM metrics (compute, memory, thermal-power, health) and the five vLLM engine metrics (concurrency, KV pool, SLO latencies) covered in the previous post paint the cluster’s cockpit, but a list without context does not teach you to diagnose. Each metric has a recurring anomalous pattern documented in the public literature (academic papers, GitHub issues, OEM KBs, operator blogs) that the veteran operator recognises instantly and the junior one does not. This post develops each metric with an analogy of its own that pins down which question it answers, and with the statistically relevant anomaly backed by figures from documented incidents. Three examples of the calibre involved: Meta published that during the training of Llama 3 405B on 16,384 H100s there were 419 unplanned failures in 54 days, one every 3 hours, with GPU + HBM3 accounting for 47 % of the total; the paper Story of Two GPUs (arXiv 2503.11901) quantifies that the H100 has 3.2× worse MTBE from uncorrectable ECC than the A100, attributable to the higher density of HBM3; issue vllm#16300 documents that on a cluster of 8×A100 80 GB TP=8 delivers worse throughput than TP=4 because NVLink saturation kills the partition speedup. The KBs Dell 000220508 and Lenovo HT514380 formalise the recurring HW Power Brake case in H100 racks overcommitted at PDU level. Issue vllm#25677 showed chunked prefill 10× slower than without it on Qwen3-30B-A3B (bad calibration of max_num_batched_tokens). Issue vllm#11912 documents a TPOT regression from 15.7 ms to 25.7 ms crossing version 0.6.4. Each case includes a verifiable URL. The operational rule: when an alert arrives, look first at the anomalous pattern associated with the metric that fired, before opening the request’s trace; 80 % of degradations match one of the documented patterns.

You are here: OBSERVE — the diagnosis layer

Family 1 — Compute

`DCGM_FI_PROF_SM_OCCUPANCY` — is there parallel work in the engines?

The analogy. An industrial kitchen with 32 hobs and a single chef. The metric answers “how many hobs have a pan on them right now?”. If half are empty, the kitchen is under-used: the orders come out one after another because the chef does not open up parallelism. If all of them are occupied but the chef is standing still watching a stopwatch, the hobs are on but nothing is being cooked (a pathological kernel saturating SMs without doing useful work).

The documented anomaly. The best-known trap: high SM occupancy does not imply real throughput. The article “GPU Utilization Is a Counter, Not a Cause” (Ingero, May 2026) put it in one exact sentence: “a kernel running at 5 % of peak FLOPS for 100 ms still reads 100 % on SM_ACTIVE”. In MoE workloads the effect turns pathological: overloaded experts produce the Straggler Effect (paper arXiv 2503.05066). The SMs look busy while waiting for the saturated expert, and the utilisation dashboard paints green while latency goes through the roof.

Operational implication. Do not trust sizing or autoscaling to SM occupancy alone. Always combine it with PIPE_TENSOR_ACTIVE (is there useful compute?) and DRAM_ACTIVE (is memory the bottleneck?). The normal LLM decode regime is 30–55 %, not 99 %; seeing a sustained 99 % with high TPOT is a symptom of a kernel bug or an MoE straggler.

`DCGM_FI_PROF_PIPE_TENSOR_ACTIVE` — are the tensor cores producing?

The analogy. A factory with two lines: the manual one (CUDA cores) and the automated one (tensor cores). The metric answers “what percentage of the time is the automated line active?”. If you buy an H100 for its tensor cores and the automated line is at 5 %, you have paid for a Ferrari to do bicycle courier work.

The documented anomaly. Issue vllm#20783 (July 2025) was titled literally “Performance Anomaly: compressed-tensors shows no speedup over BF16 on H100”. The operator expected 1.5–2× with FP8 quantisation and got parity with BF16; the PIPE_TENSOR_ACTIVE metric revealed that the FP8 path was not running on the HMMA units (the tensor unit for FP16/BF16/FP8) and was falling back to CUDA cores. Issue vllm#31475 documented the parallel case on MI300X: FP8 slower than BF16 because of a regression in the ROCm path. DCGM exposes separate counters per unit (HMMA for FP16/BF16/FP8, IMMA for INT8, DMMA for TF32/FP32); if HMMA is low even though the model is BF16, the engine is not using tensor cores.

Operational implication. Check PIPE_TENSOR_ACTIVE after every change of quantisation or engine version; a supposedly neutral change may have disabled the optimised path. For prefill expect 50–80 %; for decode 15–30 % is normal (decode is memory-bound, not compute-bound). A figure below 5 % in prefill means the engine is not using tensor cores.

`DCGM_FI_PROF_DRAM_ACTIVE` — is the HBM saturated?

The analogy. A motorway with N lanes. The metric answers “what percentage of the time are all the lanes occupied moving cars?”. When the tensor cores ask for data faster than the HBM delivers it, the motorway is at 95 % and the engines wait. In decode, this is the normal regime: you walk the model weights and the KV cache through for every token.

The documented anomaly. The paper “Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inference” (arXiv 2503.08311) quantifies that at contexts ≥ 128k, reading the KV cache dominates the total decode time and saturates the HBM3 (3.35 TB/s on the H100). The distinctive pattern: DRAM_ACTIVE > 80 % with PIPE_TENSOR_ACTIVE around 10–20 %. Raising the batch no longer helps: the bottleneck is not FLOPS, it is bandwidth. The useful lever is compressing the KV, see Quantisation for --kv-cache-dtype=fp8, which cuts the KV footprint by roughly 50 %.

Operational implication. If DRAM_ACTIVE > 95 % is sustained and gpu_cache_usage_perc < 70 %, something other than your engine is asking for HBM (a leak in a library, another process sharing the GPU without MIG). Investigate immediately with nvidia-smi and fuser /dev/nvidia*.

Family 2 — Memory

`DCGM_FI_DEV_FB_USED` — how much VRAM has been consumed?

The analogy. The fuel level in an aircraft’s tank in flight: the pilot needs to know how much is left and at what rate it is being consumed, not just the instantaneous figure. An H100 at 88 % FB used and stable can operate calmly; the same figure rising 2 %/min announces OOM in 7 minutes.

The documented anomaly. Issue dcgm-exporter#512 documents a surprise relevant to MIG clusters: DCGM_FI_DEV_FB_USED and DCGM_FI_DEV_FB_FREE are absent on H100 GPU instances with MIG enabled, present on A100 and B200, but an exporter bug hides them on H100-MIG. Operators who assume the dashboard covers everything discover the blind spot on the day of the first OOM. Issue dcgm-exporter#271 documents another detail: FB_USED + FB_FREE does not always add up to a constant because there is driver-reserved overhead that shows up in the delta. The original PagedAttention/vLLM paper estimated that pre-PagedAttention serving frameworks wasted 60–80 % of the KV cache to fragmentation; PagedAttention brought it down to under 4 %.

Operational implication. On H100 MIG clusters, check that DCGM_FI_DEV_FB_USED appears per instance before trusting alerts; if it is absent, monitor through nvidia-smi --query-gpu=memory.used directly. Operational rule: alert on the delta (a sustained rise), not just an absolute threshold.

`DCGM_FI_DEV_FB_FREE` — the absolute complement

The analogy. The “range remaining” indicator in a modern car: it complements the percentage with an absolute figure that is directly actionable.

The documented anomaly. When an aggressive PagedAttention pool leaves FB_FREE at small absolute values (< 2 GiB), any normal allocation of transient buffers (the activations of a large prefill) can push you into OOM. The classic pattern: a “green” percentage (87 %) but a “red” absolute (< 4 GiB free on an 80 GB H100).

Operational implication. A complementary alert with an absolute threshold: DCGM_FI_DEV_FB_FREE < 4096 (MiB). It is the safety net for the cases where the percentage misleads because the engine is configured with a very high gpu_memory_utilization.

`DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL` — does the internal bus hold up?

The analogy. An interstate motorway between four cities. Every car that crosses to do a tensor parallel all-reduce pays a toll and consumes width. When there are more cars than the motorway supports, the latency to reach the destination shoots up, even if each individual car is fast.

The documented anomaly. Issue vllm#16300 (April 2025) was titled “Performance degradation with tp=8 compared to tp=4 on 8×A100(80G)” and documented TP=8 delivering worse throughput than TP=4 on the same cluster, same model, same quantisation. Root cause: tensor parallelism requires an all-reduce after each attention and MLP block; at TP=8, the communication cost between 8 GPUs (even over NVSwitch) grows faster than the speedup of partitioning the compute. The practical rule that emerges: TP=4 + 2 replicas usually delivers better latency/throughput than TP=8 + 1 replica, except for extremely long contexts (≥128k) where you need the aggregate VRAM. Theoretical NVLink 4.0 capacity on the H100 SXM: around 450 GB/s per GPU; typical sustained TP=4 regime: 50–150 GB/s.

Operational implication. If NVLINK_BANDWIDTH_TOTAL > 90 % of capacity is sustained, this is not a problem solvable by raising parallelism. The opposite: lower TP. The metric is orthogonal to the sizing from capacity planning: the ceiling is not only VRAM/time, it is also the bus.

Family 3 — Thermal and power

`DCGM_FI_DEV_GPU_TEMP` — is the GPU breathing?

The analogy. The body temperature of an elite athlete under full effort. 36–37 °C is normal; 38 °C is sustainable stress; above 39 °C the body activates protective mechanisms (sweating, slowing down) that degrade performance. The GPU does the same: above a thermal threshold, it reduces its clock automatically. If it did not, it would break.

The documented anomaly. The H100 SXM5 with its 700 W TDP has thermal thresholds that are not entirely public (NVIDIA does not publish them exhaustively in the datasheet), but the behaviour is well known: above roughly 85 °C edge or roughly 95 °C HBM, the bit 0x40 HW_THERMAL appears in the clock throttle reasons. Operators on the NVIDIA developer forum report that with a rack inlet temperature above 27 °C, throttling is routine. The NVIDIA HGX Platform paper states that the minimum recommended air flow is > 1000 CFM/kW; densities above 30 kW/rack at 700 W TDP demand mandatory liquid cooling because forced air does not get there.

Operational implication. If GPU_TEMP > 83 °C is sustained, look first at CLOCK_THROTTLE_REASONS (bit 0x40) and at the rack inlet temperature: this is not an engine problem, it is an air flow one. For legacy air-cooled racks, consider redistributing thermal load or installing a rear-door HX.

`DCGM_FI_DEV_POWER_USAGE` — how much is it asking of the socket?

The analogy. The instantaneous draw of an industrial appliance plugged into a three-phase outlet with a sized breaker. If the washing machine starts at 9 kW and the breaker is 10 kW, you live on the edge; if the washing machine “gets along” with the breaker it is because somebody sized it consciously.

The documented anomaly. A published real measurement: an H100 SXM5 with vLLM running Llama 3.1 405B at batch=4 draws around 697 W at-wall sustained (NVIDIA TDP 700 W). Now the interesting operational lever: lowering nvidia-smi -pl from 700 W to 500 W delivers around 30 % energy savings with only around 20 % loss of throughput. A cluster of 4 nodes × 8 H100 at 700 W equals around 22 kW of GPU alone; at 500 W, around 16 kW. The difference pays an entire quarter’s electricity bill in clusters operated around the clock. A three-phase 415 VAC PDU branch at 60–80 A supports around 32 kW, roughly 4 DGX H100. Legacy 208 V does not support H100 density; reference: NVIDIA DGX SuperPOD Electrical Specifications.

Operational implication. A useful metric for three things: (1) detecting anomalously low workloads (unexpected idle), (2) computing per-tenant showback of real energy cost (not estimates), (3) alerting if the draw approaches the PDU branch limit. Keep GPU → PDU branch → breaker mapped in the CMDB.

`DCGM_FI_DEV_CLOCK_THROTTLE_REASONS` — who is stepping on the brake?

The analogy. The “limited mode” warning light on the dashboard of a modern car. When it lights up, the car reduces its performance automatically, but it does not tell you why unless you know how to read the combination of letters. The bits of the bitmap are those letters.

The documented anomaly. A public case formally acknowledged by two different OEMs: Dell KB 000220508 and Lenovo HT514380 address the same phenomenon: HW Power Brake Slowdown active (bit 0x80) on the H100 SXM. The cause is not the GPU: it is the chassis PDU sending an electrical power-brake signal because the rack branch is close to the breaker limit. The operator sees throughput down 30–50 % with no XID and no ECC, and the inference engine “is healthy”; the problem is in the electrics. The NVIDIA developer forum thread on “HW Power Brake Slowdown” corroborates the pattern. Bit 0x40 HW_THERMAL appears in badly ventilated racks; bit 0x04 SW_POWER_CAP appears if somebody left nvidia-smi -pl 500 set and nobody reverts it.

Operational implication. Any bit other than 0 or Idle (bit 0x01) sustained is an immediate alert. The recommended decoding: record the complete bitmap value in the log plus the attribute throttle.reasons.decoded=["HW_THERMAL", "HW_POWER_BRAKE"] on the OTel span. Without that, incident response does not know what to do.

Family 4 — Health (the catastrophic reports)

`DCGM_FI_DEV_XID_ERRORS` — the driver’s red codes

The analogy. The critical alarm lights in a nuclear control room. They do not climb gradually; they either appear or they do not. Each XID is a predefined code (XID 13 = graphics engine exception; XID 31 = MMU fault; XID 43 = stopped channel; XID 79 = GPU fallen off the bus; XID 95 = uncontained ECC), and each one has its documented procedure.

The documented anomaly. The most studied public case: Meta published that during the training of Llama 3 405B on 16,384 H100s over 54 days there were 419 unplanned failures, one every 3 hours at cluster scale. GPUs accounted for 148 (35 %) plus HBM3 for 72 (17 %), almost half of all failures. The paper “Story of Two GPUs: Characterizing the Resilience of Hopper H100 and Ampere A100” (arXiv 2503.11901) quantifies with a different dataset (2.1M GPU-hours) that the H100 has 3.2× worse MTBE for uncorrectable ECC than the A100. The ByteDance MegaScale paper reports that XID 79 (“GPU fallen off the bus”) co-occurs with PCIe errors in 43 % of cases. The NVIDIA developer forum documents persistent cases of XID 31 (MMU fault) that follow the GPU when it is moved to a different PCIe slot, a hardware bug in the module, not in the backplane.

Operational implication. Any increment of the counter is an immediate alert: many XIDs demand a node reset or an RMA of the GPU. The XID-by-XID distinction matters: XID 13/43 is usually a software bug if it coincides with a recent change; XID 31/48/79/94/95 is usually hardware. Keep a canonical xid → procedure table. See the runbooks for the translation into concrete action.

`DCGM_FI_DEV_ECC_DBE_VOL_TOTAL` — the errors that corrupt data

The analogy. An accounting ledger where sometimes someone erases an entry and rewrites it (corrected single-bit ECC: note a change in the margin and carry on) and sometimes someone burns two pages at once (double-bit: the information is lost, the audit has to stop).

The documented anomaly. The paper “Characterizing GPU Resilience” quantifies for the H100 that when XID 48 (DBE) appears, the job in flight dies with 100 % probability (5 out of 5 in the dataset studied). The documented recovery: drain the node, reset, complete the row remap, equalling around 19 hours of node downtime. HBM3 density explains the worse MTBE against HBM2e: there are more cells per unit area, and therefore a higher statistical probability of degradation. In Llama 3, HBM3 caused 72 of the 419 interruptions (17 %).

Operational implication. Any value > 0 is a critical alert. The GPU must be drained immediately, removed from the scheduler, fully reset, validated for row remap with nvidia-smi -q -d ROW_REMAPPER (Pending: No), and given an extensive smoke test before returning to the pool. If the row remap uses more than 4–8 spare pages on one GPU, plan a replacement in the next window: the degradation is progressive.

`DCGM_FI_DEV_RETIRED_DBE` — the pages marked for retirement

The analogy. The tiles a museum restorer marks with yellow tape because they are damaged. They pose no immediate danger (the room stays open), but the accumulation says the floor is degrading structurally and the full replacement has to be planned.

The documented anomaly. NVIDIA documents up to 512 spare pages per HBM bank on the H100; the RETIRED_DBE counter indicates how many have been used. Operators on NVIDIA forums report that above 4–8 retired pages on a specific GPU, the frequency of XID 48 rises. The pattern: a GPU with 6 retired pages today becomes 12 in a month, a first XID 48 two months later, and a forced drain.

Operational implication. A trend metric, not an immediate alert. Record the value per GPU and review it monthly; GPUs with rising values go into the proactive replacement plan before the catastrophic failure.

The five vLLM engine metrics

`vllm:num_requests_running` — how many requests fit in the batch?

The analogy. The number of cars a toll booth lets through at once. If the barrier opens for N at a time, car N+1 waits in the queue. Saturation shows up because the line does not get shorter.

The documented anomaly. Reaching the configured --max-num-seqs and staying there is the typical symptom of a cluster below its sizing; the engine admits up to the ceiling and no further. The query vllm:num_requests_running == max_num_seqs for more than 5 minutes indicates firm saturation.

Operational implication. Combine it with num_requests_waiting: if running is at the ceiling AND waiting > 0, you have to scale. If running is at the ceiling and waiting is 0, you are in the optimal regime (cluster used to the maximum with no queue).

`vllm:num_requests_waiting` — the primary saturation indicator

The analogy. The visible queue in front of the toll booth. As long as it is empty, the system flows; the moment a sustained queue forms, drivers start arriving late at their destination, and TTFT goes through the roof.

The documented anomaly. A public case in “11-Second Time to First Token on a Healthy vLLM Server” (Medium, Ingero, 2026): a server with no XIDs, no preemption, DCGM metrics all green, but num_requests_waiting sustained above 0 and a TTFT of 11 seconds. Issue vllm#16985 documents progressive degradation in long sessions: the queue grows slowly over hours without any other indicator moving. The root cause is not hardware, it is admission control: the arrival rate exceeds the completion rate and the system does not enqueue any more, it leaves requests in WAITING. Red Hat designates it as the primary saturation metric in its tutorial “5 steps to triage vLLM performance”.

Operational implication. The primary HPA metric in KEDA, see Autoscaling LLM on Kubernetes. Typical threshold: alert if avg_over_time(vllm:num_requests_waiting[5m]) > 5. For canaries: if the queue only forms in the v2 pool, it is a regression of the new model, not cluster load.

`vllm:gpu_cache_usage_perc` — the KV cache pool

The analogy. The capacity of an events hall where each guest takes up a variable amount of space. The maître d’ admits guests up to capacity; when a new guest arrives and there is no room, he throws out the guest who has been there longest to make space for the newcomer. That is vLLM’s preempt-on-OOM.

The documented anomaly. Issue vllm#5051 “Add num_requests_preempted metric” was born exactly out of operators observing degradation with no direct metric telling them how many requests were being thrown out. The official vLLM documentation confirms: “sustained gpu_cache_usage_perc above 90 % indicates the server is approaching its KV cache limit and will begin preempting sequences” (oldest-first). The distinctive visual pattern: a sawtooth near 100 % with preemption spikes. In swap mode, the latency of the preempted request explodes because there is a host↔device PCIe copy; in recompute mode (the default in V1), the preempted request redoes its prefill from scratch, which pushes its TTFT to double or triple.

Operational implication. If gpu_cache_usage_perc > 92 % is sustained, there are two levers: lower max_num_seqs (admit less concurrency, but none of it gets preempted) or raise gpu_memory_utilization (more pool, less VRAM for transient activations, a different risk). The choice depends on the workload. The metric that is directly missing, a preempted counter, is exported from vLLM v1.0 onwards as vllm:num_preemptions_total (see issue #5051).

`vllm:time_to_first_token_seconds` — the latency the client sees

The analogy. The time from a customer walking into a restaurant to receiving the first piece of bread at the table. Too long and the customer thinks they have been forgotten, even if the main course is going to arrive perfect.

The documented anomaly. Three documented patterns of recurring TTFT spikes:

Badly calibrated chunked prefill. Issue vllm#25677 (Qwen3-30B-A3B) reported prefill 10–11× slower with chunked prefill enabled than without it. Cause: a very low max_num_batched_tokens forces small chunks that do not fill the kernels. Issue vllm#7604 documents an equivalent regression on Llama-3-70B v0.5.4. The lever: raise max_num_batched_tokens to 4096–8192 for typical prompts above 2k.
Regression between engine versions. Issue vllm#8819 documents a regression of vllm:time_to_first_token_seconds_sum between minor versions. Issue vllm#11912 reports that with a prompt of around 8000 tokens, TPOT rose from 15.7 ms → 25.7 ms from v0.6.4.post1 onwards with no config change, a regression confirmed and trackable only with the metric.
Long-context prefill blocking decodes. The “11s TTFT on healthy server” case cited above: a 30k-token prefill monopolises the GPU for several seconds and the active decodes freeze. Fix: well-calibrated chunked prefill, or disaggregated serving (see Disaggregated serving).

Operational implication. Do not alert only on the absolute P95; alert also on the v2/v1 ratio when there is a canary (histogram_quantile(0.95, ..., version="v2") / histogram_quantile(0.95, ..., version="v1") > 1.10). If TTFT grows and the queue is stable, the bottleneck is prefill: not solvable by adding replicas, but there are levers in quantisation or chunked prefill.

`vllm:time_per_output_token_seconds` — streaming smoothness

The analogy. The speed at which the waiter brings the courses one after another after the first. If the next one takes a while, the diner notices something is wrong even though the first course arrived on time.

The documented anomaly. The distinctive pattern is the abrupt step when gpu_cache_usage_perc crosses roughly 85 %: TPOT jumps from 35 ms to 80 ms in a few seconds because the engine starts competing for the HBM with its own evictions. Issue vllm#35387 documents another anomalous case: MTP (speculative decoding) causing a 76 % latency regression on Qwen3-Next-80B-A3B-Instruct-FP8. The TPOT metric caught it before any client complaints were filed.

Operational implication. The difference from TTFT: if TTFT grows and Queue Time is stable, you are prefill bound; if TPOT grows at a steady rate, there is pressure on the HBM (KV cache pool or swap enabled). A secondary alert on the TPOT SLO, but also watch the derivative: TPOT rising 1 ms every 10 minutes is a latent regression that does not break the SLO yet but will.

The operational rule: read metrics by family, not in isolation

Three anti-patterns of the novice operator

Anti-pattern 1 — alerting only on absolute thresholds. An H100 at 87 % FB is not necessarily an alarm; an H100 at 87 % rising 2 %/min is. Alerts that fire on a fixed threshold without looking at the derivative produce twice the noise and half the value. Rule: for metrics with known dynamics (KV cache, FB, queue), alert on a sustained delta, not just on the level.

Anti-pattern 2 — confusing SBE with DBE. The DCGM_FI_DEV_ECC_SBE_VOL_TOTAL counter (single-bit, correctable) grows continuously on any HBM under load; it is not an alarm, it is physics. The one that matters is DCGM_FI_DEV_ECC_DBE_VOL_TOTAL (double-bit, uncorrectable). Confusing them yields false negatives (not alerting on a real DBE) or false positives (alerting on a harmless SBE).

Anti-pattern 3 — treating SM_OCCUPANCY at 99 % as “saturated”. The LLM decode regime is memory-bound, not compute-bound; high SM occupancy with low TENSOR_ACTIVE and high DRAM_ACTIVE is normal. Sizing for “GPU at 60 %” and asking for more hardware when the cluster is saturated in HBM (not in SMs) means buying twice the GPUs without gaining throughput. Rule: always read SM_OCCUPANCY together with TENSOR_ACTIVE and DRAM_ACTIVE; on its own it means nothing.

Applied to typical on-premise hardware

For a generic cluster of 4 nodes × 4×H100 SXM 80 GB with intra-node NVLink:

DCGM Exporter per node (a GPU Operator DaemonSet) emitting every 15 s; cardinality per GPU is around 80 series. A 16-GPU cluster gives roughly 1.3k base series, around 85k samples/min with a 15 s scrape.
vLLM /metrics per inference pod; each replica emits around 50 base series. For 16 replicas, around 800 additional series, around 3k samples/min.
Prometheus retention: 30 days at high resolution plus 1 year downsampled through a Thanos sidecar or Mimir. Estimated volume: 25–35 GB/day.
Alertmanager: the 6 critical alerts from the previous post plus derived alerts (delta, v2/v1 ratio, decoded throttle bitmap).

Each metric is worth exposing also as an OTel attribute on the spans of the GenAI tracing: gpu.fb_used_pct, gpu.dram_active, gpu.throttle_reasons.decoded. That lets you correlate a slow request with the state of the GPU at that instant, without jumping between dashboards.

What we have not covered (upcoming posts)

Per-alert runbooks — the translation of each anomalous metric into concrete action (drain, reset, RMA, scaling, rollback) in the next post: Incident response runbooks.
Tail sampling for metric ↔ trace correlation — what is preserved when an alert fires, for post-mortem investigation.
Per-tenant showback combining vllm:request_success_total × gen_ai.usage.* × DCGM_FI_DEV_POWER_USAGE to bill real energy cost.
Multi-tenant fairness metrics — when a tenant hogs the KV cache pool and how to detect it.

References

Meta — Faulty Nvidia H100 GPUs and HBM3 memory caused half of failures during Llama 3 training (Tom’s Hardware, 2024). tomshardware.com
Story of Two GPUs: Characterizing the Resilience of Hopper H100 and Ampere A100. arXiv 2503.11901. https://arxiv.org/html/2503.11901v3
ByteDance — Robust LLM Training Infrastructure at ByteDance. arXiv 2509.16293. https://arxiv.org/pdf/2509.16293
Mind the Memory Gap: Unveiling GPU Bottlenecks in Large-Batch LLM Inference. arXiv 2503.08311.
Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of Experts. arXiv 2503.05066.
NVIDIA — Analyzing Xid Errors with the Xid Catalog and Memory Error Management (docs.nvidia.com/deploy).
Dell — PowerEdge XE8640 with H100 - GPU Performance Issue HW Power Brake Slowdown - Active (KB 000220508).
Lenovo — Power brake reporting on H100 GPU (HT514380).
vLLM project — issues #5051 (preempted metric), #7604 and #25677 (chunked prefill regression), #11912 (long-prompt regression), #16300 (TP=8 worse than TP=4), #16985 (long-running degradation), #20783 (compressed-tensors no speedup), #35387 (MTP regression).
Red Hat — 5 steps to triage vLLM performance. https://developers.redhat.com/articles/2026/03/09/5-steps-triage-vllm-performance
AI21 — Go big or go OOM: the art of scaling vLLM. https://www.ai21.com/blog/scaling-vllm-without-oom/
11-Second Time to First Token on a Healthy vLLM Server (Medium, Ingero, 2026).
NVIDIA — DGX SuperPOD Electrical Specifications (docs.nvidia.com/dgx-superpod).

Sources: the full URLs are linked inline on each reference.

The LLM inference router: the L7 switchboard we called a LoadBalancer in the canary post

Tue, 02 Jun 2026 03:00:00 +0200

This post is the natural continuation of Canary, blue-green and shadow for LLM models. There the promotion mechanics dumped all the traffic-splitting complexity into a box we called “LoadBalancer”. The description was operational, good enough to understand the choreography, but structurally vague: what actually does that splitting is an L7 inference router with LLM awareness, a piece of the stack in its own right (layer 1 of the seven layers) that deserves its own post.

TL;DR

In the previous post on canary we called LoadBalancer the piece that splits traffic between the stable v1 and candidate v2 pools. The description was enough to understand the flow, but technically it was blurry: neither an L4 LoadBalancer (kube-proxy, MetalLB, IPVS) nor a generic L7 HTTP LoadBalancer (NGINX or HAProxy with no extension) knows what a model is, what a version is, how many tokens a request costs, what prefix the prompt has or which replica has which KV cache hot. The right piece is an LLM inference router: an L7 proxy with explicit knowledge of the domain. It combines four functions: model catalogue (resolving model=llama-70b@v2 → service.namespace:port), traffic splitting (applying the canary weight with a deterministic hash or deliberate stickiness for A/B), cross-cutting policy (OIDC auth, per-tenant rate limit and quota, pre-prompt PII redaction, lightweight inline guardrails, gen_ai.* tracing propagation) and failover/degradation (if v2 falls over, redirect to v1; if the whole cluster is saturated, return 503 with Retry-After instead of queueing forever). The non-obvious piece that justifies its technical existence beyond the operational one is prefix-aware routing: the router decides which fleet replica each request goes to based on the prompt prefix, so that a RAG system with the same system prompt plus the same block of retrieved documents systematically hits the prefix cache (RadixAttention in SGLang, PrefixCaching in vLLM, KV reuse in TensorRT-LLM) of the same replica, multiplying the hit rate from 5–15 % (blind round-robin) to 60–85 % (prefix affinity). The concrete pieces in May 2026 are LiteLLM Proxy (the simplest option, OpenAI-compatible, declarative YAML catalogue), vLLM Production Stack router (specific to vLLM fleets, aware of the KV cache and the prefix), Envoy AI Gateway (LLM-aware Envoy filters, integrable with Istio), Kong AI Gateway (enterprise alternative with a plugin ecosystem), KGateway (CNCF, still incubating) and the NVIDIA Dynamo router (production-grade, aware of disaggregated prefill/decode serving). In the seven-layer stack it lives in layer 1 (gateway); in the five maturity levels it appears from level 3 onwards; in the seven-phase deployment cycle it is the last piece that F6 closes. This post includes a minimal manifest applicable to a generic 4×H100 SXM cluster.

You are here: DEPLOY (layer 1 of the stack)

The background: what the canary post called a “LoadBalancer”

In Canary, blue-green and shadow for LLM models we described the flow like this: “the LoadBalancer progressively splits traffic following a schedule: 1 % → 5 % → 25 % → 100 %”. It was a correct operational description, and the reader understood the choreography without needing more. But technically it left unnamed a piece that deserves explicit treatment, because neither of the two usual senses of “LoadBalancer” does what that paragraph assumed:

An L4 LoadBalancer (kube-proxy with iptables/IPVS, MetalLB, F5 BIG-IP in TCP mode) splits IP packets without looking inside the payload. It does not know which model is being requested, nor which version, nor how many tokens it carries, nor whether the client has quota. It cannot apply the canary weight “for model X version 2”: to it, every packet heading for the vllm-llama70b VIP is indistinguishable.
A generic L7 HTTP LoadBalancer (NGINX or HAProxy in HTTP mode with no extension, a ClusterIP Service with multiple backends) does split by URL and can route by header, but it does not understand the OpenAI-compatible body of the request. It does not know that {"model": "llama-70b", "messages": [...]} carries the routing key in the model field; it does not count tokens; it does not apply policies over LLM structures; it does not do prefix-aware routing because that requires parsing messages and hashing the common prefix.

The piece the canary post assumed was doing this work is an L7 inference router with LLM awareness. A layer in its own right, with its own configuration, its own CI/CD, its own metrics and its own pitfalls. This post names it and takes it apart.

The analogy: the switchboard and triage of a hospital with multiple specialities

A large hospital receives patients who arrive at A&E through different doors and who need different specialities: trauma, cardiology, paediatrics, oncology. There are three possible models of “front door”.

A single door with no triage. All patients wait in the same room and are sent in arrival order to the first free doctor, whatever their speciality. It works in a village surgery with a single general practitioner. When there are 200 patients a day and 12 specialities, it falls into dysfunction fast: the cardiologist treats sprains, the paediatrician treats heart attacks, specialised resources are wasted. This is the equivalent of the L4 LoadBalancer, which distributes bodies without understanding what they bring.

A door with a receptionist who asks about the symptom. Someone at the front desk asks “what’s wrong?” and directs the patient to the right corridor. The cardiologist sees only cardiology, the paediatrician only children. Better, but the receptionist is slow, does not gauge urgency and does not know the state of the rooms: they may send you to the cardiologist in corridor A when the one in B is free. This is the equivalent of a generic L7 HTTP proxy with path-based routing, which distributes by category but with no information about internal state.

Professional triage with full awareness. A trained triage nurse who knows the catalogue of specialities, knows which bay is busy and which is free, remembers the recurring patient whose file is already open in the system (and sends them to the same doctor for continuity), applies cross-cutting policy (checks insurance cover, records allergens, redirects to paediatric A&E if the patient is a minor) and, if the cardiology room goes down because the electrocardiograph has broken, redirects to the hospital on the other side of the city. This is the piece a large hospital needs. In LLM terms it is called an inference router.

The analogy holds down to the last detail, including the “file already open” one: the patient who returns to the same doctor is exactly the client whose prompt shares a prefix with the one from five minutes ago. If the router sends it to the same replica, that replica still has the KV cache hot and the request hits the prefix cache. If it sends it to a different replica because that one was “next in round-robin”, the KV cache has to be rebuilt from scratch and TTFT doubles. The triage nurse knows this. The blind LoadBalancer does not.

The four functions of the inference router

Function 1 — Model catalogue

The router keeps a declarative catalogue mapping model identity to a concrete deployment:

models:
 - name: "llama-70b" # stable alias
 version: "v2" # canary version
 weight: 5 # 5% of the traffic
 endpoint: "vllm-llama70b-v2.inference.svc.cluster.local:8000"
 capabilities: [chat, tool_use]
 lifecycle: canary
 - name: "llama-70b"
 version: "v1"
 weight: 95
 endpoint: "vllm-llama70b-v1.inference.svc.cluster.local:8000"
 capabilities: [chat, tool_use]
 lifecycle: stable
 - name: "embedding-multilingual"
 version: "v1"
 weight: 100
 endpoint: "tei-bge-m3.inference.svc.cluster.local:8080"
 capabilities: [embeddings]
 lifecycle: stable

The client sends {"model": "llama-70b", "messages": [...]} without knowing there are two replica pools behind it. The router resolves. If tomorrow you migrate from vLLM to SGLang for a specific version, the client never notices; you change the endpoint in the catalogue and that is it.

What this decoupling buys you is the freedom to move topology without breaking clients. What it costs is keeping the naming convention disciplined (llama-70b is always the stable alias; llama-70b@v2 is the specific version for canary). Without that discipline, the aliases get dirty with llama-70b-prod-fixed-real-final-v3 and the catalogue stops being navigable within a few weeks.

Function 2 — Traffic splitting

The splits from the canary post (1 % → 5 % → 25 % → 100 %) materialise here, not in the inference engine. The router computes a deterministic hash of the request_id (or of the user_id, if you want stickiness) and maps it to the weight range in the catalogue. For a weight of [v1: 95, v2: 5], 5 % of the hash space falls in v2 and 95 % in v1.

Three design decisions that matter:

Hash by random request_id = independent sampling. Each request is an independent observation of the v1 vs v2 distribution. It is the right setting for statistically comparable canaries.
Hash by user_id = per-user stickiness. The same client always sees the same pool. Useful for A/B testing with persisted conversational memory, but it breaks the statistical comparability of the canary because user populations are not symmetric, a pitfall explained in the previous post.
Hash by tenant_id = hard partitioning. Tenant A goes to v1, tenant B to v2. This is the pattern for clients with different SLAs, or for validating v2 in an internal tenant before exposing it to external clients.

Function 3 — Cross-cutting policy

Once, across all models, the router applies:

Auth: OIDC with JWT tokens validated against Keycloak / Authentik. Authorization: Bearer ... headers translated into tenant_id and roles.
Rate limit: token bucket per tenant (X req/min) or per model (Y req/min for llama-70b because it is expensive).
Quota: monthly quota of tokens consumed per tenant. The router counts gen_ai.usage.input_tokens + gen_ai.usage.output_tokens and rejects with 429 Quota exceeded when it runs out.
Pre-prompt PII redaction: Presidio or Llama Guard inline before the prompt touches the model. What the model does not see is not trained on, not logged, not leaked.
Lightweight inline guardrails: PromptGuard 2, Llama Guard 4, Granite Guardian, the ones covered in Guardrails and safety in LLMs, run in the router because their latency (30–150 ms) fits within the TTFT budget.
gen_ai.* tracing propagation: the router starts the parent span, propagates traceparent to the engine and emits the gen_ai.system, gen_ai.request.model and gen_ai.request.version attributes that OTel GenAI tracing consumes.
Semantic cache: for prompts that repeat exactly or with high semantic similarity (embedding cosine > 0.97 against the previous cache), it returns the cached answer without touching the engine. Typical saving in RAG with frequently asked questions: 20–40 % of requests.

Function 4 — Failover and degradation

The router knows the health state of each endpoint (active health probes every 5–15 s, recent TTFT latency) and decides:

If v2 returns persistent 5xx or does not respond, the circuit breaker opens: the router redirects the traffic that was going to v2 towards v1 until the probes go green again. This is automatic canary rollback in its simplest form.
If the whole cluster is saturated (all replicas reporting num_requests_waiting > N for T seconds), the router returns 503 Service Unavailable with Retry-After: 30 instead of queueing forever. It is better to tell the client “come back in 30 seconds” than to keep them waiting four minutes and then time out.
If there is multi-region or multi-cluster, cross-cluster failover via DNS or L7: the primary region goes down and the secondary router takes over.

The non-obvious piece: prefix-aware routing

This is the function a conventional LoadBalancer cannot do and the one that justifies an LLM-specific router beyond the four generic ones.

The KV cache in vLLM, SGLang and TensorRT-LLM can reuse common prefixes across requests, see KV cache. Specifically:

vLLM with --enable-prefix-caching: it detects that the current request shares a prefix (of a length that is a multiple of the block size, 16 tokens by default) with a previous request whose pages are still in HBM, and reuses those pages instead of reprocessing them.
SGLang with RadixAttention: it structures the cache as a radix tree indexed by tokens; each request hits the common path of the tree and only computes the tail.
TensorRT-LLM: a similar feature, called KV cache reuse.

The prefix cache hit rate is the key metric: every token that hits is a token that is not processed in prefill, reducing TTFT in direct proportion. For a typical RAG system, with a 400-token system prompt plus 2,000 tokens of retrieved documents plus a 50-token user question, the common prefix (system_prompt + docs) is 2,400 of the 2,450 total tokens. If the cache hits, prefill only processes 50 tokens instead of 2,450: TTFT falls to roughly a twentieth.

But the cache lives per replica, not globally. If two requests with the same 2,400-token prefix land on different replicas, both do the full prefill: the first one’s cache is no use to the second. The second pays the full cost.

With blind round-robin (any conventional LB), requests are spread uniformly across N replicas. For a cluster of 4 replicas and 1,000 requests with the same system_prompt + docs, each replica receives ~250 requests, but all 4 do their own “first prefill” and the following 249 benefit within their replica. The global hit rate is decent but not optimal. For traffic with many different system prompts and little intra-prefix repetition, the hit rate hovers around 5–15 %.

With prefix-aware routing, the router computes a hash of the prompt prefix (the first N tokens, or the system_prompt declared in messages[0]) and keeps an affinity table of hash → replica. All requests with the same prefix land on the same replica. The first pays the full prefill; the following 999 hit the cache. Global hit rate: 60–85 %.

The cost of implementing it: the router has to parse the request body (not just the HTTP header), apply a lightweight tokeniser or a byte-based hash, and maintain an LRU/consistent-hash affinity table that rebalances when a replica comes in or out. That is server work, not generic proxy work. vLLM Production Stack router implements it natively. NVIDIA Dynamo does too. LiteLLM has a beta in its enterprise version. Envoy AI Gateway is adding it as an experimental filter.

Correction (September 2026). The previous sentence about LiteLLM is false: prefix-aware routing does not exist in any version, open or paid, verified against the code of 1.102.0. The detail, what LiteLLM does have, and the real state of the field are in Prefix routing: what LiteLLM does not do.

The operational difference for a production RAG system: with prefix-aware routing, the same cluster serves 2–4× more requests without adding GPUs, simply because prefill disappears in most cases.

Token-aware load balancing

The second non-obvious piece. Classic round-robin splits by number of requests; but a 50-token prompt and an 8,000-token one cost radically different amounts (a factor of ~160× in prefill). Splitting equally by count severely unbalances the real load.

Token-aware load balancing adds up the expected prefill tokens (prompt length) and expected decode tokens (the client’s max_tokens) per active replica, and sends the new request to the replica with the lowest accumulated load. It is what both vLLM Production Stack and NVIDIA Dynamo implement as the default strategy when it is enabled.

The metric feeding the calculation is, once again, vllm:num_requests_running and vllm:gpu_cache_usage_perc, see GPU observability for LLM inference, ideally complemented with an estimator of the incoming prompt’s tokens. Mature routers use tiktoken or the model’s real tokeniser to count prompt tokens before choosing a replica.

Comparison of concrete pieces (May 2026)

Piece	LLM awareness	Prefix-aware	Token-aware LB	Multi-model	Semantic cache	Plug & play
LiteLLM Proxy	High	Beta (enterprise)	Yes	Excellent	Yes (Redis)	Very high
vLLM Production Stack router	vLLM-specific	Yes, native	Yes	vLLM only	No (external)	Medium
NVIDIA Dynamo router	High + disagg-aware	Yes	Yes	vLLM/SGLang/TRT-LLM	No (external)	Low
Envoy AI Gateway	Medium (filters)	Experimental	Yes	Yes	Via filter	Medium
Kong AI Gateway	Medium (plugins)	No	Yes	Yes	Yes (plugin)	Medium
KGateway	Medium	Roadmap	Yes	Yes	Roadmap	Low (CNCF incubating)
NGINX + custom Lua	Manual	No	Manual	Manual	No	Low (build it yourself)

LiteLLM Proxy is the default option to start with. OpenAI-compatible, simple YAML, supports the commercial providers plus any self-hosted OpenAI-compatible backend. The OSS version covers the four basic functions and semantic cache; prefix-aware routing and the enterprise version add advanced multi-tenancy.

vLLM Production Stack router is the right option if the fleet is 100 % vLLM. Aware of the KV cache, the prefix and the LoRA loaded per replica. It integrates better with native vLLM metrics.

NVIDIA Dynamo router is the most complete production-grade option, especially if you operate disaggregated serving (separate prefill workers and decode workers). It requires an NVIDIA-aligned stack.

Envoy AI Gateway and Kong AI Gateway are the options if the organisation already has Envoy or Kong as its corporate gateway and wants to extend it with LLM awareness without introducing another new piece.

Minimal manifest: LiteLLM Proxy on a generic cluster

apiVersion: v1
kind: ConfigMap
metadata: { name: litellm-config, namespace: inference }
data:
 config.yaml: |
 model_list:
 - model_name: llama-70b
 litellm_params:
 model: openai/llama-70b
 api_base: http://vllm-llama70b-v1.inference.svc:8000/v1
 weight: 95
 model_info:
 version: v1
 lifecycle: stable
 - model_name: llama-70b
 litellm_params:
 model: openai/llama-70b
 api_base: http://vllm-llama70b-v2.inference.svc:8000/v1
 weight: 5
 model_info:
 version: v2
 lifecycle: canary
 - model_name: embedding-multilingual
 litellm_params:
 model: openai/bge-m3
 api_base: http://tei-bge-m3.inference.svc:8080
 router_settings:
 routing_strategy: least-busy # basic token-aware
 num_retries: 1
 timeout: 60
 general_settings:
 master_key: "os.environ/LITELLM_MASTER_KEY"
 database_url: "os.environ/DATABASE_URL"
 litellm_settings:
 cache: true
 cache_params:
 type: redis
 host: redis.inference.svc
 port: 6379
 similarity_threshold: 0.97
 success_callback: ["langfuse"]
 failure_callback: ["langfuse"]
---
apiVersion: apps/v1
kind: Deployment
metadata: { name: litellm-router, namespace: inference }
spec:
 replicas: 3
 selector: { matchLabels: { app: litellm } }
 template:
 metadata: { labels: { app: litellm } }
 spec:
 containers:
 - name: litellm
 image: ghcr.io/berriai/litellm:v1.55.0
 args: ["--config=/config/config.yaml", "--port=4000", "--num_workers=4"]
 ports: [{ containerPort: 4000, name: http }, { containerPort: 4000, name: metrics }]
 env:
 - { name: LITELLM_MASTER_KEY, valueFrom: { secretKeyRef: { name: litellm-secret, key: master_key } } }
 - { name: DATABASE_URL, valueFrom: { secretKeyRef: { name: litellm-secret, key: db_url } } }
 - { name: LANGFUSE_PUBLIC_KEY, valueFrom: { secretKeyRef: { name: langfuse-keys, key: public } } }
 - { name: LANGFUSE_SECRET_KEY, valueFrom: { secretKeyRef: { name: langfuse-keys, key: secret } } }
 volumeMounts: [{ name: config, mountPath: /config }]
 readinessProbe: { httpGet: { path: /health, port: 4000 } }
 volumes: [{ name: config, configMap: { name: litellm-config } }]
---
apiVersion: v1
kind: Service
metadata: { name: litellm-router, namespace: inference }
spec:
 selector: { app: litellm }
 ports: [{ name: http, port: 80, targetPort: 4000 }]
---
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata: { name: litellm-metrics, namespace: inference }
spec:
 selector: { matchLabels: { app: litellm } }
 podMetricsEndpoints:
 - port: metrics
 path: /metrics
 interval: 15s

The end client points at litellm-router.inference.svc:80/v1/chat/completions, sets model=llama-70b, and the router decides on each request whether it goes to v1 (95 %) or v2 (5 %), applies the rate limit, looks in the semantic cache, propagates tracing to Langfuse, and translates from OpenAI-compatible to the OpenAI-compatible API of the destination vLLM. Three router replicas for HA and so the gateway itself can scale horizontally with KEDA if needed, see LLM autoscaling on Kubernetes.

Four operational pitfalls

Pitfall 1 — the router becomes a SPoF if it is not replicated. Three or more replicas of the router itself, behind a LoadBalancer Service (this one genuinely L4) with healthchecks. A single router replica means every configuration deploy closes the whole service for a few seconds.

Pitfall 2 — the router’s latency adds to the model’s. Each function adds milliseconds: body parsing (5–10 ms), JWT auth (2–5 ms), rate limit (1–2 ms), PII redaction with Presidio (20–80 ms), inline guardrails with Llama Guard (50–150 ms), prefix hash (5–10 ms), token counting with a tokeniser (10–30 ms). In total, 100–300 ms of overhead before touching the engine. If the model’s TTFT is 400 ms and the router’s is 200 ms, the client sees 600 ms, so it is worth measuring what each function costs and disabling the non-critical ones in the low-latency path.

Pitfall 3 — the catalogue drifts from the real state of the cluster. The router believes vllm-llama70b-v2 exists because it is in its YAML, but the deployment was withdrawn three days ago and nobody updated the config. The router returns 502 on 5 % of the traffic. Fix: validate the catalogue against kubectl get svc in CI; no endpoint in the catalogue may point at a non-existent Service. Or better: the router discovers the available endpoints dynamically via a label selector (app=vllm,model=llama-70b) and applies the catalogue weights over the ones that are alive.

Pitfall 4 — semantic cache with an outdated embedding. The semantic cache compares the new prompt’s embedding against the embeddings of cached prompts. If you update the embedding model (see RAG corpus curation), the distances are computed in a different space and the cache stops working correctly (false hits or false misses). Policy: the cache is invalidated when the embedding model changes; generations are never mixed.

Fit within the stack and maturity

In the seven-layer stack, the router is layer 1: the front door that precedes the inference engine (layer 2), the KV cache + PagedAttention (layer 3) and the rest. It is the only piece that sees all the traffic from the outside; any policy not applied here gets duplicated N times in the engines.

In the five maturity levels, the router appears from level 3 (MANAGED) onwards: without OIDC + RBAC + cert-manager + default-deny NetworkPolicy, the router has nobody to authenticate and nobody to apply quotas to; before level 3, what you should build is a minimal proxy with no pretension of a catalogue. Platforms that try to have a polished router at level 1 end up with a large YAML nobody maintains.

In the seven deployment phases, the router is what closes F6: the last atomic step that puts the cluster into production. Without a router, F6 does not finish, because the catalogue, the quotas, the canaries and the failovers are a necessary condition for opening production traffic.

Applied to typical on-premise hardware

For a generic cluster of 4 nodes × 4×H100 SXM 80 GB, the inference router consumes modest resources: 3 replicas of the router pod (2 CPU cores, 4 GiB of memory each) are enough to sustain thousands of RPS because its work is light (parsing, hashing, routing, no inference). The router lives on non-GPU nodes of the cluster (control plane nodes or general workload nodes), and never consumes nvidia.com/gpu.

Traffic volume that a LiteLLM with 3 replicas and 4 workers each will sustain: 2,000–5,000 RPS routing to a vLLM backend, with 80–150 ms of overhead on the full path (auth + rate limit + cache check + propagation). If more is needed, scaling the router with KEDA on litellm:requests_per_second is trivial.

For larger clusters (16+ GPU nodes), consider the vLLM Production Stack router or the NVIDIA Dynamo router, which are more complex but squeeze out the prefix-aware routing and token-aware LB that LiteLLM OSS does not cover. For multi-region clusters, Envoy AI Gateway with Istio Service Mesh is the standard choice.

What we have not covered (upcoming articles)

Deep comparison of LiteLLM vs vLLM PStack vs Dynamo, with prefix-aware benchmarks on a real on-premise cluster.
Semantic cache with Redis Stack + RedisVL: hit rate, false positives, TTL policy.
Multi-region routing: how the router decides between the DC1 and DC2 clusters based on latency, health and load.
AI Gateway specific features: token-bucket cost-based rate limiting (penalising long prompts), guardrails policy engine in the router.
Migration path: how to introduce a router into a cluster that already has clients pointing straight at the vLLM service, with no downtime.

References

LiteLLM project — litellm.ai (documentación de Proxy, routing strategies, semantic cache).
vLLM Production Stack — github.com/vllm-project/production-stack (router con prefix-aware nativo).
NVIDIA Dynamo — developer.nvidia.com/blog/nvidia-dynamo-1-production-ready/ (router production-grade con disaggregated-aware).
Envoy AI Gateway — gateway.envoyproxy.io/docs/tasks/ai-gateway/ (proyecto en gestación dentro de Envoy).
Kong AI Gateway — konghq.com/products/kong-ai-gateway (proxy enterprise con plugin LLM).
KGateway — kgateway.dev (alternativa CNCF en gestación).
Zheng et al. — SGLang: Efficient Execution of Structured Language Model Programs (NeurIPS 2024) — RadixAttention y prefix caching.
vLLM project — Automatic Prefix Caching (docs.vllm.ai/en/latest/features/automatic_prefix_caching.html).
Patel et al. — SplitWise: Efficient Generative LLM Inference Using Phase Splitting (ISCA 2024) — base teórica del routing prefill/decode aware.

Canary, blue-green and shadow for LLM models: how to deploy a new version without breaking the SLO

Mon, 01 Jun 2026 16:30:00 +0200

This post complements those on LLM autoscaling on Kubernetes (the autoscaler coexists with the rollout and must respect it), GPU observability for LLM inference (the metrics that act as gates come from there), Evals for LLMs (the eval that decides whether the new model is ready), LLM-as-judge (the technique that puts the “quality” into the canary gate) and Retrain: closing the loop (the previous step that the new model comes out of).

TL;DR

Promoting a new version of an LLM model to the production cluster without cutting traffic or breaking the SLO demands progressive deployment. The three canonical strategies, blue-green, canary and shadow, answer different questions and have different costs. Blue-green: a complete new pool brought up in parallel, atomic load balancer switch. Instant rollback (point back at the old pool); it demands twice the GPUs during the window. Canary: traffic is shared progressively between the old version and the new one (1 % → 5 % → 25 % → 100 %), measuring regression gates at each step; it consumes incrementally less hardware but exposes real users to the new model from the first percentage point. Shadow / mirror: the old model serves 100 % of the real traffic to the client and, in parallel, a copy of every request goes to the new model without returning its response to the user; it isolates you from quality risk but spends the new model’s GPU on responses nobody consumes, and it does not work well with long SSE streaming. The choice depends on three factors: available GPU budget, service criticality and the availability of a fast automatic eval. The five regression metrics any LLM canary should gate on are: TTFT P95, error rate (HTTP 5xx + premature finish_reason="length"), quality score with LLM-as-judge over a golden set, statistical drift of output embeddings (Wasserstein or KL against the baseline distribution) and cost per request (tokens/s and kW/request). On Kubernetes, Argo Rollouts manages traffic and the AnalysisTemplates as automatic gates; Flagger is the more opinionated alternative. vLLM v1 does not support hot model swap robustly as of May 2026, so the rollout unit is the whole replica (deployment v2 alongside deployment v1). The three specific pitfalls: LB sticky sessions break the statistical comparability of the canary (a client A always lands on the new one, B on the old one, so the populations are not equivalent); semantic eval with LLM-as-judge takes 2–8 seconds per sample and is no use as a real-time gate (it is used in post-analysis or offline before promotion); SSE streaming complicates shadowing because the new model’s response has to be discarded without affecting the old one’s. This post includes a minimal Argo Rollouts manifest applicable to a generic cluster with the NVIDIA GPU Operator.

You are here: DEPLOY (and the transition to RETRAIN)

A new model does not appear in the cluster by magic: it comes from the retrain loop or from a weights provider update. The step between “I have an artifact that passed offline eval” and “it is serving 100 % of the traffic” is exactly this post.

The analogy: opening night of a play

A theatre company is about to premiere a new version of a play that has been running successfully for a year. The company knows several hard facts: the current audience pays for a consistent experience; a bad opening damages the business for months; but premiering nothing leaves the company obsolete against the competition.

The three opening routes the management can choose are the same three of the LLM rollout.

Dress rehearsal behind closed doors (shadow / mirror). The new actors perform the whole play to an empty theatre. There is no audience; nobody buys a ticket. Three full run-throughs are enough to check continuity, timing and the chemistry of the cast. It is expensive because there are salaries and theatre rent, but it does not expose the audience to risk. Useful when the new cast is untested and the director wants to see how it holds up over a complete performance before selling it. In LLM terms: the new model processes every real request in parallel with the old one but its responses are discarded; you spend the new model’s GPU on responses nobody sees.

Cast by performance, alternating (canary). Instead of changing the whole cast at once, Thursday performances belong to the new cast, Friday ones to the old one, Saturday ones half and half. The management reads the visitors’ book comments and the audience numbers performance by performance, deciding after two weeks whether to promote the new cast to permanent or withdraw it. Cheaper than the dress rehearsal because the performances still sell tickets, but it exposes a real audience to risk from the first Thursday. In LLM terms: traffic is shared progressively between the old version and the new one, measuring gates at each step.

Double company with an atomic switch (blue-green). The company hires the new cast, prepares it for a month behind closed doors, and one Saturday announces: “from the next opening night every performance is with the new cast”. If the first performance goes badly, they go back to the old cast in 24 hours, but during that month of preparation double salaries are paid to both companies. In LLM terms: two complete pools of the same size, instant LB switch from one to the other, rollback in seconds if the metrics break.

The analogy also supports the decision: the choice depends on how critical the play is for the business (criticality of the LLM service), how much budget there is to sustain two casts at once (GPU budget), and how much confidence there is in the new cast from the chamber rehearsals (offline eval prior to the canary).

The three strategies in detail

Blue-green

The operator keeps two replica pools identical in size: the blue one (production version v1) and the green one (candidate version v2). When v2 is validated offline (eval passed, smoke tests), the LoadBalancer switch redirects 100 % of the traffic from blue to green in a single step. If the SLO metrics break, the switch goes back in seconds.

Cost: 2× GPUs for the whole window (preparing v2 + the post-switch observation window). For a 16-GPU cluster serving Llama 70B with TP=4 (4 replicas), preparing the blue-green needs 16 extra GPUs for 1–3 days.

Risk: the switch is atomic. If v2 has a problem that did not show up in offline eval but does show up at scale (for example, edge cases that only appear at 200 RPS), 100 % of users notice it at once. Rollback is instant, but the requests of the first minute after the switch were already affected. So blue-green is preferable when you have high confidence in v2 (a minor change: same architecture, same format, just a new version of the weights) and you prioritise immediate rollback over gradual exposure.

Canary

The operator deploys v2 with a small number of replicas (typically 1) next to the v1 pool. The LoadBalancer shares the traffic out progressively on a schedule: 1 % for 30 minutes → 5 % for 1 hour → 25 % for 2 hours → 50 % for 4 hours → 100 %. Between each step, an analysis gate evaluates regression metrics over the traffic that is already landing on v2. If the gate fails, the rollback withdraws traffic from v2 automatically and leaves v1 serving everything.

Cost: incremental. At the start (1 % of traffic) one v2 replica is enough; at 50 % you need half as many v2 replicas as v1 has in total. Peak extra GPU during the canary: about 30–50 % above baseline.

Risk: real users are seeing v2 from the first 1 %. If v2 produces responses with degraded quality but normal TTFT and error rate, the affected users perceive the degradation without the gate detecting it (unless the gate includes quality drift, which is slow). So canary is preferable when you have medium confidence in v2 (a significant change: different architecture or training) and you accept that a low percentage of users will be guinea pigs.

Shadow / mirror

The LoadBalancer sends 100 % of the real traffic to v1 (which answers the client) and duplicates every request towards v2 (whose response is discarded or stored for offline analysis). The client never sees v2; it is never exposed to the risk.

Cost: 100 % extra compute for v2 with no direct user value for the whole shadow window. For a 16-GPU cluster serving Llama 70B with TP=4 (4 replicas), a shadow of the same size consumes 16 extra GPUs full time.

Risk: shadow is the safest for the user. But it has two serious limitations: (a) if v2 has a bottleneck that makes the copied request to the shadow take a long time, the shadowing proxy can consume LB connections; it must be out-of-band (asynchronous); (b) long SSE streaming complicates mirroring because you have to keep two parallel streams and discard one while the other flows to the client. Common pattern: shadow only non-streaming requests (short completions, classification), manual offline eval of the streaming requests.

The five regression metrics that act as a gate

Without automatic gates, the “canary” is just a nice name for “manual rollout with a variable percentage”. The gates are the piece that turns the canary into a defensible operation.

Metric 1 — TTFT P95. Comparison of the new model’s P95 against the baseline (v1) P95 in 5-minute windows. Gate: ttft_p95(v2) / ttft_p95(v1) < 1.10. It detects prefill latency regressions (a slower new model) or engine problems (suboptimal config). Source: vllm:time_to_first_token_seconds_bucket (see GPU observability for LLM inference).

Metric 2 — Error rate. Sum of HTTP 5xx + unexpected 4xx + rate of premature finish_reason="length" (responses cut short because the new model does not generate EOS). Gate: error_rate(v2) - error_rate(v1) < 0.01 (1 percentage point). It detects engine crashes, a broken tokenizer, generation problems. Source: vllm:request_success_total{status=...}.

Metric 3 — Quality score (LLM-as-judge). Over a golden set of 200–1,000 representative prompts, v1 and v2 are run offline and a judge model (typically larger: GPT-4 class, Claude, local Llama 405B) scores each pair. Typical gate: mean_score(v2) >= mean_score(v1) - 0.05. This metric is not measured in real time during the canary: judge inference takes 2–8 seconds per sample and does not scale as an inline gate. It is used as an offline pre-promotion gate (before starting the canary) and as a post-mortem over a sample of real traffic captured during the canary. See LLM-as-judge for the mechanics.

Metric 4 — Statistical output drift. For every request that lands on v2 during the canary, embed the response with a lightweight embedding model (e5, BGE) and compare the distribution of v2 embeddings against the distribution of the v1 baseline over the same window. Usable metrics: Wasserstein distance, KL divergence, or more simply, comparing means and variances per dimension. Gate: normalised distance < a calibrated threshold (typically Wasserstein < 0.15). It detects subtle changes in style, length and vocabulary that LLM-as-judge does not capture without going through it too. It is fast: the lightweight embedding takes around 50 ms per response.

Metric 5 — Cost per request. Output tokens per request and kW per request. Gate: cost_per_request(v2) / cost_per_request(v1) < 1.20. It detects new models that generate significantly longer responses or that consume more energy for the same load (quantisation degradation, optimisations failing). Without this gate, an “update” can silently double the bill.

Metric	Type	Measurement latency	Typical gate	Detection
TTFT P95	Quantitative	5 min	`< 110% baseline`	Latency regression
Error rate	Quantitative	1 min	`< 1pp over baseline`	Crashes, generation broken
Quality (LLM-judge)	Offline semantic	hours, over golden set	`> baseline − 0.05`	Functional quality
Statistical drift	Statistical	~5 min	Wasserstein < 0.15	Style, length, vocabulary
Cost per request	Quantitative	5 min	`< 120% baseline`	Economic/energy efficiency

The mechanics on Kubernetes: Argo Rollouts

Argo Rollouts extends the standard Kubernetes Deployment with a new Rollout resource that orchestrates traffic progression and automatic analyses. It integrates with any service mesh (Istio, Linkerd) or ingress controller that supports traffic splitting (NGINX, Traefik, Gateway API).

A minimal example of a 1 → 5 → 25 → 100 % canary with TTFT and error rate gates:

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata: { name: vllm-llama70b }
spec:
 replicas: 4
 strategy:
 canary:
 canaryService: vllm-llama70b-canary
 stableService: vllm-llama70b-stable
 trafficRouting:
 nginx:
 stableIngress: vllm-llama70b-ingress
 steps:
 - setWeight: 1
 - pause: { duration: 30m }
 - analysis: { templates: [{ templateName: ttft-error-gate }] }
 - setWeight: 5
 - pause: { duration: 1h }
 - analysis: { templates: [{ templateName: ttft-error-gate }] }
 - setWeight: 25
 - pause: { duration: 2h }
 - analysis: { templates: [{ templateName: ttft-error-gate }, { templateName: drift-gate }] }
 - setWeight: 50
 - pause: { duration: 4h }
 - analysis: { templates: [{ templateName: ttft-error-gate }, { templateName: drift-gate }] }
 - setWeight: 100
 selector: { matchLabels: { app: vllm-llama70b } }
 template:
 metadata: { labels: { app: vllm-llama70b } }
 spec:
 containers:
 - name: vllm
 image: vllm/vllm-openai:v0.10.0
 args: [ --model=/models/llama-70b-fp8-v2 ]  # new version
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata: { name: ttft-error-gate }
spec:
 metrics:
 - name: ttft-p95-ratio
 interval: 1m
 count: 5
 failureLimit: 1
 successCondition: result < 1.10
 provider:
 prometheus:
 address: http://prometheus.observability.svc:9090
 query: |
 histogram_quantile(0.95, sum by(le)(rate(vllm:time_to_first_token_seconds_bucket{version="v2"}[5m])))
 /
 histogram_quantile(0.95, sum by(le)(rate(vllm:time_to_first_token_seconds_bucket{version="v1"}[5m])))
 - name: error-rate-diff
 interval: 1m
 count: 5
 failureLimit: 1
 successCondition: result < 0.01
 provider:
 prometheus:
 address: http://prometheus.observability.svc:9090
 query: |
 sum(rate(vllm:request_total{version="v2",status=~"5.."}[5m])) / sum(rate(vllm:request_total{version="v2"}[5m]))
 -
 sum(rate(vllm:request_total{version="v1",status=~"5.."}[5m])) / sum(rate(vllm:request_total{version="v1"}[5m]))

If any of the AnalysisTemplates fails, Argo Rollouts backs out automatically: it sets weight=0 on the canary, alerts the operator and keeps v1 serving 100 %. Human operation is reduced to investigating the failure and deciding whether to relaunch or abort.

Flagger offers a more opinionated alternative: weight progression is automatic as a function of metric success instead of a fixed pause; the operator defines a target (maxWeight: 100, stepWeight: 10, metrics: [...]) and Flagger raises or lowers it according to behaviour. Both are mature as of May 2026; the choice is usually dictated by which service mesh is already in the cluster.

The vLLM detail: why there is no “hot swap” of the model

As of May 2026, vLLM v1 does not support hot swapping the model inside the same replica without restarting the engine. The --model argument is evaluated at start-up; changing it requires re-instantiating the LLMEngine, which restarts connections and discards the KV cache. So the rollout unit is the whole replica: you do not do “v1 loads the new model on one of its GPUs” but “a v2 replica is brought up next to a v1 replica and traffic is shared out via the LB”.

TensorRT-LLM with Triton has a similar mechanism: changing the model requires a reload of the Triton backend. SGLang does not support robust hot swap either. The operational consequence: LLM rollout is always going to cost extra GPUs during the window, and the choice between blue-green, canary and shadow is exactly the question of how many extra and for how long.

The three pitfalls specific to LLM rollout

Pitfall 1 — sticky sessions break canary comparability. If the LoadBalancer does session affinity by client IP (common in NGINX and Traefik with loadbalancer.kubernetes.io/session-affinity: ClientIP), a user A always lands on v2 while B always lands on v1. Load distributions, prompt profiles and client behaviour are not random between the two pools, which statistically invalidates any gate comparison. Solution: for a canary, disable session affinity (sessionAffinity: None) or use affinity by random request-id. If the client app requires stickiness for functional reasons (conversational memory persisted in cache), canary is not the right strategy; use blue-green or shadow.

Pitfall 2 — LLM-as-judge is not an inline real-time gate. The temptation to use quality score as a live gate is strong, but the judge’s latency (2–8 s per sample) makes it unfeasible to evaluate more than a 1–2 % sampling of the traffic, and the results arrive minutes late. Operational solutions: (a) offline pre-canary eval over a golden set as a prerequisite for starting (if it fails, the canary does not even begin); (b) during the canary, capture requests + responses from v2 in real time and run the judge asynchronously in a batch job that finishes before the next step; (c) use statistical embedding drift as a fast proxy for inline quality, and reserve the judge for intermediate gates between steps.

Pitfall 3 — SSE streaming complicates shadowing. Classic traffic mirroring (NGINX mirror, Istio MirrorPolicy) is designed for request/response HTTP: it copies the request, lets the primary server answer the client, and duplicates the request to the secondary, discarding the response. With SSE, the secondary’s response is a continuous stream of several seconds, and keeping two streams in parallel doubles the load on the proxy. Solutions: (a) shadow only non-streaming requests (chat without streaming, embeddings, classification, batch eval), (b) shadow the streaming traffic but with a short timeout on the secondary (discard the shadow if it takes longer than 30 s), (c) replace the shadow with a small-weight canary (1 %), which does handle streaming well.

Applied to typical on-premise hardware

For a generic cluster of 4 nodes × 4×H100 SXM 80 GB = 16 GPUs, serving Llama 70B FP8 with TP=4 (4 possible replicas, one per node):

Blue-green: impossible to keep two complete pools of 4 replicas without extra GPUs. Practical solution: blue-green with reduced pools (2 v1 replicas + 2 v2 replicas) during the window, with accepted capacity degradation (half the sustained RPS SLO), or having a parallel cluster (another node) reserved for rollouts.
Canary: feasible. Start with 3 v1 replicas + 1 v2 replica (25 % nominal weight, though the traffic weight also varies). Move to 2 v1 + 2 v2 at 50 %, then 1 v1 + 3 v2, finally 0 v1 + 4 v2.
Shadow: awkward because of the GPU cost. Reserve it for pre-canary validation of major changes, during a short window (4–8 hours) with shadowed traffic limited to a sample (10–20 % of requests, not 100 %).

For clusters of 8 GPU nodes, the three patterns are all sustainable. The operational rule: the rollout budget is typically 25–30 % of the cluster’s sustained capacity, and buying for the peak plus that head-room makes the capacity planning numbers add up.

What we have not covered (upcoming articles)

Multi-region rollouts: how to coordinate a canary when the cluster is geographically distributed.
A/B testing of prompts (not of models): the same model with two different system prompts, measuring conversion.
Embedding rollback: changing the embedding model of a RAG system means re-embedding the whole corpus, so the canary mechanics are different. See RAG corpus curation.
Feature flags for LLM: granularity per tenant or per feature within the same model.
End-to-end continuous deployment: integration with the retrain pipeline so a new adapter is promoted automatically after passing evals.

References

Argo Rollouts project — argoproj.io/argo-rollouts (Rollout and AnalysisTemplate CRDs).
Flagger project — fluxcd.io/flagger (alternative with automatic progression).
Istio — Traffic Mirroring (mirror configurable at VirtualService level).
NGINX Ingress — Canary annotations (nginx.ingress.kubernetes.io/canary-*).
vLLM project — issue tracker on hot model swap (status as of May 2026: in design, not production-ready).
Hou et al. — DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving (OSDI 2024) — reference on goodput metrics applicable to canary gates.
Bürkner et al. — Statistical methods for detecting model drift in production (various articles on Wasserstein and KL in ML monitoring).

Autoscaling LLM inference on Kubernetes: HPA with custom metrics and KEDA for vLLM

Mon, 01 Jun 2026 16:00:00 +0200

This post complements those on GPU observability for LLM inference (where the metrics that feed the HPA come from), Capacity planning (what ceiling and what head-room the autoscaler assumes) and Continuous batching (which explains why num_requests_waiting is the primary metric).

TL;DR

Classic Kubernetes autoscaling, HPA on cpu or memory, is no use for LLM inference. The reason: the vLLM pod consumes little CPU (the GPU does the work) and the process RSS is flat; both metrics can sit at 30 % while the GPU is saturated and the request queue grows unchecked. The four viable signals that do respond to the real load are: vllm:num_requests_waiting (the queue, the primary metric), vllm:gpu_cache_usage_perc (pressure on the KV cache pool), TTFT P95 via the vllm:time_to_first_token_seconds_bucket histogram (the SLO guarantee) and the batch fill ratio num_requests_running / max_num_seqs (utilisation of the concurrency ceiling). For an HPA to consume Prometheus metrics you need an adapter; as of May 2026 there are two mature options: prometheus-adapter (cluster-wide sigma, static configuration, external.metrics.k8s.io output) and KEDA (ScaledObject with a Prometheus trigger, configurable polling, optional scale to zero, cron integration). KEDA is the dominant option for LLM on a generic cluster because it solves the “warm pool + cron + engine metric” pattern in a single CRD. The dominant operational challenge is not the scaling logic but the cold start: a vLLM pod with Llama 70B BF16 (140 GB) takes between 90 seconds (model pre-cached on a local PV) and 6 minutes (image pull + model download from object store) to serve the first token. The five levers that cut it down are an image pre-pulled via DaemonSet, the model cached on a regional PV or tmpfs, a warm pool with minReplicaCount > 0, predictive scaling via KEDA cron when the traffic pattern is predictable (offices 9–18 h), and parallel model download. The three specific LLM scale-down pitfalls: cutting streaming SSE connections halfway through a response (graceful drain with terminationGracePeriodSeconds ≥ 60 s), scale-out/in oscillation from a badly calibrated stabilization window, and forgetting that the HPA only scales pods; GPU nodes are scaled with cluster-autoscaler over labelled nodepools. This post includes the minimal YAML manifests.

You are here: DEPLOY

The analogy: the bakery with wood-fired ovens

An artisan bakery has three wood-fired ovens. Each oven takes 25 minutes to reach temperature from cold. Once hot, it bakes bread continuously with a run of 18 minutes per batch. The manager wants to maximise bread sold per day without burning wood for nothing, and she knows three things: that there is a demand peak at 7:30 every morning, that Mondays sell almost nothing, and that when the bread on the counter runs out the customers go to the supermarket next door.

The cheap strategy, lighting ovens when there is a queue in the shop, does not work. By the time the queue grows and the manager lights the second oven, that oven will not be ready until 25 minutes later; the customers in that window are lost. The “queue at the counter” signal arrives late.

The smart strategy: light the second oven at 6:55, ahead of the predictable 7:30 peak, and leave it running until 10:00 even though the queue drops at 8:15. Keep the third oven off from Monday to Wednesday because demand does not arrive; light it proactively on Thursdays at 12:00 because historically it goes up. Keep a stock of pre-proved raw dough in the chiller so that when the oven is ready the bread goes in within 30 seconds and there is no two-hour proving wait.

Autoscaling an LLM inference cluster works the same way:

Lighting ovens from cold = reactive scale-out when the queue grows (slow, loses customers).
Proactive cron = predictive scaling when the pattern is known (working hours, expected peaks).
Pre-proved dough = warm pool of replicas with the model loaded but under 0 load.
Switching off ovens with no bread inside = scale-down respecting the active streams (you do not shut the oven with bread in it).

The key metric, “how many customers are in the queue”, is called num_requests_waiting. The metric that says “the oven is about to run out of dough for new loaves” is called gpu_cache_usage_perc. And the quality-of-service metric, “how long the first loaf takes to come out when a new customer walks in”, is called TTFT.

Why HPA on CPU is no use

The classic Kubernetes HPA looks at the pod’s resource.cpu. For a conventional HTTP service, Node.js or a REST API, CPU moves linearly with traffic and the HPA scales with reasonable accuracy. For a vLLM or SGLang pod on GPU, the pod’s CPU typically lives between 5 % and 15 % regardless of whether the GPU is at 30 % or 99 % load: the real work is done by the device, not the process. Result: the CPU-based HPA never triggers scale-out even though the GPU is bursting, and clients pile up in the queue until TTFT P95 crosses the SLO. The operator discovers the problem through the TTFT alert, not through the HPA.

memory is no use either: the vLLM process RSS is flat after startup (model plus buffers loaded in one go); it does not reflect the real pressure on the GPU. The only things that rise and fall with the useful inference load are metrics the engine publishes explicitly: request queue, KV cache pool, SLO latencies. Without an adapter that makes them visible to the HPA, autoscaling is blind.

The four viable signals

Signal 1, vllm:num_requests_waiting (queue). It is the most direct metric: how many requests are waiting to enter the batch. It reacts the instant the target concurrency saturates. It is robust against model changes (the number of requests is the same concept whether it is Llama 7B or 70B). It is the primary metric of the LLM HPA. Typical threshold: target = 5 requests waiting on average; if the queue grows above 5 sustained for 2 minutes, scale out.

Signal 2, vllm:gpu_cache_usage_perc (KV pool). It moves before the queue does: the KV pool fills up while the batch slots are still free, until the engine starts rejecting new requests for OOM prevention and the queue forms. It is therefore predictive: it triggers scale-out before the client notices degradation. Typical threshold: target = 0.85 (85 % of pool used).

Signal 3, TTFT P95. The contractual guarantee. If TTFT P95 leaves the SLO, scale out even though the queue and the KV pool look reasonable (there may be a spike of long prompts). It is reactive, it leaves the SLO before your HPA reacts, but it works as a final guardrail.

Signal 4, batch fill ratio. The num_requests_running / max_num_seqs quotient (the latter is engine config, not a metric). Useful for scale-down: if the ratio stays below 0.4 for 10 minutes, there is spare capacity and replicas can be reduced safely.

The recommended policy combines all four: the queue and the KV pool trigger scale-out (whichever arrives first), TTFT confirms it as a guardrail, and the batch fill ratio manages scale-down. Implementing that in a single HPA demands external metrics; KEDA makes this manageable.

The wiring: KEDA as the Prometheus adapter

KEDA introduces two main CRDs: TriggerAuthentication (how to authenticate against the source) and ScaledObject (which deployment to scale with which triggers). For a vLLM deployment with Prometheus as the source:

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
 name: vllm-llama70b-scaler
 namespace: inference
spec:
 scaleTargetRef:
 name: vllm-llama70b
 minReplicaCount: 2 # warm pool
 maxReplicaCount: 20
 pollingInterval: 15
 cooldownPeriod: 300 # 5 min before scale-down
 advanced:
 horizontalPodAutoscalerConfig:
 behavior:
 scaleDown:
 stabilizationWindowSeconds: 600 # large window to avoid oscillation
 policies:
 - type: Pods
 value: 1
 periodSeconds: 120
 scaleUp:
 stabilizationWindowSeconds: 30
 policies:
 - type: Pods
 value: 2
 periodSeconds: 60
 triggers:
 - type: prometheus
 metadata:
 serverAddress: http://prometheus.observability.svc:9090
 metricName: vllm_queue_depth
 threshold: "5"
 query: |
 avg(vllm:num_requests_waiting{deployment="vllm-llama70b"})
 - type: prometheus
 metadata:
 serverAddress: http://prometheus.observability.svc:9090
 metricName: vllm_kv_cache
 threshold: "0.85"
 query: |
 avg(vllm:gpu_cache_usage_perc{deployment="vllm-llama70b"})
 - type: prometheus
 metadata:
 serverAddress: http://prometheus.observability.svc:9090
 metricName: vllm_ttft_p95
 threshold: "1.5"
 query: |
 histogram_quantile(0.95,
 sum by(le)(rate(vllm:time_to_first_token_seconds_bucket{deployment="vllm-llama70b"}[5m])))

Three non-obvious operational details:

minReplicaCount: 2. This is the warm pool. Keeping at least two replicas guarantees availability if a node is lost and absorbs spikes without waiting for the cold start of the first scale-up. Dropping it to 0 saves GPU off-peak but introduces 90 s–6 min of latency for the first new client.

stabilizationWindowSeconds: 600 on scale-down. Ten minutes. Models are not nginx: if a replica closes prematurely and two minutes later there is another peak, the cold start of a new pod is what the client waits. Better to keep extra replicas around for twice as long as you would for a normal web service.

scaleUp: stabilizationWindowSeconds: 30. Thirty seconds. Scale-out has to be fast: the new pod’s cold start adds its own delay, and if on top of that the HPA waits several more minutes before triggering, the SLO is already broken.

The big operational problem: cold start

A vLLM pod loading Llama 70B goes through these phases before serving the first token:

Phase	Typical time	Can be accelerated with
Image pull (4–6 GB)	30–90 s	DaemonSet pre-pull
Model download (140 GB BF16)	60–300 s	Cached regional PV, S3 + multi-thread
Loading the model into HBM	30–90 s	tmpfs or local NVMe
CUDA graph capture	20–60 s	`--enforce-eager` (slower at runtime but fast startup)
PagedAttention warmup	5–15 s	—
Health check ready	10–30 s	probe tuning

Total without optimisation: 4–10 minutes. That is how long a new replica takes to absorb traffic. With all the levers combined: 45–90 seconds. The difference between those two numbers is the main platform work for LLM autoscaling.

The five levers

Lever 1, pre-pulled image. A trivial DaemonSet runs ctr image pull (or crictl pull) on the GPU nodes as soon as they join the cluster. The inference engine image stays on disk; new pods skip the 30–90 s of pull. Cost: around 6 GB of disk per node.

apiVersion: apps/v1
kind: DaemonSet
metadata: { name: vllm-image-warmer }
spec:
 selector: { matchLabels: { app: vllm-warmer } }
 template:
 metadata: { labels: { app: vllm-warmer } }
 spec:
 nodeSelector: { workload: gpu }
 initContainers:
 - name: pull
 image: vllm/vllm-openai:v0.10.0
 command: ["/bin/true"]
 containers:
 - name: pause
 image: registry.k8s.io/pause:3.10

Lever 2, model on a regional PV. The model download (140 GB BF16 or 35 GB FP8) from central object storage is the dominant component of the cold start. Caching the model on a zone or rack PV, Rook-Ceph RBD or local NVMe provisioned by the operator, cuts 60–300 s down to 5–15 s. The antipattern: downloading the model on every startup from external S3.

volumeMounts:
 - name: model-cache
 mountPath: /models
 readOnly: true
volumes:
 - name: model-cache
 persistentVolumeClaim:
 claimName: llama70b-fp8-pvc  # RWX shared, filled offline

Lever 3, warm pool. minReplicaCount > 0 keeps pre-loaded replicas idle. The cost is idle GPU; the benefit is 0 s of cold start for the first client of a peak. For production clusters with continuous traffic: a warm pool of 2–3 replicas. For nightly batch clusters with 0 traffic: warm pool 0 and accept the cold start, or KEDA with a cron that switches on 10 minutes earlier.

Lever 4, predictive scaling with cron. When the pattern is predictable (offices 9–18 h):

triggers:
 - type: cron
 metadata:
 timezone: Europe/Madrid
 start: "30 8 * * 1-5" # 8:30 Monday–Friday
 end: "0 19 * * 1-5" # 19:00
 desiredReplicas: "6"

Combined with reactive triggers. The HPA scales according to the maximum of the signals: if the cron asks for 6 and the queue asks for 10, the result is 10.

Lever 5, parallel download and an efficient format. For PVs that are not pre-loaded, tools such as nvidia-modelmanager, s5cmd or aria2c parallelise the model download. Going from serial download (~150 MB/s) to parallel with 8 threads (~1.2 GB/s) divides the time by 8. And formats such as safetensors load into HBM faster than the original PyTorch pickle.

When to scale nodes, not just pods

The HPA scales pods. If the cluster has no free GPU nodes, the new pod stays Pending for lack of resources. To scale nodes, you need cluster-autoscaler with a specific GPU nodepool, labelled:

# nodepool config (Karpenter or cluster-autoscaler equivalent)
labels:
 workload: gpu
 gpu-model: h100-sxm-80gb
taints:
 - key: nvidia.com/gpu
 effect: NoSchedule
limits:
 min: 2 nodes
 max: 8 nodes

Without this, the HPA can ask for 10 replicas but the cluster only delivers those that fit on nodes already up. The cold start of a new node (bare metal or cloud provisioning, PXE, OS boot, NVIDIA drivers, cluster join) is much longer than the cold start of a pod: typically 5–15 minutes on preconfigured bare metal, 30–60 minutes on real provisioning. For on-premise clusters, the nodepool must be always sized to the expected maximum, and the “scaling” is only on the pod side. The concept of reactive node scale-out only applies to clouds; on-premise you have to buy for the peak.

Three pitfalls specific to LLM scale-down

Pitfall 1, cutting streaming SSE connections. When a replica enters Terminating, Kubernetes sends SIGTERM to the pod and, by default, kills it 30 seconds later. For vLLM that means cutting streaming SSE connections halfway through the response. The client gets a 502 error with the partial output lost. Solution: terminationGracePeriodSeconds: 120 plus a preStop hook that tells the engine not to accept new requests but to finish those in flight:

spec:
 terminationGracePeriodSeconds: 120
 containers:
 - name: vllm
 lifecycle:
 preStop:
 httpGet:
 path: /shutdown
 port: 8000

This requires the engine to expose a graceful shutdown endpoint; vLLM v1 supports it via --enable-graceful-shutdown. Without it, scale-down breaks the SLO even though the metrics do not capture it (cut requests never enter the TTFT histogram).

Pitfall 2, scale-up/scale-down oscillation. If the scale-down stabilizationWindowSeconds is short (~60 s default), the next drop in the queue triggers scale-down, and two minutes later the next peak triggers scale-up. The system oscillates, pays repeated cold starts, and never reaches a stable regime. Solution: scale-down with a window of 10 minutes minimum and conservative policies (type: Pods, value: 1, periodSeconds: 120, at most one replica fewer every 2 minutes).

Pitfall 3, vllm:num_requests_waiting with avg when there is rebalancing. If two replicas are unbalanced (one with queue 20, the other with queue 0), avg gives 10 and the HPA triggers scale-out when the right move would be to rebalance via the load balancer. To detect it: add an alert on stddev(vllm:num_requests_waiting) per deployment. If the dispersion is high, the problem is not capacity but routing.

Full example manifest

For a vLLM deployment with Llama 70B FP8 on 4×H100 SXM per replica, KEDA with warm pool 2:

apiVersion: apps/v1
kind: Deployment
metadata:
 name: vllm-llama70b
 namespace: inference
spec:
 replicas: 2 # managed by KEDA afterwards
 selector: { matchLabels: { app: vllm-llama70b } }
 template:
 metadata:
 labels: { app: vllm-llama70b, deployment: vllm-llama70b }
 spec:
 terminationGracePeriodSeconds: 120
 nodeSelector: { workload: gpu, gpu-model: h100-sxm-80gb }
 tolerations:
 - key: nvidia.com/gpu
 operator: Exists
 effect: NoSchedule
 containers:
 - name: vllm
 image: vllm/vllm-openai:v0.10.0
 args:
 - --model=/models/llama-3.3-70b-fp8
 - --tensor-parallel-size=4
 - --max-num-seqs=64
 - --enable-prefix-caching
 - --enable-graceful-shutdown
 ports:
 - { name: http, containerPort: 8000 }
 - { name: metrics, containerPort: 8000 }
 resources:
 limits:
 nvidia.com/gpu: "4"
 memory: 200Gi
 readinessProbe:
 httpGet: { path: /health, port: 8000 }
 initialDelaySeconds: 60
 periodSeconds: 10
 failureThreshold: 30 # tolerates the warmup
 lifecycle:
 preStop:
 httpGet: { path: /shutdown, port: 8000 }
 volumeMounts:
 - { name: model-cache, mountPath: /models, readOnly: true }
 - { name: dshm, mountPath: /dev/shm }
 volumes:
 - name: model-cache
 persistentVolumeClaim: { claimName: llama70b-fp8-pvc }
 - name: dshm
 emptyDir: { medium: Memory, sizeLimit: 16Gi }
---
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata: { name: vllm-llama70b-metrics, namespace: inference }
spec:
 selector: { matchLabels: { app: vllm-llama70b } }
 podMetricsEndpoints:
 - port: metrics
 path: /metrics
 interval: 15s
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata: { name: vllm-llama70b-scaler, namespace: inference }
spec:
 scaleTargetRef: { name: vllm-llama70b }
 minReplicaCount: 2
 maxReplicaCount: 20
 pollingInterval: 15
 cooldownPeriod: 300
 advanced:
 horizontalPodAutoscalerConfig:
 behavior:
 scaleDown:
 stabilizationWindowSeconds: 600
 policies:
 - { type: Pods, value: 1, periodSeconds: 120 }
 scaleUp:
 stabilizationWindowSeconds: 30
 policies:
 - { type: Pods, value: 2, periodSeconds: 60 }
 triggers:
 - type: prometheus
 metadata:
 serverAddress: http://prometheus.observability.svc:9090
 metricName: vllm_queue
 threshold: "5"
 query: avg(vllm:num_requests_waiting{deployment="vllm-llama70b"})
 - type: prometheus
 metadata:
 serverAddress: http://prometheus.observability.svc:9090
 metricName: vllm_kv
 threshold: "0.85"
 query: avg(vllm:gpu_cache_usage_perc{deployment="vllm-llama70b"})
 - type: cron
 metadata:
 timezone: Europe/Madrid
 start: "30 8 * * 1-5"
 end: "0 19 * * 1-5"
 desiredReplicas: "6"

This set is the minimum viable one for LLM autoscaling on a generic cluster with the NVIDIA GPU Operator. Each team adapts it to its own SLO.

Applied to typical on-premise hardware

For a generic cluster of 4×H100 SXM 80 GB per node, 4 GPU nodes:

Each node hosts one vLLM replica with TP=4 and Llama 70B FP8 (one model per node, they are not shared).
Warm pool of 2 replicas off-peak; the KEDA cron raises it to 4 during working hours.
Cluster-autoscaler does not apply (4 physical nodes bought; scaling is pods only). The number of concurrent replicas is at most the number of available nodes (if each replica uses all 4 GPUs of the whole node).
If the sizing requires more simultaneous replicas than nodes, there are two routes: (a) lower the TP of each replica so two fit per node, (b) expand the physical nodepool. The decision is dictated by capacity planning, see Capacity planning for on-premise LLM inference.

KEDA event volume: around 5 evaluations/min per ScaledObject. For 10 models served in parallel, 3,000 evaluations/h. Manageable with one KEDA operator per cluster.

What we have not covered (upcoming articles)

Cluster-autoscaler for on-premise GPU nodes: how to orchestrate bare metal provisioning (Tinkerbell, Metal³) as a function of demand.
Multi-cluster autoscaling: scaling across clusters in different DCs for geographic resilience.
Cost-aware autoscaling: prioritising nodes by hourly energy cost (on clusters with indexed tariffs).
Predictive ML-based scaling: instead of a static cron, training a model that predicts demand 30 minutes ahead.
Multi-tenant quotas and fairness: KEDA with namespace quotas so that one tenant does not monopolise the HPA.

References

KEDA project — keda.sh (documentación oficial de triggers Prometheus y cron).
Kubernetes — Horizontal Pod Autoscaler walkthrough (kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale).
NVIDIA — GPU Operator on Kubernetes (Helm chart oficial con DaemonSet de drivers y DCGM).
vLLM project — production_monitoring/ (métricas Prometheus expuestas por el servidor).
Karpenter — NodePool spec (etiquetado y taints para nodepools GPU).
Cluster Autoscaler — Scaling GPU nodes (caveats de descubrimiento de recursos GPU).
Kubernetes — Pod lifecycle and termination (preStop, terminationGracePeriodSeconds).

GPU observability for LLM inference: the twelve DCGM and vLLM metrics that dictate the health of your production

Mon, 01 Jun 2026 15:30:00 +0200

This post complements those on LLM tracing with OpenTelemetry GenAI (the tracing layer above the metrics), Capacity planning (what was sized and what must be watched) and Continuous batching (the mechanism that explains several of the engine metrics).

TL;DR

Observability for an LLM inference cluster is built on two complementary sources: the GPU hardware metrics exposed by the DCGM (Data Center GPU Manager) Exporter, part of the NVIDIA GPU Operator, and the metrics from the inference engine (vLLM, SGLang, TensorRT-LLM) exposed on a Prometheus-compatible /metrics. Neither is enough on its own. The classic nvidia-smi metric called GPU utilization is misleading for LLMs: it reads high whenever any kernel is running, without distinguishing tensor cores burning from SMs waiting on HBM. The full cockpit has twelve DCGM metrics in four families (compute: DCGM_FI_PROF_SM_OCCUPANCY, DCGM_FI_PROF_PIPE_TENSOR_ACTIVE, DCGM_FI_PROF_DRAM_ACTIVE; memory: DCGM_FI_DEV_FB_USED, DCGM_FI_DEV_FB_FREE, DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL; thermal-power: DCGM_FI_DEV_GPU_TEMP, DCGM_FI_DEV_POWER_USAGE, DCGM_FI_DEV_CLOCK_THROTTLE_REASONS; health: DCGM_FI_DEV_XID_ERRORS, DCGM_FI_DEV_ECC_DBE_VOL_TOTAL, DCGM_FI_DEV_RETIRED_DBE) and five vLLM engine metrics (vllm:num_requests_running, vllm:num_requests_waiting, vllm:gpu_cache_usage_perc, vllm:time_to_first_token_seconds, vllm:time_per_output_token_seconds). Each one has a defensible green/amber/red threshold, a PromQL query for alerting, and at least one common false reading that confuses the junior operator. The six critical alerts that any production cluster must fire are: HBM > 92 %, thermal or power throttle, XID error, ECC double-bit, KV cache pool > 95 %, and TTFT P95 outside SLO for 5 minutes. The point of having this panel: that the operator on shift diagnoses the origin of a degradation in under five minutes, without opening an SSH console to the GPUs. When that holds, the cluster has moved to professional operation; until then, it is operated on intuition.

You are here: OBSERVE (the other half of tracing)

Tracing, already covered in LLM tracing with OpenTelemetry GenAI, answers what happened in this specific request. Metrics answer what is happening across the cluster in aggregate. They are complementary: an alert on the metrics side tells you “the cluster is degrading”, tracing tells you “and here is the specific trace that proves it”. A cluster without tracing but with metrics operates; a cluster without metrics but with tracing does not operate, it debugs.

The analogy: the cockpit of a modern aircraft

In a modern commercial aircraft, the pilot’s instrument panel has more than 70 active indicators. If there were only one, the altimeter, say, the aircraft would fly into the ground at the first moment of low visibility. You need the altimeter and the attitude indicator, and the airspeed indicator, and the turn indicator, and the fuel gauge, and the oil pressure gauges for each engine, and the turbine exit temperatures. Each answers a different question. And together they cover the operational question: is the aircraft healthy, is it where it should be, and is it going where we want?

Observability for an LLM inference cluster works the same way. A single metric, “GPU utilization 99 %”, answers nothing useful. It is like looking only at the car’s odometer to diagnose why the engine is making a noise. The full cockpit is twelve instruments on the hardware side plus five on the inference engine side, organised into families that answer different questions:

Compute and efficiency: are the tensor cores doing the work we expect, or are they waiting?
Memory: is there VRAM left for new requests, or are we on the edge of OOM?
Thermal and power: is the hardware healthy, or is it silently limiting throughput?
Health and errors: is there hardware degradation under way (ECC, XID, NVLink)?
Inference engine: is the queue growing, is the KV pool saturated, is the SLO being met?

The first four answer “is the GPU fine?”. The fifth answers “is it delivering the service we promised?”. The two questions are different and both must have an answer at a glance.

Why `nvidia-smi` `GPU-Util` misleads with LLMs

The classic metric that appears in nvidia-smi as GPU-Util corresponds to DCGM_FI_DEV_GPU_UTIL. Its official definition: “percentage of time during which one or more kernels were executing on the GPU”. The problem with LLMs: the decode phase is memory-bound, not compute-bound. When the inference engine decodes token by token, the GPU spends 90 % of the time waiting for the HBM to finish delivering the model weights and the KV cache. There is a kernel running (the HBM read), so GPU-Util reports values close to 100 %. But the tensor cores are idle: the bottleneck is memory, not compute.

Practical result: the operator sees “GPU-Util 99 %” in Grafana and assumes “GPU saturated, no more load can go in”. But the reality may be “compute at 25 %, HBM saturated at 95 %”, which changes the operational decisions (quantisation, batch size, parallelism). The classic metric lies by oversimplifying.

The right move is to look at the three DCGM profiling metrics of the _FI_PROF_* subsystem:

DCGM_FI_PROF_SM_OCCUPANCY: ratio of active warps over the maximum per SM. Is there parallel work?
DCGM_FI_PROF_PIPE_TENSOR_ACTIVE: % of cycles with tensor cores effectively active. Is compute working?
DCGM_FI_PROF_DRAM_ACTIVE: % of cycles with HBM transferring. Is memory saturated?

A typical decode-bound GPU running Llama 70B on an H100 shows: SM occupancy 35–55 %, tensor active 15–30 %, DRAM active 80–95 %. That is the real “saturated GPU” for LLMs. The three together tell the regimes apart; none of them alone says anything actionable.

How they are wired up in production

The platform side is covered in Five maturity levels (level 4, the GPU plane) and Seven deployment phases (phase F5). For the observer, the key pieces are:

NVIDIA GPU Operator. Helm manifests that deploy on each GPU node: drivers, container toolkit, MIG manager and DCGM Exporter. The last one exposes /metrics in Prometheus format with all the DCGM_FI_* listed above. It is scraped from the cluster’s internal Prometheus.

Inference engine. vLLM exposes /metrics on port 8000 (default) with vllm:* metrics. SGLang exposes it too, with the sglang: prefix. TensorRT-LLM exposes it through Triton Inference Server with the nv_inference: prefix. The basic naming convention is similar across the three engines; the thresholds and queries in this post assume vLLM, but they translate.

ServiceMonitor / PodMonitor. The Prometheus operator resource that says what to scrape. Minimal example:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
 name: vllm-inference
spec:
 selector:
 matchLabels: { app: vllm }
 podMetricsEndpoints:
 - port: metrics
 interval: 15s

Dashboards. The NVIDIA operator publishes reference Grafana dashboards for DCGM in nvidia/dcgm-exporter (the official repo). vLLM publishes one in vllm-project/vllm (the examples/ folder). Both work as a base; each team adds the panels specific to its own SLO.

The twelve DCGM metrics organised by family

Family 1 — Compute

DCGM_FI_PROF_SM_OCCUPANCY: ratio of active warps per SM over the maximum possible. Value between 0 and 1.

Green: 0.30–0.70 (typical LLM decode regime).
Amber: < 0.20 sustained (batch too small, GPU under-used in parallelism).
Red: 0.95 sustained with low DRAM_ACTIVE (pathological kernel saturating the SMs).

DCGM_FI_PROF_PIPE_TENSOR_ACTIVE: % of cycles with tensor cores executing. The key metric for “is compute producing?”.

Green in prefill: 50–80 %.
Green in decode: 15–30 % (decode is memory-bound, this is not a symptom of a problem).
Red: < 5 % sustained in prefill, or the engine is not using the tensor cores at all (bad config, incompatible format).

DCGM_FI_PROF_DRAM_ACTIVE: % of cycles with HBM transferring data. The key metric for detecting memory saturation.

Green in decode: 60–85 %.
Amber: > 90 % sustained (HBM is a firm bottleneck, which explains high TPOT).
Red: > 95 % sustained with KV cache pool < 70 % (something other than the engine is asking for HBM; investigate leaks).

Family 2 — Memory

DCGM_FI_DEV_FB_USED: Frame Buffer (HBM) used, in MiB.

Green: 70–85 % of total.
Amber: 86–92 %.
Red: > 92 % (risk of OOM on the next paged-attention allocation).

PromQL for the cluster-wide percentage: 100 * sum(DCGM_FI_DEV_FB_USED) / sum(DCGM_FI_DEV_FB_TOTAL).

DCGM_FI_DEV_FB_FREE: free Frame Buffer. Complementary to the previous one; useful for absolute alerts (< 4096 MiB free).

DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL: aggregate NVLink bandwidth in MB/s. For TP (tensor parallel) topologies that cross GPUs over NVLink, this metric reveals whether the parallelism split is saturating the bus.

Green: varies with topology. On 4×H100 SXM with NVLink 4.0, theoretical capacity is 450 GB/s per GPU. A typical TP=4 regime: 50–150 GB/s sustained.
Red: > 90 % of capacity sustained (check whether the model would fit with lower TP or with pipeline parallel).

Family 3 — Thermal and power

DCGM_FI_DEV_GPU_TEMP: die temperature in °C.

Green: < 75 °C.
Amber: 75–82 °C.
Red: > 83 °C (close to the H100’s automatic thermal throttle; check ventilation, air flow, rack inlet temperature).

DCGM_FI_DEV_POWER_USAGE: draw in watts. For the H100 SXM, nominal TDP is 700 W. Useful for three things: spotting an unusually low workload (suspect idle or a stall), billing real energy cost, and firing alerts if the draw approaches the PDU limit.

DCGM_FI_DEV_CLOCK_THROTTLE_REASONS: encoded bitmap with the active throttle reasons. This is the metric that silently explains TPOT degradations.

Relevant bits:

0x0000000000000001: Idle (not a problem).
0x0000000000000002: App clocks setting.
0x0000000000000004: SW Power Cap (software limit, e.g. from nvidia-smi -pl).
0x0000000000000008: HW Slowdown.
0x0000000000000010: Sync Boost (NVIDIA Sync).
0x0000000000000020: SW Thermal Slowdown (software thermal limit).
0x0000000000000040: HW Thermal Slowdown (hardware thermal limit, an emergency).
0x0000000000000080: HW Power Brake Slowdown (PSU voltage drop).
0x0000000000000100: Display Clock Setting.

Any throttle other than Idle with a sustained value > 0 is an alert. TPOT degradation with DRAM_ACTIVE already high and thermal throttle active is the classic “the rack is badly ventilated, it is not the engine’s fault”.

Family 4 — Health

DCGM_FI_DEV_XID_ERRORS: cumulative counter of driver XID errors. XIDs are critical event codes that NVIDIA documents exhaustively (XID 13: graphics engine exception; XID 31: GPU memory page fault; XID 43: reset channel verif error; XID 79: GPU has fallen off the bus; XID 95: uncontained ECC error; and so on). Any increment is an immediate alert: many XIDs require a node reset or an RMA of the GPU.

DCGM_FI_DEV_ECC_DBE_VOL_TOTAL: volatile double-bit ECC errors (uncorrectable). Unlike single-bit ones (which ECC corrects silently and which are counted in DCGM_FI_DEV_ECC_SBE_*), double-bit errors corrupt data. Any value > 0 is a critical alert: the GPU must be drained and inspected.

DCGM_FI_DEV_RETIRED_DBE: physical HBM pages retired because of accumulated double-bit errors. NVIDIA retires faulty pages automatically to prevent future corruption. More than 4–8 retired pages on one GPU suggests silicon degradation: document it and plan a replacement in the next maintenance window.

The five inference engine metrics (vLLM)

The DCGM metrics answer “is the GPU healthy?”. The engine metrics answer “is the service meeting the SLO?”. Without them, you know the hardware works but you do not know whether the clients are happy.

vllm:num_requests_running: requests currently in the batch. If it reaches the configured --max-num-seqs and does not come down, the engine is saturated on concurrency (check VRAM and rebalance through the autoscaler, see Autoscaling LLM on Kubernetes).

vllm:num_requests_waiting: requests queued, not yet in the batch. Any value > 0 sustained over minutes means the cluster is not scaling with the load. This is the primary metric for HPA.

vllm:gpu_cache_usage_perc: % of the KV cache pool in use.

Green: 50–80 %.
Amber: 80–92 %.
Red: > 92 % (risk of preempt-on-OOM: vLLM will drop requests to free memory, which raises TTFT visibly).

vllm:time_to_first_token_seconds: histogram of TTFT per request. Consumed as histogram_quantile(0.95, sum by(le)(rate(vllm:time_to_first_token_seconds_bucket[5m]))). Compared against the TTFT P95 SLO, it fires the primary service alert.

vllm:time_per_output_token_seconds: histogram of TPOT. Equivalent to the previous one but for streaming smoothness. Compared against the TPOT P95 SLO, it fires the secondary alert.

The six alerts that must page in production

Any serious production cluster fires these six alerts to a channel with an on-call rotation. Without them, the SLO is met by luck, not by process.

groups:
 - name: gpu-llm-critical
 rules:
 - alert: GpuHbmNearOom
 expr: 100 * (DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_TOTAL) > 92
 for: 2m
 labels: { severity: critical }
 annotations:
 summary: "HBM on {{ $labels.gpu }} at {{ $value }}% — OOM risk"

 - alert: GpuThermalOrPowerThrottle
 expr: (DCGM_FI_DEV_CLOCK_THROTTLE_REASONS != 0) and ignoring(reason) (DCGM_FI_DEV_CLOCK_THROTTLE_REASONS != 1)
 for: 1m
 labels: { severity: warning }
 annotations:
 summary: "GPU {{ $labels.gpu }} throttling (reasons={{ $value }})"

 - alert: GpuXidErrorDetected
 expr: increase(DCGM_FI_DEV_XID_ERRORS[5m]) > 0
 labels: { severity: critical }
 annotations:
 summary: "XID error on GPU {{ $labels.gpu }} — investigate immediately"

 - alert: GpuEccDoubleBit
 expr: DCGM_FI_DEV_ECC_DBE_VOL_TOTAL > 0
 labels: { severity: critical }
 annotations:
 summary: "ECC double-bit on GPU {{ $labels.gpu }} — drain node"

 - alert: VllmKvCachePoolNearFull
 expr: vllm:gpu_cache_usage_perc > 0.95
 for: 3m
 labels: { severity: warning }
 annotations:
 summary: "KV cache pool > 95% on {{ $labels.instance }}"

 - alert: VllmTtftP95OutOfSlo
 expr: histogram_quantile(0.95, sum by(le, instance)(rate(vllm:time_to_first_token_seconds_bucket[5m]))) > 1.5
 for: 5m
 labels: { severity: warning }
 annotations:
 summary: "TTFT P95 over SLO ({{ $value }}s > 1.5s)"

These six cover 80 % of the incidents that affect the SLO. The remaining 20 % demand investigation with tracing (see LLM tracing with OpenTelemetry GenAI).

Master table: thresholds and queries

Metric	Green	Amber	Red	Base query (PromQL)
SM occupancy	0.30–0.70	0.15–0.30	< 0.10 sustained	`DCGM_FI_PROF_SM_OCCUPANCY`
Tensor active (decode)	15–30 %	< 10 %	< 3 %	`DCGM_FI_PROF_PIPE_TENSOR_ACTIVE`
DRAM active	60–85 %	85–95 %	> 95 % with low KV	`DCGM_FI_PROF_DRAM_ACTIVE`
FB used	70–85 %	86–92 %	> 92 %	`100 * DCGM_FI_DEV_FB_USED / DCGM_FI_DEV_FB_TOTAL`
NVLink BW	< 70 % cap	70–90 % cap	> 90 % cap	`DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL`
GPU temp	< 75 °C	75–82 °C	> 83 °C	`DCGM_FI_DEV_GPU_TEMP`
Power usage	< 90 % TDP	90–98 % TDP	> 98 % TDP	`DCGM_FI_DEV_POWER_USAGE`
Throttle reasons	0 or Idle	App/SW	HW Therm/Power	`DCGM_FI_DEV_CLOCK_THROTTLE_REASONS`
XID errors	no change	—	any delta	`increase(DCGM_FI_DEV_XID_ERRORS[5m])`
ECC DBE	0	—	> 0	`DCGM_FI_DEV_ECC_DBE_VOL_TOTAL`
Retired pages	< 4	4–8	> 8	`DCGM_FI_DEV_RETIRED_DBE`
KV cache used	50–80 %	80–92 %	> 92 %	`vllm:gpu_cache_usage_perc`
Requests waiting	0	1–5 sustained	> 10 sustained	`vllm:num_requests_waiting`
TTFT P95	< SLO	80–100 % SLO	> SLO	see alert query above
TPOT P95	< SLO	80–100 % SLO	> SLO	`histogram_quantile(0.95, sum by(le)(rate(vllm:time_per_output_token_seconds_bucket[5m])))`

Three pitfalls that confuse the junior operator

Pitfall 1 — “GPU-Util at 99 % = saturated”. As explained at the start, DCGM_FI_DEV_GPU_UTIL lights up with any kernel. The right move is to look at the three _PROF_* metrics (SM occupancy, tensor active, DRAM active) together. GPU util 99 % + tensor active 8 % + DRAM active 92 % means “saturated by memory, not compute”; GPU util 99 % + tensor active 75 % + DRAM active 50 % means “saturated by compute, prefill heavy”. The two situations call for different levers.

Pitfall 2 — confusing single-bit ECC (SBE) with double-bit (DBE). Single-bit errors are corrected silently and are unavoidable in any HBM under load (cosmic radiation, voltage fluctuations). An SBE counter growing slowly is not an alert, it is physics. DBE is: it corrupts data. Telling the two metrics apart avoids false alarms and false negatives in equal measure.

Pitfall 3 — alerting on num_requests_waiting > 0 without context. An instantaneous value of 1 or 2 during a spike is normal. What matters is a sustained queue: use for: 5m with a threshold of 3–5. Without that window, the system floods the alert channel with noise.

Applied to typical on-premise hardware

For a generic cluster of 4×H100 SXM 80 GB with intra-node NVLink:

DCGM Exporter deployed through the NVIDIA GPU Operator, one DaemonSet per GPU node.
Internal Prometheus with 30-day retention for high-frequency metrics, 1 year for downsampled ones (Thanos/Mimir if the volume justifies it).
Grafana with three standard dashboards: GPU hardware (DCGM), engine (vLLM), SLO (TTFT/TPOT/RPS against written targets).
Alertmanager with an on-call rotation and rate limiting through silences grouped by node.
Cardinality kept under control: gpu (local id), node, pod, model. Do not add request_id or other high-cardinality labels to metrics; that is tracing’s job.

Estimated volume for a 16-GPU cluster scraped every 15 s: around 2 million samples/min, around 25 GB/day of raw Prometheus. Manageable with one Prometheus per cluster plus retention; if the team scales beyond 64 GPUs, consider a Thanos sidecar or VictoriaMetrics. See OSS LLMOps tooling catalogue for equivalent alternatives.

What we have not covered (upcoming articles)

Tracing of LLM workloads: already covered in LLM tracing with OpenTelemetry GenAI.
Autoscaling based on these metrics: see Autoscaling LLM on Kubernetes.
Incident response runbooks: how each of these alerts translates into concrete action (drain, restart, RMA, scaling, rollback).
Cost accounting: using DCGM_FI_DEV_POWER_USAGE and vllm:request_success_total for per-tenant cost showback.
Multi-tenant fairness monitoring: when several tenants share a cluster, which metrics detect that one of them is hogging the KV cache.

References

NVIDIA — DCGM Exporter (repo nvidia/dcgm-exporter, documented metrics and units).
NVIDIA — DCGM Field Identifiers reference (complete list of DCGM_FI_*).
NVIDIA — XID Errors documentation (catalogue of XID codes and remediation procedures).
NVIDIA — NVIDIA GPU Operator (official Helm chart).
vLLM project — examples/production_monitoring/ (reference PromQL and Grafana dashboards).
Prometheus — Histogram and summary best practices (for building defensible percentile queries).
NVIDIA — H100 Tensor Core GPU datasheet (TDP, HBM bandwidth, NVLink capacities).

Capacity planning for on-premise LLM inference: how to size GPUs from an SLO

Mon, 01 Jun 2026 15:00:00 +0200

This post complements the ones on KV cache (the piece that dominates the VRAM budget), Continuous batching (what defines the effective utilisation of the compute) and Seven layers of the stack (the pieces the sizing takes for granted). Before reading this one, make sure your team has written down the SLOs it is going to chase; without that input the calculation is not defensible.

TL;DR

LLM inference capacity planning does not answer “how many tokens per second does a GPU give” — that question has no universal answer, because throughput depends on concurrency, the prefill/decode split, context length, the inference engine and quantisation. The right question has three inputs (SLO: TTFT P95, TPOT P95, sustained RPS), a hardware reference (GPU model, VRAM, HBM bandwidth, effective FLOPs) and a model (parameters, GQA/MHA/MoE architecture, weight format). The calculation resolves into two coupled budgets that cross each other. VRAM budget: from the GPU’s total you subtract model weights and activations, what remains is the KV cache budget, and from there you derive the maximum concurrency possible at the average context you expect. Time budget: the engine (vLLM, SGLang, TensorRT-LLM) has a tokens/s ceiling in decode given by HBM bandwidth and another in prefill given by useful FLOPs; from there you derive the expected TPOT and, dividing prefill_tokens by prefill throughput, the expected TTFT. Both budgets must meet the SLO simultaneously: whichever is tighter dictates the sizing. On the Llama 70B BF16 example with tensor parallel 4 on 4×H100 SXM, a single replica saturates at ~28 concurrent requests and delivers ~3,200 tokens/s of aggregate decode with a median TPOT of 35 ms; for 200 sustained RPS at a profile of 800 prompt tokens + 250 of output, you need between 4 and 5 replicas with a 25 % cushion over the observed peak. Quantisation (FP8 → INT4) divides VRAM cost and decode time by between 1.5 and 4×, but it degrades quality measurably — it is not assumed free, it is validated with evals. The five habitual traps: confusing the mean with P95, ignoring the prefill/decode split of the real workload, sizing without head-room for retraining or rollback, forgetting that a GPU at 100 % SM util means nothing if the HBM is saturated, and not documenting the assumptions behind the calculation (a sizing with no written assumptions is a throwaway calculation).

You are here: DEPLOY (with one foot in OBSERVE)

Capacity planning is a piece with dual residency. It lives in DEPLOY because without a valid sizing you do not buy hardware or configure the inference engine. But its input is real observations: the distribution of prompt and output lengths, the prefill/decode mix of the workload, the real P95s already being seen in pre-production. Without that data the calculation is a napkin sketch, defensible only until the first customer arrives who does not fit the assumed average.

The analogy: the hotel with variable-sized rooms

Imagine a hotel where the rooms have no fixed size: each guest pays for the square metres they need, and the building’s floor plan reorganises itself dynamically to accommodate whoever arrives. Management wants to maximise occupancy, but it has two real constraints and one quality metric.

Constraint 1 — physical space. The floor has 1,000 m² in total. If a family comes in needing 200 m², that family occupies that surface and it cannot be handed to the next guest. The largest room limits how many simultaneous guests fit.

Constraint 2 — service staff. There are 10 receptionists. Each can handle the check-in of one guest every two minutes. When 60 guests arrive within an hour, the last ones wait in a queue; the time from walking into reception to receiving their key depends on how many are ahead of them.

Quality metric — a promise about time. The brochure says “check-in in under 15 minutes”. If too many guests arrive at once, that promise breaks even if there is physical space free.

The physical space is the GPU’s VRAM. Each room is a request with its KV cache (larger the longer the conversation). The receptionists are the compute units (Streaming Multiprocessors + Tensor Cores). The check-in is the prefill phase; the nights the guest spends afterwards are the decode steps. The 15-minute promise is the TTFT P95 SLO.

The hotel’s capacity planning is exactly this: given the expected guest profile (how many arrive per hour, how much space they ask for on average, how many minutes of waiting they tolerate), work out how many floors and how many receptionists are needed. It is not done by estimating “rooms per hour” in the abstract; it is done by crossing the two budgets with the promise about time. The analogy holds up the calculation all the way to the end.

The three SLO inputs

Before putting a single number on the sheet, you have to write down the three dimensions of the SLO. Without this, the calculation is aesthetics, not engineering.

TTFT P95 (Time-To-First-Token). The time from the client sending the request until it receives the first token. It is dominated by the prefill phase (processing the whole prompt in one go) plus the scheduler queue. For conversational chat, a reasonable target sits between 0.5 and 2 seconds P95. For coding assistants with large prompts (5–10 K tokens of context), between 2 and 4 s P95. Below 500 ms you enter UX territory for voice-style conversations, but that demands serious architectural compromises.

TPOT P95 (Time-Per-Output-Token). The time between consecutive tokens during decode. It dominates the “perceived fluency” of streaming. Above 80 ms/token the human reader perceives pauses; below 30 ms/token the output flows faster than it can be read. The usual industry target: 40–60 ms P95.

Sustained RPS meeting the SLO. The throughput the system must support without violating TTFT or TPOT. This is DistServe’s key metric, called goodput (see Continuous batching). “200 RPS peak” is not the same as “200 RPS with TTFT P95 ≤ 1.5 s”. Without the SLO condition, the RPS number means nothing.

These three dimensions come with a workload profile: the distribution of prompt and output lengths. Medians are not enough; you need P50, P95, P99. A badly measured profile is the main reason sizings fail.

The central formula: two budgets that cross

The calculation resolves into two independent sums that are then crossed. The smaller of the two rules.

VRAM budget

For a GPU with total VRAM $V$, the space available for KV cache is:

$$V_{\text{kv}} = V - V_{\text{model}} - V_{\text{activations}} - V_{\text{overhead}}$$

where:

$V_{\text{model}}$ is the size of the weights: for a model of $P$ parameters in a format of $b$ bytes/parameter, $V_{\text{model}} = P \cdot b$. Llama 70B BF16 = $70 \times 10^9 \times 2 = 140$ GB. Under tensor parallel TP=4, each GPU carries $140 / 4 = 35$ GB.
$V_{\text{activations}}$ are the intermediate buffers of the forward pass. For vLLM with a reasonable batch, between 2 and 6 GB per GPU depending on batch size and maximum length.
$V_{\text{overhead}}$ is the CUDA context, NCCL buffers, the PagedAttention pool, reserved paged blocks. 2–4 GB typical.

The KV cache budget per GPU is what is left over. For an H100 SXM 80 GB with Llama 70B TP=4 BF16:

$$V_{\text{kv}} = 80 - 35 - 4 - 3 = 38 \text{ GB per GPU} = 152 \text{ GB aggregated over TP=4}$$

The KV cache cost per token for a model with $L$ layers, $H_{\text{kv}}$ KV heads (GQA), per-head dimension $d_h$, in a format of $b$ bytes:

$$\text{kv per token} = 2 \cdot L \cdot H_{\text{kv}} \cdot d_h \cdot b$$

The factor of 2 is because both K and V are stored. For Llama 70B (L=80, $H_{\text{kv}}$=8 with GQA, $d_h$=128, BF16 = 2 bytes):

$$\text{kv per token} = 2 \cdot 80 \cdot 8 \cdot 128 \cdot 2 = 327,680 \text{ bytes} = 320 \text{ KB/token}$$

And the maximum concurrency at average context $C$:

$$N_{\text{max}} = \frac{V_{\text{kv}}}{C \cdot \text{kv per token}}$$

With an aggregated $V_{\text{kv}}$ of 152 GB and an average context of 1,500 tokens (800 prompt + 700 generated at the worst moment of the conversation):

$$N_{\text{max}} = \frac{152 \times 10^9}{1,500 \cdot 320 \times 10^3} \approx 316 \text{ concurrent requests}$$

This is the physical ceiling of concurrency for that replica. It is not what you are going to use; it is what you cannot exceed without OOM. The operational number sits well below it (head-room for spikes).

Time budget

Two sub-calculations come in here: decode (memory-bound) and prefill (compute-bound).

Decode TPOT. For every token generated, the model weights (the ones relevant to that request) have to be walked through and the accumulated KV cache has to be read. The bottleneck is HBM bandwidth. For a GPU with bandwidth $B$ GB/s and a model of $V_{\text{model per gpu}}$ GB of weights:

$$\text{tpot}_{\text{theoretical}} \approx \frac{V_{\text{model per gpu}}}{B}$$

For an H100 SXM with HBM3 at 3.35 TB/s and Llama 70B TP=4 BF16 (35 GB/GPU):

$$\text{tpot}_{\text{theoretical}} \approx \frac{35}{3,350} \approx 10.4 \text{ ms/token}$$

This is the best theoretical case with batch=1 and 100 % HBM efficiency. In practice vLLM on H100 with Llama 70B TP=4 reaches 12–18 ms/token at low batch and 30–45 ms/token at high batch (at concurrency 32, tokens compete for the shared HBM). The defensible operational number: 35 ms/token at concurrency 24–32.

Prefill throughput. Prefill processes N prompt tokens in a single forward pass. It is compute-bound: the bottleneck is FLOPs. For an H100 SXM with 989 sustained BF16 TFLOPs and Llama 70B (each forward pass costs roughly $2 \cdot P \cdot N$ FLOPs per sequence of length N):

$$\text{prefill tps} = \frac{4 \cdot \text{TFLOPs} \cdot \eta}{2 \cdot P} = \frac{4 \cdot 989 \times 10^{12} \cdot 0.5}{2 \cdot 70 \times 10^9} \approx 14,000 \text{ tokens/s}$$

(the factor of 4 is the GPUs in TP, $\eta$ is real efficiency between 0.4 and 0.6 on H100). An 800-token prompt takes this long in prefill:

$$\text{prefill time} = \frac{800}{14,000} \approx 57 \text{ ms}$$

Adding a typical queue of 100–300 ms at high concurrency, TTFT P95 ≈ 350–500 ms for that profile. Well below the 1.5 s target; there is margin.

The crossing

The real operational concurrency $N_{\text{op}}$ is the minimum of the VRAM ceiling, the concurrency at which TPOT starts to degrade past the SLO, and the concurrency at which TTFT starts to degrade past the SLO (prefill queue). For the example:

VRAM ceiling: 316.
TPOT degrades to 80 ms (SLO) at around concurrency ~80–100 (measured empirically with a benchmark, not a closed formula).
TTFT degrades to 1.5 s at around concurrency ~40–60 because of the prefill queue.

The replica’s operational concurrency is ~50. Applying 25 % head-room for spikes and rebalancing, target concurrency per replica ≈ 35–40.

Step-by-step worksheet: Llama 70B BF16 on 4×H100 SXM

The exercise’s input:

SLO: TTFT P95 ≤ 1.5 s; TPOT P95 ≤ 60 ms; 200 sustained RPS.
Workload: prompt P50=600, P95=1,200, P99=2,500; output P50=180, P95=500, P99=900. Average prompt 800, output 250.
Generic hardware: 4×H100 SXM 80 GB with NVLink, vLLM v1 engine, tensor parallel 4, BF16.

Step 1 — VRAM per GPU. Weights 35 GB, activations 4 GB, overhead 3 GB → KV budget 38 GB/GPU = 152 GB aggregated. KV/token for Llama 70B GQA = 320 KB. Ceiling of live tokens in cache: $152 \times 10^9 / 320 \times 10^3 \approx 475,000$ tokens. At the operational average context (800 prompt + 200 already generated = 1,000 live tokens per request), concurrency ceiling $\approx 475$.

Step 2 — average duration of a request. Prefill 800 tokens / 14,000 tps = 57 ms. Decode 250 tokens × 35 ms/token = 8,750 ms. Total $\approx 8.8$ s per request.

Step 3 — replica throughput. If the replica sustains operational concurrency 40 and each request lasts 8.8 s, the replica delivers roughly $40 / 8.8 \approx 4.5$ requests/s in steady state.

Step 4 — number of replicas. For a 200 RPS target: $200 / 4.5 \approx 45$ replicas. That is 45 × 4 = 180 GPUs. Too much: this sizing does not work because the cost per request is high.

Step 5 — review the levers. Before buying more hardware, there are three levers to explore, in this order:

Quantisation. Dropping to FP8 reduces weights to 17.5 GB/GPU (leaving more VRAM for KV cache → more concurrency), roughly doubles decode tokens/s (HBM saturated by half), and degrades MMLU quality typically by 0.5–1.5 points on models like Llama 70B. Rewriting the calculation in FP8: TPOT drops to ~18 ms, total time per request to 4.7 s, RPS per replica rises to ~8.5, replicas needed ≈ 24, equivalent to 96 GPUs.
Speculative decoding. With a small drafter and 60–70 % acceptance, effective TPOT falls 30–40 %. RPS per replica rises to ~12, replicas ≈ 17 = 68 GPUs.
Disaggregated serving. Separating prefill workers and decode workers lets each be scaled to the real mix of the workload (see Disaggregated serving). It usually shaves off another 20–40 % under asymmetric workloads.

Step 6 — recommended sizing. For the example, with FP8 + speculative decoding and 25 % head-room: 20 vLLM replicas at TP=4 over 80 H100 SXM. If the team does not want to depend on aggressive quantisation (pure BF16 for maximum fidelity), the calculation rises to 30 replicas = 120 GPUs and forces a renegotiation of the SLO or the budget.

Step 7 — write down the assumptions. This is the part no valid sizing skips. In the team’s repo, next to the calculation:

# sizing/llama70b-prod.yaml
fecha: 2026-06-01
slo:
 ttft_p95_ms: 1500
 tpot_p95_ms: 60
 rps_target: 200
workload:
 prompt_tokens_p50: 600
 prompt_tokens_p95: 1200
 output_tokens_p50: 180
 output_tokens_p95: 500
 asunto: chat productivo con RAG ligero
modelo:
 arquitectura: llama-70b-instruct
 formato_pesos: fp8
 motor: vllm-v1
hardware:
 gpu: H100-SXM-80GB
 topologia: TP=4 con NVLink intra-nodo
 red_inter_replica: 25 GbE
optimizaciones:
 - paged_attention
 - chunked_prefill
 - speculative_decoding (drafter llama-1.1b, aceptación esperada 65%)
asunciones_criticas:
 - utilizacion_hbm_eficiente: 0.55
 - head_room_pico_sobre_p95: 0.25
 - aceptacion_speculative_min: 0.55
plan_validacion:
 - benchmark vllm bench serve antes de procurement
 - canary 10% durante 7 días post-deploy

Without this YAML, the calculation is not reproducible a month later.

MoE case: Mixtral 8×22B (~141 B total, 39 B active)

MoEs change the calculation along one key dimension: the total weights are large but the active weights per token are small. For Mixtral 8×22B with top-2 routing:

Weight VRAM: $141 \times 2 = 282$ GB BF16. With TP=4 → 70 GB/GPU. It does not fit in an H100 80 GB alongside KV cache + activations. You need TP=8 (~35 GB/GPU) or FP8 with TP=4 (~35 GB/GPU).
Decode TPOT: dominated by the weights read per token, which are $\sim 39 / 8 \cdot 2 \approx 9.75$ GB/GPU with TP=4 (one top-2 expert per token, divided across 4 GPUs). On an H100 with HBM at 3.35 TB/s, theoretical TPOT ≈ 3 ms/token. In practice, 10–20 ms at reasonable concurrency.
Prefill: similar to the dense model of the active weights, ~39 B FLOPs/token.

MoE sizing usually delivers more RPS per GPU than an equivalent dense model: the low cost per token compensates for the extra VRAM. See MoE inference for the detail of routing and why a high batch is decisive so that each expert sees enough tokens.

Sensitivity table: context and quantisation

For Llama 70B on 4×H100 SXM (TP=4), operational concurrency per replica with an SLO of TTFT 1.5 s / TPOT 60 ms:

Average context	BF16	FP8	INT4 (AWQ)
500 tokens	55	110	180
1,000 tokens	40	80	130
2,000 tokens	24	50	85
4,000 tokens	12	26	48
8,000 tokens	6	13	25

Approximate numbers from public vLLM benchmarks as of June 2026, with ±20 % variation depending on engine version and the headroom adopted. To validate on your own hardware: vllm bench serve with your profile of real prompts.

The five habitual traps

Trap 1 — confusing the mean with P95. Average throughput over an hour may be 50 RPS while the 5-minute peak reaches 180 RPS. Sizing against the mean guarantees breaking the SLO at every peak. Rule: size against the hourly P95, with 20–30 % head-room over P95.

Trap 2 — not measuring the real prefill/decode split. A “RAG with short answers” workload spends 70–80 % of GPU time in prefill; a “writing assistant generating essays” spends 80 % in decode. The useful optimisations (chunked prefill vs speculative decoding) change radically. Without measuring it, you buy badly balanced hardware.

Trap 3 — sizing with no head-room for retraining or rollback. The production cluster is not just the inference engine: there is a re-embedding batch when the embeddings model changes, continuous canary evals (see Canary, blue-green and shadow), light fine-tuning, hot stand-by for rollback. Reserve 15–25 % of capacity for those non-negotiable workloads.

Trap 4 — “GPU at 100 % SM utilization” as a target. SM occupancy of 95 % with saturated HBM produces the same throughput as SM at 60 % with saturated HBM. The bottleneck in decode is the HBM. Optimising for “GPU usage 100 %” without looking at HBM utilisation and arithmetic intensity makes you spend more on GPU without gaining throughput. See GPU observability for LLM inference for which metrics to actually watch.

Trap 5 — not documenting the assumptions. A sizing with no reproducible YAML (workload, model, engine, head-room, critical assumptions) leaves the team with no way of knowing what changed when the cluster stops meeting the SLO six months later. Documenting is cheap; losing a quarter to debugging is not.

Applied to typical on-premise hardware

For a generic cluster of 4×H100 SXM 80 GB with intra-node NVLink and 25 GbE between nodes, the recurring configurations in May 2026 are:

Model	Format	TP	Replicas that fit	Typical RPS per node (ctx 1K)
Llama 8B	BF16	1	4 (one per GPU)	240–320
Llama 8B	FP8	1	4	450–600
Llama 70B	BF16	4	1	30–45
Llama 70B	FP8	4	1	60–90
Llama 70B	INT4 AWQ	2	2	90–130
Mixtral 8×22B	FP8	4	1	90–140
Qwen 72B	BF16	4	1	28–42

These numbers are orders of magnitude to start the conversation, not commitments. The definitive sizing is validated with vllm bench serve or genai-perf (NVIDIA) using the customer’s real prompt/output profile. The prefill/decode asymmetry of each case’s workload can move these numbers 30–50 % up or down.

For clusters of 8×H100 SXM (typical of DGX servers or equivalent replicas), the options open up to TP=8 for 405B-class models or multi-replica TP=2 for 70B models with greater density. The metric that decides is always the same: tokens meeting SLO per kW and per euro of amortised hardware.

How the sizing is validated before buying

The spreadsheet sizing is the first half. The second is the validation benchmark.

Stage 1 — back-of-the-envelope sizing. This post’s formulas applied to the SLO and the expected workload. Output: an approximate number of replicas and a topology.

Stage 2 — synthetic micro-benchmark. On a borrowed GPU or one rented by the day, bring up the engine with the chosen model and run vllm bench serve with prompts of representative lengths. Validate TPOT, prefill TPS and the concurrency ceiling. Calibrate the HBM efficiency factor ($\eta$) used in the formulas.

Stage 3 — load test with realistic traffic. Generate traffic following the real distribution of the customer’s workload (not Poisson, not constant: the real trace). Measure P50/P95/P99 of TTFT, TPOT, throughput. Confirm the head-room.

Stage 4 — canary in production. With the cluster sized, route 5–10 % of real traffic for 7–14 days before closing the procurement of additional hardware. See Canary, blue-green and shadow for the mechanics.

Jumping from Stage 1 to full procurement is the most frequent cause of a cluster that is 40 % oversized and 60 % undersized at the same time, in different regions of the workload. Four weeks of validation done well save four months of refactoring.

What we have not covered (upcoming articles)

The observability metrics that close the sizing loop in production, see GPU observability for LLM inference.
Autoscaling that adjusts replicas to the real traffic curve, see Autoscaling LLM on Kubernetes.
Detailed cost accounting per tenant (showback / chargeback) over the sized hardware.
Sizing for continuous fine-tuning (PEFT and light training) that shares a cluster with inference.

References

Kwon et al. — vLLM: Easy, Fast, and Cheap LLM Serving with PagedAttention (SOSP 2023).
Zhong et al. — DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving (OSDI 2024).
Agrawal et al. — Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve (OSDI 2024).
NVIDIA — H100 Tensor Core GPU Architecture Whitepaper (HBM3 memory, bandwidth, sustained FLOPs).
vLLM project — vllm bench serve reference (benchmarking CLI included in the repo).
NVIDIA — genai-perf (official tool for benchmarking LLM services).

Technical controls: the ENS × ISO 42001 × EU AI Act cross-mapping over the on-premise LLM architecture

Mon, 01 Jun 2026 06:30:00 +0200

Third post in the blog’s AI governance trilogy. The first, ISO/IEC 42001, broke down the certifiable management system. The second, EU AI Act, broke down the directly applicable legal regulation. This one covers the third framework that appears when the client is a Spanish public administration or an essential-services entity: the ENS (Esquema Nacional de Seguridad, Spain’s national security framework, Royal Decree 311/2022). The editorial challenge: triple compliance is not the arithmetic sum of three jobs; it is one single set of technical evidence labelled for three lenses. This post takes apart how that evidence is built, measure by ENS measure.

TL;DR

Royal Decree 311/2022 updated the Spanish Esquema Nacional de Seguridad (ENS), aligning it with NIS2 and with the modern cyber-threat landscape. It applies mandatorily to the entire Spanish public sector (central, regional and local administration, universities, autonomous bodies) and to suppliers that provide IT services to them. The framework defines 74 security measures organised into three blocks (organisational framework org, operational framework op, protection measures mp), three application categories (Basic / Medium / High) according to the assessment of five dimensions (Confidentiality, Integrity, Traceability, Authenticity, Availability), and an Annex II with the measure × category matrix that dictates what is required at each level. This post maps the technical measures relevant to LLM systems (those of the op block: planning, access control, operation, external services, continuity, monitoring; and those of the mp block: communications, applications, information, services) against Annex A of ISO/IEC 42001 and the operational articles of the EU AI Act (Arts. 9 to 15, 17, 72 and 73), and shows that the overlap is massive: a single technical artefact from the blog’s OSS stack, the OTel spans of tracing, the datasets versioned with DVC, the LLM Guard scanners, the guardrail decisions, the incidents of the retrain loop, simultaneously satisfies ENS measures, 42001 controls and AI Act articles, provided it is labelled with the correct codes from the moment it is captured. The editorial thesis: the difference between a system that passes all three audits and one that suffers three separate certifications is not budget, it is disciplined labelling and a common vocabulary. The post builds the master table of triple compliance, walks through the case of the multi-tenant chatbot for a High-category public administration as a live checklist, and closes with the five traps of triple compliance.

The analogy: the triple inspection of a critical building

A critical building, say the control centre of a regional electricity grid, a reference hospital, or a 112 emergency response centre, goes through three different inspections of the same physical elements:

The regional inspection reviews the planning permit, local fire regulations and accessibility under regional legislation.
The European inspection reviews CE marking of equipment and conformity with EU directives on energy efficiency.
The ISO quality inspection reviews processes and maintenance under a management standard.

All three inspectors look at the same fire detector hanging from the ceiling. The regional one wants to see the installation certificate with the installer company’s number; the European one wants to see the CE mark stamped on the casing; the ISO one wants to see the monthly maintenance record with signatures. There is only one detector. The evidence each inspector needs is different. If the building management team kept one folder per inspector, the installation certificate, the photo of the CE mark and the maintenance sheet would live in three places, and on audit day inconsistencies between the three folders would surface. The professional way to operate: one single file per asset, with tags pointing at the three sets of legislation.

The on-premise LLM system is that critical building. The OTel spans from tracing are the fire detector: a single technical piece that satisfies ENS measure op.exp.8 (activity logging), 42001 control A.8.2 (information to interested parties), and EU AI Act article 12 (record-keeping) provided the spans carry the correct metadata: traceparent propagated, gen_ai.* semantic conventions, WORM retention, PII redacted by LLM Guard Vault. Without consistent labelling from capture onwards, audit day produces three folders that do not add up.

The analogy matters because it bounds an architectural decision: labelling evidence is not a compliance topic, it is a technical design topic. It is decided when the OTel Collector is set up, when the Vault schema is designed, when the format of retrain incidents is agreed. Postpone it, and the debt compounds.

ENS in 15 seconds

Regulatory framework: Royal Decree 311/2022, of 3 May, in force since 5 May 2022. It replaces RD 3/2010. It implements the update of the Esquema with NIS2, the NCS (Estrategia Nacional de Ciberseguridad, Spain’s national cybersecurity strategy) and the post-2020 reality.
Mandatory scope: the Spanish public sector (central, regional and local administration, universities, autonomous bodies), public-sector entities with their own legal personality, and private suppliers that provide IT services to them (a very important clause in consultancy: if your client is the Junta de Andalucía, the Comunidad de Madrid or Bilbao City Council, you fall under ENS by contractual extension).
Operational applicability in 2026: any AI project financed with NextGenerationEU European funds, plus any project with public data, plus any integration with electronic administrative systems.
Categorisation: three categories, Basic / Medium / High, according to the assessment of the five dimensions (C, I, T, A, D). The category dictates which measures are required and at what depth.
Certification: for Medium and High categories, a formal audit by an accredited body. For Basic category, self-assessment. Certification cycle is biennial.
Authority: the Centro Criptológico Nacional (CCN-CERT, Spain’s national cryptologic centre), part of the CNI intelligence agency. It maintains the ens.ccn.cni.es portal with the STIC guides (800 series).
Total number of measures: 74, organised into three blocks, plus Annex II with the measure × category matrix that defines what applies at each level.

The five security dimensions and their AI mapping

To classify the system as Basic / Medium / High, the ENS requires each dimension to be rated on a scale (not applicable / low / medium / high). The final category is the highest of the five. For LLM systems, the typical rating:

Dimension	Meaning	Typical LLM rating	Reason
C Confidentiality	Information protected from disclosure	Medium-High	PII in prompts, secrets in context, intellectual property in the RAG corpus
I Integrity	Information protected from modification	Medium-High	An altered RAG corpus produces false answers; a manipulated model produces directed bias
T Traceability	Actions attributable to users	High	Audit: who asked what, when, what were they answered, which dataset trained the model?
A Authenticity	Identity of users and origin of information	Medium	Robust authentication plus identification of chunks by source
D Availability	Service available when needed	Medium	Typical SLA, known recovery time

Typical result: Medium category for internal LLM systems with no sensitive data, High if they handle PII or regulated sectoral data (health, tax, judicial). The category triggers different controls.

The three blocks of measures

Organisational framework (org), 4 measures that always apply, transversal:

org.1 Security policy
org.2 Security regulations
org.3 Security procedures
org.4 Authorisation process

Operational framework (op), 31 measures in 6 subgroups, these are how it is operated:

op.pl Planning (5)
op.acc Access control (6)
op.exp Operation (10)
op.ext External services (4)
op.cont Service continuity (4)
op.mon System monitoring (2)

Protection measures (mp), 39 measures in 7 subgroups, these are what is protected:

mp.if Facility protection (7)
mp.per Personnel management (4)
mp.eq Equipment protection (4)
mp.com Communications protection (4)
mp.si Information media protection (5)
mp.sw Application protection (2)
mp.info Information protection (6)
mp.s Service protection (4)

Of those 74, roughly 25 matter operationally for LLM systems. The rest are either transversal (policies, personnel management, physical facilities) or specific to other layers (telephony, physical media in the literal sense). The rest of this post drops down to the 25 relevant ones.

Technical mapping: key ENS measures × 42001 controls × EU AI Act articles

What follows, by ENS subgroup, are the measures relevant to LLM with their cross-mapping. The Technical evidence from the blog column points to the operational artefact that materialises the measure.

Operational framework — planning (`op.pl`)

ENS measure	Requirement	42001 control	AI Act article	Technical evidence
`op.pl.1` Risk analysis	Formal periodic analysis, MAGERIT or OCTAVE methodology	A.5.4 (alignment with AI risk treatment)	Art. 9 Risk management	Risk document linked to the LLMOps pipeline + evals metrics
`op.pl.2` Security architecture	Architecture documentation, layer segregation	A.4.2 documented info	Art. 15 (technical robustness)	Seven stack layers + seven deployment phases
`op.pl.3` Acquisition	Acquisition criteria that include security	A.10.3 suppliers	Art. 53 (GPAI obligations)	OSS vs hyperscalers lock-in analysis + copyright analysis
`op.pl.4` Sizing	Capacity to support the expected load	A.4.5 system resources	Art. 15 (consistent performance)	Five maturity levels + GPU capacity study
`op.pl.5` Certified components	Preference for certified components	A.10.5 third parties	Art. 53 + Art. 15	OSS inventory with licence analysis + supply chain audit (cosign + SLSA)

Operational framework — access control (`op.acc`)

ENS measure	Requirement	42001 control	AI Act article	Technical evidence
`op.acc.1` Identification	Unique identification of users and processes	A.3.2 roles	Art. 14 (human oversight)	Keycloak / OIDC + JWT with unique `sub` + `user_id_hashed` in OTel spans
`op.acc.2` Access requirements	Need to know, authorisation	A.9.4 intended use	Art. 14 + Art. 26 (deployer)	Per-tenant allowlist in the AI Gateway + RBAC over adapters
`op.acc.3` Segregation of duties	No incompatible accumulation	A.3.2 + A.3.3	Art. 17 (QMS)	Separate roles: AI lead / data steward / SRE / DPO
`op.acc.5` Authentication	Proportionate mechanism, MFA at High	A.4.4 tooling	Art. 14	Keycloak + WebAuthn + mandatory MFA for High category
`op.acc.6` Local access	Protection against physical access	mp.if + A.4.5	Art. 15 (cybersec)	Cluster in a datacenter with physical control (strictly outside the blog’s scope)
`op.acc.7` Remote access	VPN, encryption, endpoint control	A.4.5 + mp.com	Art. 15	WireGuard / Defguard + mTLS for administrative access to the cluster

Operational framework — operation (`op.exp`)

This is the densest subgroup and the one that overlaps most with the LLM architecture.

ENS measure	Requirement	42001 control	AI Act article	Technical evidence
`op.exp.1` Asset inventory	CMDB with all system components	A.4 resources	Art. 49 (EU DB registration)	Inventory of models, adapters and datasets in a CMDB + Helm tags + OpenLineage lineage
`op.exp.2` Security configuration	Documented hardened configuration	mp.eq + A.4	Art. 15	vLLM, KServe and Cilium configurations documented in GitOps + CIS Benchmarks for K8s
`op.exp.3` Configuration management	Traceable and authorised changes	A.6.2 + A.4.2	Art. 9 + Art. 15	GitOps with Argo CD / Flux + mandatory PR review + immutable tags
`op.exp.5` Change management	Changes planned, authorised, recorded	A.6.2.6 + clause 8	Art. 9 (lifecycle) + Art. 72 (post-market)	CI/CD pipeline from the LLMOps post + change advisory board
`op.exp.6` Protection against malicious code	Antivirus, EDR, malware control	mp.eq	Art. 15 (cybersec)	Image scanning (Trivy / Grype) + container runtime security (Falco / Tetragon)
`op.exp.7` Incident management	Detection → response → recovery procedure	A.3.3 + clause 10	Art. 73 (serious incident reporting)	Incident-driven retrain + CCN-CERT channel + notification template
`op.exp.8` Activity logging	Auditable logs, minimum 2-year retention at High category	A.8.2	Art. 12 + Art. 19	OTel GenAI tracing + Tempo / Jaeger + Loki
`op.exp.9` Incident management records	Incident log with root cause and corrective action	clause 10	Art. 73	Ticketing system + structured retrain incident events
`op.exp.10` Record protection	Immutable logs, cryptographic integrity, guaranteed retention	A.8.2 + A.4.2	Art. 12	WORM storage (Ceph + immutable bucket) + log signing (sigstore) + 24-36 month retention
`op.exp.11` Cryptographic keys	Key lifecycle management, HSM at High category	mp.info.4	Art. 15	HashiCorp Vault / SOPS + HSM (Yubico, AWS KMS on-prem) for High category

Operational framework — external services (`op.ext`)

Critical when the system integrates GPAI (Llama, Mistral) hosted elsewhere, or when the deployer is a third party.

ENS measure	Requirement	42001 control	AI Act article	Technical evidence
`op.ext.1` Contracting external services	Contract with security clauses, SLA, right to audit	A.10.3 suppliers	Art. 25 + Art. 53	Contract with ENS clauses + annual audit review + Cloud Act analysis
`op.ext.2` Alternative means	Plan B if the supplier goes down	A.4.5 + clause 6	Art. 15 (resilience)	Multi-cluster with failover + qualified alternative GPAI (Llama→Mistral→Qwen)
`op.ext.3` Supply chain protection	Assessment of suppliers and sub-suppliers	A.10 + clause 6	Art. 53 + NIS2 supply chain	SBOM + cosign + SLSA + continuous vulnerability scanning
`op.ext.4` System interconnection	Interconnection agreements, secure gateways	mp.com + A.4.5	Art. 15	API Gateway with mTLS + JWT signing + rate limiting + WAF

Operational framework — continuity (`op.cont`)

ENS measure	Requirement	42001 control	AI Act article	Technical evidence
`op.cont.1` Impact analysis	BIA per system	A.5.5 (impacts on individuals)	Art. 9 (risk management)	Documented BIA with RPO / RTO per system
`op.cont.2` Continuity plan	Documented DRP, RTO/RPO	clause 6 + A.4	Art. 15 (resilience)	DRP + Velero K8s backups + DVC datasets in a secondary bucket
`op.cont.3` Periodic testing	Drills at a defined frequency (annual at High category)	clause 9 (evaluation)	Art. 15	Annual game-day with a simulated disaster + recovery stopwatches
`op.cont.4` Alternative means	Capacity to continue with degraded means	clause 6 + A.4.5	Art. 15	Secondary cluster in a different datacenter + reserved GPU pool + replicated datasets

Operational framework — monitoring (`op.mon`)

ENS measure	Requirement	42001 control	AI Act article	Technical evidence
`op.mon.1` Intrusion detection	IDS/IPS over network and applications	A.9.2 use	Art. 15 (cybersec)	Guardrails as a semantic WAF + LLM Guard PromptInjection + Tetragon eBPF runtime
`op.mon.2` Metrics system	Measurable operational metrics, auditable dashboard	clause 9 (performance evaluation)	Art. 72 (post-market monitoring)	Prometheus + VictoriaMetrics + Grafana + Langfuse dashboards
`op.mon.3` Surveillance (High category)	24×7 monitoring with alerting	clause 9 + clause 10	Art. 72 + Art. 73	SOC with SIEM alerts (Wazuh, OpenSearch, Vector + custom) + on-call rotation

Protection measures — communications (`mp.com`)

ENS measure	Requirement	42001 control	AI Act article	Technical evidence
`mp.com.1` Secure perimeter	Firewall, segmentation, DMZ	A.4.5	Art. 15	Cilium NetworkPolicy + Calico + ingress controllers with WAF (mod_security)
`mp.com.2` Confidentiality protection	Encryption in transit (TLS 1.2+ mandatory, 1.3 recommended)	A.4.5	Art. 15	TLS 1.3 mandatory + cert-manager + Let’s Encrypt or an internal CA
`mp.com.3` Integrity and authenticity protection	Encryption + origin authentication	A.4.5 + A.4.4	Art. 15	Intra-cluster mTLS + JWT signing at the gateway + checksums on artifacts
`mp.com.4` Flow separation	Segmentation of mgmt vs production vs external traffic	A.4.5	Art. 15	Cilium policies + network namespaces + east-west / north-south segregation

Protection measures — applications (`mp.sw`)

ENS measure	Requirement	42001 control	AI Act article	Technical evidence
`mp.sw.1` Application development	Secure SDLC, code review, SAST/SCA	A.6.2.3 responsible design	Art. 9 + Art. 15	Forgejo CI with SAST (Semgrep, CodeQL) + SCA (Trivy, Grype) + mandatory PR review
`mp.sw.2` Acceptance and go-live	Acceptance tests, eval gates before production	A.6.2.5 V&V	Art. 9 + Art. 15	Eval gates from the evals post + canary deploy + pre-go-live metrics

Protection measures — information (`mp.info`)

ENS measure	Requirement	42001 control	AI Act article	Technical evidence
`mp.info.1` Personal data	GDPR compliance + technical and organisational measures	A.5.5 + A.7.6	Art. 10 + Art. 26	LLM Guard Vault + Presidio + minimisation in RAG corpus curation
`mp.info.2` Information classification	Labelling by level (public / internal / confidential / restricted)	A.7.2 data	Art. 10	Schema contracts from data versioning with a `classification` field
`mp.info.3` Encryption	At-rest minimum at Medium category, High category with HSM	A.4.5 + mp.eq	Art. 15	LUKS / dm-crypt on disks + encryption in the Ceph bucket + keys in Vault
`mp.info.4` Electronic signature	Documents signed with a valid certificate (Medium category and above)	A.8.2	Art. 12	Log signing with sigstore + signing of published models with cosign
`mp.info.5` Timestamps	Qualified timestamp for temporal integrity (High category)	A.8.2	Art. 12	Timestamping with a qualified TSA + RFC 3161 on critical events
`mp.info.6` Document cleaning	Removal of unauthorised metadata, anonymisation	A.7.6 + A.5.5	Art. 10	LLM Guard Anonymize (input) + Sensitive (output) + Vault with TTL

Protection measures — services (`mp.s`)

ENS measure	Requirement	42001 control	AI Act article	Technical evidence
`mp.s.1` Email	Anti-spam, anti-malware, optional encryption	A.4.5	—	Outside the direct LLM scope
`mp.s.2` Protection of web services and applications	WAF, hardening, vulnerability management	A.9.2 + mp.com	Art. 15 (cybersec)	AI Gateway (LiteLLM / Envoy AI / Kong AI) with policies + ModSecurity + Cloudflare-like
`mp.s.3` Protection against denial of service	Rate limiting, anti-DDoS, capacity planning	A.4.5 + A.9.2	Art. 15	Rate limiting at the gateway + token quotas + circuit breakers
`mp.s.4` Protection against external threats (High category)	Advanced monitoring, threat intel	A.9.2 + clause 9	Art. 15 + Art. 72	Guardrails, 4 lines + advanced LLM Guard scanners + threat intel feed (CCN-CERT MISP)

Master table of triple compliance — the 25 relevant controls consolidated

The key reading of the table: one row per technical capability. The organisation does not build three solutions for “activity log” (ENS) + “information to interested parties” (42001) + “record-keeping” (AI Act). It builds one single piece of OTel tracing with the correct metadata and presents it labelled for whichever inspector is asking.

The labelling typically materialises in three mechanisms:

Tags in the CI/CD pipelines: every artefact produced carries tags ens:op.exp.8, iso42001:A.8.2, aia:art.12 in its Helm / Argo CD metadata.
Semantic OTel attributes: gen_ai.compliance.ens = "op.exp.8", gen_ai.compliance.iso42001 = "A.8.2" as custom attributes on the relevant spans (not standard yet, but useful internally).
A mapping table in the wiki: a living document of ENS measure → 42001 control → AI Act article → technical runbook + owner + last verification. This is the artefact the auditor consults.

Applied case: multi-tenant chatbot for public administration

A variant of the multi-tenant chatbot from the forensic post, where the client is now a Spanish regional government offering citizen assistance on administrative procedures. The three lenses:

ENS: applies mandatorily, being public sector plus a service to citizens. Category High (citizens’ PII plus a critical service).
ISO 42001: the regional government requires certification from the supplier as a contractual condition.
EU AI Act: if the chatbot informs about procedures but decides nothing on the citizen’s behalf, it is limited risk (Art. 50 transparency). If it automates decisions (admission to a programme, refusal of a benefit), it is high risk.

We assume high risk for the most demanding walkthrough.

The chatbot’s 25 key technical capabilities consolidated

Technical capability	ENS	42001	AI Act	Status
Robust citizen authentication	`op.acc.5` (MFA High category)	A.4.4	Art. 14	Cl@ve + digital certificate + WebAuthn
Unique tenant + user identification	`op.acc.1`	A.3.2	Art. 14 + 26	JWT with `tenant_id` + `user_id_hashed`
Per-tenant allowlist	`op.acc.2`	A.9.4	Art. 14	LiteLLM Proxy with policies + Envoy filter
Encryption in transit, mTLS	`mp.com.2-3`	A.4.5	Art. 15	TLS 1.3 + cert-manager + intra-mesh mTLS
Encryption at rest + HSM keys	`mp.info.3` + `op.exp.11`	A.4.5	Art. 15	LUKS + Vault + Yubikey or nCipher HSM
PII anonymisation input/output	`mp.info.1 + .6`	A.7.6	Art. 10	LLM Guard Vault + Presidio + Llama Guard 4
Traceable logging + WORM	`op.exp.8 + .10`	A.8.2	Arts. 12 + 19	OTel + Tempo + Loki + Ceph WORM 36 months
Log signing + timestamps	`mp.info.4 + .5`	A.8.2	Art. 12	sigstore + qualified TSA (FNMT)
Runtime threat detection	`op.mon.1 + mp.s.4`	A.9.2	Art. 15	Guardrails, 4 lines + Tetragon + Falco
Operational metrics dashboard	`op.mon.2`	clause 9	Art. 72	Prometheus + Grafana + Langfuse
24×7 SOC surveillance	`op.mon.3` (High category)	clause 9	Art. 72	SOC with SIEM + on-call rotation
Incident management with notification	`op.exp.7 + .9`	clause 10	Art. 73	Incident-driven retrain + CCN-CERT channel
CI/CD pipeline with eval gates	`mp.sw.1 + .2`	A.6.2.3-5	Arts. 9 + 15	Forgejo CI + Semgrep + Trivy + DeepEval gates
Backups + DRP	`op.cont.1-4`	clause 6 + A.4	Art. 15	Velero + DVC datasets in a secondary bucket + annual game-day
Systematic risk analysis	`op.pl.1`	A.5 + cl.6	Art. 9	MAGERIT + FRIA + ISO/IEC 23894 impact assessment
Documented secure architecture	`op.pl.2`	A.4.2	Art. 15	Seven layers + seven phases
Supplier management	`op.ext.1 + .3`	A.10.3	Art. 53	Contracts with ENS clauses + SBOM + Cloud Act analysis per GPAI
Certified components	`op.pl.5`	A.10.5	Art. 53	Inventory with licence + supply chain audit
Configuration hardening	`op.exp.2 + .3`	A.4 + A.6	Art. 15	CIS Benchmarks for K8s + GitOps + immutable tags
Protection against malicious code	`op.exp.6`	mp.eq	Art. 15	Image scanning + runtime security
Asset inventory	`op.exp.1`	A.4	Art. 49	CMDB + Helm tags + OpenLineage
Network segmentation	`mp.com.4`	A.4.5	Art. 15	Cilium NetworkPolicy + namespaces
Rate limiting + anti-DoS	`mp.s.3`	A.4 + A.9	Art. 15	LiteLLM rate limit + token quotas
Information classification	`mp.info.2`	A.7.2	Art. 10	Schema contracts with a `classification` field
Transparency towards the user	—	A.9.4	Art. 50	UI banner + disclaimer in responses

Result: the 25 technical capabilities are common to all three frameworks. Only two require evidence for a single framework in isolation: mp.info.4-5 (qualified signature and timestamp, ENS-specific, not required so literally by 42001 or the AI Act) and Art. 50 (transparency banner, AI Act-specific). The rest are the same technical piece with three labels. A well-organised team certifies all three in series with only marginal extra work between the second and the third.

The five traps of triple compliance

Trap 1 — Measuring the same thing three times. Inexperienced teams create three separate dashboards (one for ENS, another for 42001, another for the AI Act) with the same metrics duplicated. The result: three sources of truth that diverge, three audit teams with different figures, three corrections to resolve one and the same deviation. The rule: one metric, three labels.

Trap 2 — Losing the control that covers only one framework. mp.info.4 (electronic signature) is ENS-specific and gets forgotten while the team is focused on 42001 + AI Act. On ENS audit day, the gap surfaces. Solution: the master table keeps every control visible, orphans included.

Trap 3 — Bias towards the most recent framework. The team spends 80% of the effort on the AI Act because it is the newest, and forgets the rigour of the ENS, which has been in force for 14 years. The ENS measures are technically more prescriptive than the AI Act (which is legally stricter but leaves implementation freedom). Moving up a level to ENS High category introduces specific requirements (HSM, qualified timestamps, 24×7 surveillance) that the AI Act does not spell out. The ENS deserves respect for its technical detail, not for its legal stature.

Trap 4 — Mixing ENS Medium and High categories. The Annex II matrix of RD 311/2022 dictates which measures are required and at what depth per category. Moving from Medium to High changes 15-20 controls, which is not marginal. The category is decided at the start of the project and documented; changing it halfway forces a complete re-audit.

Trap 5 — Audit fatigue inside the technical team. Three audits a year (biennial ENS + annual 42001 surveillance + ad-hoc AI Act by the authority) exhausts the team if they are not planned and if evidence is not reused. The professional approach: one single quarterly internal audit cycle with rotating scope, producing evidence consumable by all three external auditors. The difference between 30 days a year and 90 days a year lost to audits is the discipline of single evidence plus disciplined labelling.

What we have not covered (upcoming posts)

The CCN-CERT STIC series applicable to AI systems: STIC 800-159 (web service operation), STIC 800-105 (cryptography), STIC 800-150 (cloud environments). Each adds technical detail on how to materialise the ENS measures.
The GDPR as a fourth lens: privacy and personal data protection. It overlaps with mp.info.1 and with A.7.6 of 42001 plus Art. 10 of the AI Act. Material for an analogous pass over LOPDGDD + GDPR + AEPD against the three frameworks seen here.
Concrete templates of technical evidence with minimum fields: a log entry with every attribute required by the three lenses, an incident report with every field demanded by the three frameworks, a unified ENS / AI Act / 42001 declaration of conformity.
The NextGenerationEU funds case: specific compliance requirements for AI projects financed with European funds, where the AI Act + ENS are contractually mandatory.
42001 + ENS High category combined with DORA (Digital Operational Resilience Act, Reg. 2022/2554) for Spanish financial entities deploying AI.

References

RD 311/2022 — por el que se regula el Esquema Nacional de Seguridad. https://www.boe.es/buscar/act.php?id=BOE-A-2022-7191.
CCN-CERT — Portal ENS: https://ens.ccn.cni.es/. Guías STIC Series 800.
CCN-STIC 803 — Valoración de los sistemas y de la información. Metodología para asignar categoría ENS.
CCN-STIC 804 — Esquema Nacional de Seguridad. Guía de implantación.
CCN-STIC 824 — Informe del estado de seguridad. Plantilla para auditoría ENS.
MAGERIT v3 — Metodología de Análisis y Gestión de Riesgos del Ministerio de Asuntos Económicos. Insumo de op.pl.1.
ISO/IEC 42001:2023 — Sistema de gestión IA. Norma certificable que solapa con ENS.
Regulation (EU) 2024/1689 (EU AI Act) — texto consolidado.
NIS2 (Dir. 2022/2555) — directiva de ciberseguridad que el ENS implementa parcialmente en su versión 2022.

Fundamentals on lo0 — Tech Blog

Anatomy of an LLM request in production, May 2026: a tour of the six stages following a single request

TL;DR

You are here: every stage at once

The analogy: forensic analysis of a request

The scenario: a multi-tenant support chatbot for regulated customers

Step 0 — The frame: the answer the user sees

Rewinding backwards: what was already in place before the request

t = −90 days — The previous Retrain stage closes the earlier cycle

t = −60 days — The Data stage: the enriched dataset is versioned and enters circulation

t = −45 days — The Tune stage: the customer_support_v7 adapter is trained

t = −38 days — The Eval stage: the v7 adapter goes through eval gates

t = −31 days — The Deploy stage: the v7 adapter enters serving

Moving forward: the user’s request crosses the system

t = 0 ms — Entry through the gateway

t = 8 ms — Pulling the versioned prompt

t = 12 ms — RAG retrieval

t = 40 ms — Building the final payload

t = 45 ms — Prefill

t = 580 ms — Decode (streaming)

t = 4,200 ms — Complete answer, root span closed

In parallel: Observe is watching

The feedback: the loop closes

Insertion into feedback_signals

Triage by root cause

The next cycle picks it up

What goes in each trace: identity and traceability

Synthesis diagram: how the pieces fit

The on-premise stack applied

Ten bridges between stages where the system breaks

How to walk through the blog

What we have not covered (yet)

See also

References

Knative and scale-to-zero for LLM inference: when switching off the GPU saves money and when it costs you the SLO

TL;DR

The analogy: the restaurant that switches off the kitchen

How Knative scales

KPA versus HPA

Why an LLM cold start is not starting a pod

The techniques that shorten the wait

Knative and KServe: the two modes

The economics

Operational traps and honest scepticism

For an inference factory

See also

Sources

KubeVela and Score: handing over the platform by CLI without teaching the developer Kubernetes

TL;DR

The analogy: the order form against the shopping list

KubeVela: the application as the unit

What it is and what state it reaches 2026 in

The OAM model in five pieces

CUE, the power and the toll

The vela CLI: Heroku on the outside, GitOps on the inside

Multi-cluster, which is what matters to you

VelaUX, the lightweight portal thrown in

Score: describe the workload once

What it is and what it solves

The two implementations

Score against KubeVela and Helm

Where each one fits: the cross-cutting comparison

The fit with inference: a pattern, not an official recipe

Caveats, no frills

For your sovereign inference factory

See also

Sources

Crossplane: the control plane that turns your cluster into a private cloud API

TL;DR

The analogy: the thermostat against the light switch

What Crossplane is and what state it reaches 2026 in

The v2 turn (which changes how you think about it)

The model, piece by piece

Composition functions: the real logic

Crossplane against Terraform, OpenTofu and Pulumi

The awkward question: is it an AWS-style CLI?

On-premise providers: provisioning without touching AWS

The fit with GPU inference: from pattern to Modelplane

What it costs you to operate your own control plane

How it lives alongside what you already have